Method and system for displaying eye images to a subject and for interacting with virtual objects - Patent Application 20070122999

Abstract

The processing subsystem generates a sensory image from information-bearing neural impulses transmitted from one or more eyes of the subject along one or more nerves to a visual processing region of the subject's brain in response to the subject viewing a real-world scene. The processing subsystem generates a display image based on the sensory image and controls a display device to display the display image to the subject. In certain embodiments, the processing subsystem generates the display image by manipulating or modifying the sensory image to include a virtual image and provides a type of virtual pointing to the display image that is used to trigger one or more actions.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. Patent Application No. 17 / 740,425, filed May 10, 2022, which is a continuation-in-part of U.S. Patent Application No. 17,534,622 (now U.S. Patent No. 11,395,620), filed November 24, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 196,274, filed June 2, 2021. The entire disclosures of these applications are incorporated herein by reference.

[0002] The present disclosure relates to displaying eye images based on conversion between neural impulses carrying image information and image data. [Background technology]

[0003] The human visual system includes the eye, the brain, and parts of the nervous system. Generally, light is sensed by photoreceptors (rods and cones) in the eye and converted into nerve impulses that are transmitted by the optic nerve to the brain, where they are interpreted as images and visual perception. Summary of the Invention [Means for solving the problem]

[0004] According to embodiments of the present invention, systems and methods are provided for displaying images to a subject. The displayed images are based on images generated from neural impulses carrying image information that are transmitted along one or more nerves from one or more eyes to visual processing regions of the subject's brain in response to the subject viewing a real-world scene. The images generated from the neural impulses are referred to as "perceptual images" or "eye images." These eye images serve as the basis for images displayed to the subject, and in certain embodiments, these eye images can be manipulated and / or modified to include virtual images. In certain embodiments, the systems and methods provide a type of "virtual pointing" to the eye image or manipulated / modified eye image, which can be used to trigger one or more actions.

[0005] Certain preferred embodiments according to the first aspect of the present invention provide a processing subsystem communicatively coupled to a display device, the display device being deployable relative to a subject (i.e., a user), e.g., between the subject's eyes and a scene. The processing subsystem operates to receive signals associated with neural impulses transmitted by one or more nerves associated with the one or more eyes to a visual processing region of the subject's brain in response to the one or more eyes viewing the scene during a sample period. The processing subsystem processes the received signals to generate a perceptual image, thereby performing a conversion from the neural impulses to a generated perceptual image, where the perceptual image represents the visual perception of the scene by the visual processing region (e.g., visual cortex) of the subject's brain. The processing subsystem further operates to provide a display image based on the generated perceptual image to the display device for display during the display period. In certain preferred, but non-limiting, embodiments, the processing subsystem further operates to controllably and repeatedly switch the display device between a display mode associated with the display period and a non-display mode associated with the sample period, whereby the subject's field of view alternates between the display image (projected by the display device) and the scene.

[0006] Certain preferred embodiments according to the second aspect of the present invention provide a processing subsystem and display device similar to those described above, but the processing subsystem is operative to generate a display image by modifying the perceived image with an image of the object so that the object appears superimposed on the scene in the display image, and to identify a data marker or data object that marks / tags / identifies one or more data elements in the display image that correspond to the spatial location of the object in the display image. For example, the processing subsystem is preferably operative to identify a data record (e.g., in a database) having metadata associated with one or more data elements of image data for the display image, where the one or more data elements are associated with the spatial location of a portion of the display image that includes the object (or at least a portion of the object). In certain preferred, but non-limiting, embodiments, the processing subsystem is further operative to initiate or cause a response action associated with the object in response to identifying the data record, for example, if the data record includes a "trigger" (for initiating a response action) associated with the spatial location. In certain preferred, but non-limiting, embodiments, the processing subsystem is operative to identify the data record for a portion of the display image that includes the object in response to a subject-initiated action.

[0007] It should be noted that the two aspects of the present invention presented herein are independently useful, and that display mode switching is not limited to use in embodiments that perform data record identification. Similarly, data record identification embodiments are not limited to use in embodiments that perform display mode switching. However, certain synergistic effects may be achieved by implementing both of the two aforementioned aspects of the present invention. Furthermore, it should be noted that the present invention may include additional aspects other than the two explicitly mentioned above, and that the present invention is not limited to only these two aspects.

[0008] In accordance with the teachings of one embodiment of the present invention, there is provided a system for use with a subject having a brain including an area responsible for visual perception, comprising: a display device operable in a display mode and a non-display mode; and a processing subsystem for communicating with at least one nerve associated with an eye of the subject, the processing subsystem being configured to: receive signals associated with nerve impulses transmitted by the at least one nerve in response to viewing at least a portion of a scene with the eye during a sample period, the sample period being contemporaneous with operation of the display device in the non-display mode; process the received signals to generate a perceived image representing visual perception of the scene by the area of ​​the brain; and provide to the display device an image for display based on the perceived image during a display period coincident with operation of the display device in the display mode.

[0009] Optionally, the processing subsystem is further configured to switch the display device between a non-display mode and a display mode, the display device not displaying images when operating in the non-display mode.

[0010] Optionally, the display device is an at least partially transparent display, such that when the display device is operating in the non-display mode, at least a portion of the scene is visible by the eye through the display device.

[0011] Optionally, when the display device is operating in the display mode, the display device displays an image that is visible to the eye, and when the display device is operating in the non-display mode, the display device does not display an image.

[0012] Optionally, the display device is configured to operate in a non-display mode during a number of sampling periods and to operate in a display mode during a number of display periods.

[0013] Optionally, the processing subsystem is further configured to controllably switch the display device so that it alternates between operating in a non-display mode and operating in a display mode, the processing subsystem being configured, when the display device operates in the non-display mode, to receive signals related to nerve impulses transmitted by the at least one nerve in response to the eye viewing at least a portion of the scene during a corresponding one of the sample periods, and the processing subsystem being configured, when the display device operates in the display mode during a corresponding one of the display periods subsequent to the corresponding one of the sample periods, to provide to the display device a corresponding display image based on the corresponding perceived image generated by the processing subsystem as a result of processing the received signals related to nerve impulses transmitted by the at least one nerve in response to the eye viewing at least a portion of the scene during the corresponding one of the sample periods.

[0014] Optionally, the display device comprises at least one of a surface mounted display, a head mounted display, a head up display, or a holographic display.

[0015] Optionally, the system further comprises an interface arrangement that enables the processing subsystem to communicate with the at least one nerve.

[0016] Optionally, the interface configuration is external to the subject.

[0017] Optionally, the image is a perceptual image.

[0018] Optionally, the image is a modified version of the perceived image.

[0019] Optionally, the image is the perceived image extended to include images of objects that are overlaid on the perceived image.

[0020] Optionally, the processing subsystem is further configured to identify data records associated with one or more elements of image data of the image, the one or more elements of image data being associated with a spatial location of a portion of the image that includes at least a portion of the object, and initiate at least one response action in response to the identification, the at least one response action being associated with the object in the image.

[0021] Optionally, the subject has a pair of eyes and the display device is arranged to project an image for viewing by both of the subject's eyes.

[0022] Optionally, the subject's eye is a first eye of the subject, the display device being arranged to project an image for viewing by the first eye, and the system further comprising a second display device arranged to project an image for viewing by a second eye of the subject, the second display device being operable in a display mode and a non-display mode.

[0023] Optionally, the display device is configured to operate in a display mode when the second display device is operating in a non-display mode, and to operate in a non-display mode when the second display device is operating in a display mode.

[0024] According to one embodiment of the teachings of the present invention, there is also provided a system for use with a subject having a brain including an area responsible for visual perception, comprising: a display device configured to be positioned in a spatial relationship with the subject; and a processing subsystem for communicating with at least one nerve associated with an eye of the subject, the processing subsystem configured to: receive signals associated with nerve impulses transmitted by the at least one nerve in response to the eye viewing a scene; process the received signals to generate a perceived image representative of the visual perception of the scene by the area of ​​the brain; generate a displayed image by combining the perceived image and an image of an object such that the object appears superimposed on the scene in the displayed image; provide the displayed image to the display device as seen by the subject; and identify a data record associated with one or more elements of image data for the displayed image, the one or more elements of image data being associated with a spatial location of a portion of the displayed image including at least a portion of the object.

[0025] Optionally, the processing subsystem is further configured to initiate at least one responsive action associated with the object in response to identifying the data record.

[0026] Optionally, the at least one responsive action comprises at least one of activating or manipulating at least one electronic device associated with the object.

[0027] Optionally, the at least one response action is selected from a plurality of response actions, each response action of the plurality of response actions being associated with a corresponding data record of the plurality of data records, each data record of the plurality of data records being associated with a corresponding one or more elements of the image data.

[0028] Optionally, the processing subsystem is configured to identify a data record in response to detecting an event associated with a portion of the displayed image that includes at least a portion of the object.

[0029] Optionally, the processing subsystem is further configured to initiate feedback provided to the subject in response to detecting an event associated with a portion of the displayed image that includes at least a portion of the object.

[0030] Optionally, the system further comprises a haptic interface associated with the object, and the processing subsystem is further configured to actuate the haptic interface to provide haptic feedback to the object in response to detecting an event associated with a portion of the displayed image that includes at least a portion of the object.

[0031] Optionally, the object is one of a plurality of objects, and the system further comprises at least one storage medium for maintaining a plurality of data records including said data record, the plurality of data records including a plurality of subsets of the data records, each subset being associated with a corresponding object of the plurality of objects.

[0032] According to one embodiment of the teachings of the present invention, there is further provided a method for use in a subject having a brain including a region responsible for visual perception, the method including: disposing a processing subsystem in communication with i) at least one nerve associated with an eye of the subject, and ii) a display device disposed in a spatial relationship with the subject and configured to operate in a display mode and a non-display mode; receiving, by the processing subsystem, signals associated with nerve impulses transmitted by the at least one nerve in response to viewing at least a portion of a scene with the eye during a sample period, the sample period being contemporaneous with operation of the display device in the non-display mode; processing, by the processing subsystem, the received signals to generate a perceived image representing visual perception of the scene by the region of the brain; and providing, by the processing subsystem, an image for display to the display device based on the perceived image during a display period contemporaneous with operation of the display device in the display mode.

[0033] According to one embodiment of the teachings of the present invention, there is further provided a method for use in a subject having a brain including an area responsible for visual perception, the method including: disposing a processing subsystem in communication with i) at least one nerve associated with an eye of the subject, and ii) a display device disposed in spatial relationship to the subject and configured to operate in a display mode and a non-display mode; receiving, by the processing subsystem, signals associated with nerve impulses transmitted by the at least one nerve in response to the eye viewing a scene; processing, by the processing subsystem, the received signals to generate a perceived image representing the visual perception of the scene by the area of ​​the brain; generating, by the processing subsystem, a displayed image by combining the perceived image and an image of an object such that in the displayed image, the object appears superimposed on the scene; providing, by the processing subsystem, the displayed image to the display device so as to be visible to the subject; and identifying, by the processing subsystem, a data record associated with one or more elements of image data for the displayed image, the one or more elements of image data associated with a spatial location of a portion of the displayed image including at least a portion of the object.

[0034] As used herein, the term "perceptual image" generally refers to an image generated by converting nerve impulses or signals related to nerve impulses into image data, where the converted nerve impulses (or nerve impulses related to the converted signals) are nerve impulses transmitted from one or more eyes of the subject to visual processing regions of the subject's brain, the regions of the subject's brain responsible for visual perception, in response to the subject viewing a scene. In other words, a "perceptual image" is an image generated or formed that represents what a subject sees with one or more eyes of the subject when viewing a scene. The term "eye image" is also used interchangeably with the term "perceptual image."

[0035] Also, as used herein, the term "displayed image" generally refers to an image based on a "perceptual image" and provided to a display device for display. In other words, a "displayed image" includes image elements or components that are elements / components of the "perceptual image" or are derived from elements / components of the "perceptual image." In some cases, a "displayed image" can be the "perceptual image" itself. In other cases, a "displayed image" is a modified version of a "perceptual image." This modified version of a "perceptual image" may be a "perceptual image" that has been modified to change one or more image parameters and / or characteristics, such as, but not limited to, color, brightness, contrast, size, etc., of one or more elements (e.g., pixels) of the "perceptual image." A modified version of a "perceptual image" may also be a cropped version of a "perceptual image" in which certain elements of the "perceptual image" have been removed (i.e., deleted). A modified version of a "perceptual image" may be a "perceptual image" extended to include elements or components of other images, such as elements or components corresponding to images of one or more objects that are not part of the scene on which the "perceptual image" is based, or a modified version of a perceptual image subject to one or more of the modifications described above.

[0036] Also, as used herein, the term "image data" generally refers to data items or data objects associated with a given image that maintain image information for that image data. For example, image data may include pixel data and / or information for each pixel of the image, such as color values, intensity values, etc.

[0037] All technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains, unless otherwise defined herein. Although methods and materials similar or equivalent to those described herein can be used in practicing or testing embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the present specification, including definitions, will control. Furthermore, each material, method, and example is merely illustrative and is not intended to be necessarily limiting.

[0038] Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. Referring in particular to the drawings, it is emphasized that the details shown are exemplary and are shown for the purpose of illustratively explaining embodiments of the invention. In this regard, the description using the drawings will make apparent to those skilled in the art how embodiments of the invention may be practiced.

[0039] Turning now to the drawings, in which like reference numbers or letters indicate corresponding or similar elements in each drawing. [Brief explanation of the drawings]

[0040] [Figure 1] FIG. 1 is a schematic diagram of a system having a processing subsystem that converts neural impulses transmitted to a visual processing region of a subject's brain into image data and provides a display image based on the image data to a display device, according to one embodiment of the present invention. [Figure 2] 2 is a block diagram of the processing subsystem of FIG. 1, where the processing subsystem is illustratively shown as including a processing device and a control unit, according to one embodiment of the present invention. [Figure 3] FIG. 3 is a schematic diagram of an exemplary arrangement of the processing subsystem of FIGS. 1 and 2, in which the processing device is interfaced with the visual processing region of a subject's brain by implantation in the optic nerve, according to one embodiment of the present invention. [Figure 4]FIG. 1 is a schematic diagram of an exemplary wired interface according to an embodiment of the present invention, the wired interface including an electrode array that can be used to interface between a processing subsystem and visual processing regions of a subject's brain. [Figure 5] FIG. 1 is a schematic diagram of an exemplary wireless interface that can be used to interface between a processing subsystem and a visual processing region of a subject's brain, showing a transmitting unit connected to a processing device and an electrode array connected to a receiving unit, in accordance with an embodiment of the present invention. [Figure 6] 1 is a schematic diagram of a display image projected by a display device as seen from the viewpoint of a subject viewing the display device, the display image being an augmented image of a real scene seen by the subject by including virtual objects. [Figure 7] 1 is a timing diagram illustrating an alternating sequence of sample periods and display periods corresponding to operation in non-display and display modes of a display device according to one embodiment of the present invention. [Figure 8] A timing diagram similar to FIG. 7 but showing a single timing cycle of operation of the display device in non-display mode and display mode, where the non-display mode sample period is a sub-period of the non-display period corresponding to operation of the display device in the non-display mode. [Figure 9] 1 is a schematic diagram of a network system environment in which a system may operate, according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0041] According to an embodiment of the present invention, a system and method for displaying an image to an object is provided.

[0042] The principles and operation of the systems and methods according to the present invention may be better understood with reference to the drawings accompanying this specification.

[0043] Before describing at least one embodiment of the present invention in detail, it is to be understood that the invention is not necessarily limited in its application to the precise construction and arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0044] Referring now to the drawings, FIG. 1 is a schematic diagram of a system, generally designated 10, according to one embodiment of the present invention. Broadly speaking, system 10 includes a computerized processing subsystem 12 (hereinafter interchangeably referred to as the “processing subsystem”) that interfaces with a visual processing region 43 of a brain 42 of a subject (also referred to as a “user”) 40 via, for example, at least one nerve 46, shown here as a pair of nerves 46. The visual processing region 43 is a region of the brain responsible for visual perception. In the illustrated exemplary embodiment, subject 40 is a human subject, and thus, the visual processing region 43 of brain 42 is the visual cortex. In the illustrated embodiment, processing subsystem 12 is coupled to at least one of optic nerves 46, which are a pair of cranial nerves that serve as a pathway between subject's 40's eyes 44 and brain 42. However, as will become apparent from subsequent sections of this disclosure, it should be noted that embodiments of the present invention are also applicable to non-human animal subjects that have brain regions that perform visual processing and are responsible for visual perception.

[0045] System 10 further includes a display device 60 communicatively coupled to processing subsystem 12 and configured to be positioned in a spatial relationship relative to subject 40. In certain arrangements, display device 60 is positioned facing one or more eyes 44, allowing subject 40 to view display device 60 with one or more eyes 44 without moving their body or head, while in other arrangements, display device 60 is positioned in a non-facing relationship with one or more eyes 44 (e.g., behind subject 40), requiring subject 40 to move their head and / or body to view display device 60. In the non-limiting arrangement shown in FIG. 1 , display device 60 is positioned between a scene (represented here schematically as a star 80) and at least one of the eyes 44 of subject 40. However, it should be noted that other arrangements of the display device 60 are also contemplated herein, including, for example, an arrangement in which the display device 60 is adjacent to the scene 80 (e.g., to the side or behind the subject 40) and the subject 40 must change his or her gaze direction and / or head or body position to switch between viewing the scene 80 and viewing the display device 60.

[0046] Display device 60 can be implemented in a variety of ways, including, but not limited to, a head-up display (HUD), a head-mounted display (HMD) (e.g., a goggle or eyeglass-form factor HMD), a holographic display, a surface (e.g., a wall, table, desk, etc.) mounted electronic display (e.g., a liquid crystal display (LCD) screen, a light-emitting diode (LED) display, a projection display, etc.), etc. It should be noted that a projection display typically includes an image projection device that projects an image onto a suitable background upon which the image can be displayed, which may be an opto-mechanical device such as a cinema projector. Such a background may include, for example, a projection screen consisting of a white and / or reflective surface supported by a support structure, but may also include a wall structure, etc.

[0047] As described below, display device 60 operates to display images viewable by subject 40, and in certain embodiments is configured to switch operation between a display mode in which display device 60 projects (i.e., displays) images for viewing by subject 40, and a non-display mode in which display device 60 does not project images. Operation of display device 60 in the display mode places system 10 in the display mode, and operation of display device 60 in the non-display mode places system 10 in the non-display mode.

[0048] The processing subsystem 12 is configured to perform processing and control functions. FIG. 2 shows a block diagram of the processing subsystem 12 according to a specific, non-limiting embodiment, in which the processing and control functions are functionally separated between a computerized processing device 13a, which performs processing-related functions, and a computerized control unit 13b, which performs control-related functions. In the illustrated, non-limiting embodiment, the processing device 13a and the control unit 13b include one or more processors 14a and one or more processors 14b, respectively. The processors 14a and 14b are coupled to computerized storage media 16a and 16b, respectively. The computerized storage media 16a and 16b may include, for example, computerized memory. Each of the one or more processors 14a and 14b may be implemented as any number of computerized processors, including, but not limited to, a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), and a field-programmable logic array (FPLA). In a microprocessor implementation, the microprocessor may be a conventional processor, such as those used in servers, computers, and other computerized devices. For example, the microprocessor may include x86 processors manufactured by AMD and Intel, Intel Xeon® processors and Pentium® processors, and combinations thereof. It is also contemplated herein that one or more of processors 14 a and 14 b may be implemented as a quantum computer processor. Such a computerized processor may include or be in electronic communication with a computer-readable medium, where the computer-readable medium stores program code or a set of instructions that, when executed by the computerized processor, cause the computerized processor to perform actions.Types of computer-readable media include, but are not limited to, electronic, optical, magnetic, or other storage or transmission devices capable of providing computer-readable instructions to a computerized processor. It should be noted that the above-described implementations of one or more processors 14a and 14b represent a non-exhaustive list of example implementations. Those skilled in the art will recognize that other implementations of processing devices are contemplated herein and that processing technologies not described herein or not yet fully developed, such as biological processors or organic semiconductors in the field of biological computing, may be suitable for implementing any of the processing devices described herein.

[0049] Each storage medium 16a and 16b may be any storage medium and, while typically illustrated as a single component, may each comprise multiple components. Storage media 16a and 16b may be implemented in various ways, such as, for example, one or more volatile or non-volatile memories, flash memories, read-only memories, random access memories, etc., or any combination thereof. In particular embodiments, storage medium 16a may include one or more components for storing and maintaining one or more mapping functions (described further below) and at least one component configured to store machine-executable instructions executable by one or more processors 14a.

[0050] 2 illustrates a functional division in which processing device 13a performs processing-related functions and control unit 13b performs control-related functions, other divisions of processing and control are within the scope of the present invention. For example, in certain embodiments, processor 14a of processing device 13a can perform both processing and control functions, thereby eliminating the need for control unit 13b, and in other embodiments, processor 14b of control unit 13b can perform both processing and control functions, thereby eliminating the need for processing device 13a. In yet other embodiments, processors 14a and 14b may each perform both processing and control functions, and the processing and / or control tasks of processors 14a and 14b may be dynamically changed based on changing processing and control requirements.

[0051] The following paragraphs describe processing-related functions performed by processing subsystem 12. In the non-limiting exemplary embodiment shown in Figure 1, these processing-related functions are performed by processing device 13a, although, as noted above, other divisions of processing are within the scope of the present invention.

[0052] With the above in mind, the processing subsystem 12 operates to receive signals related to nerve impulses that carry image information and are transmitted to a region 43 of the brain 42. The received signals may be the nerve impulses themselves, or may be signals formed (i.e., generated) in response to measuring or sampling nerve impulses by some microdevice, such as a microelectrode or microtransducer, that is in communication with the processing subsystem 12. This process of receiving signals by the processing subsystem 12 is generally referred to herein as "collecting nerve impulses." Nerve impulses are typically transmitted along a pathway from the eye 44 to a region 43 of the brain 42 by nerves 46 in response to the eye 44 viewing a scene (interchangeably referred to herein as one or more visual stimuli (light) presented to the eye 44). As discussed in the Background section, light corresponding to a scene is sensed by photoreceptors in the eye 44, converted into nerve impulses, and the nerve impulses are transmitted by the optic nerve 46 to the brain 42, where they are interpreted by the brain 42 as an image and a visual perception. The interpretation of nerve impulses by the brain 42 is referred to herein as "visual perception" or "perception."

[0053] The processing subsystem 12 further operates to process the received signals (collected nerve impulses) to generate (form) image data (perceptual image) representative of the scene as perceived (by the subject 40). In other words, the generated image data (perceptual image) represents what the subject 40 sees with the eyes 44 when the eyes 44 view (i.e., are exposed to) the scene. The processing performed by the processing subsystem 12 converts the nerve impulses into image data by applying one or more of the mapping functions described above to the signals (collected nerve impulses). The one or more mapping functions contain mapping data and perform a mapping between nerve impulses and image data. That is, the one or more mapping functions provide a conversion from nerve impulses to image data and vice versa, such that the received signals (representing nerve impulses) are transformed (converted) into image data as a result of application of the mapping function by the processing subsystem 12. This nerve impulse-to-image data mapping function is preferably a one-to-one mapping and is hereinafter interchangeably referred to as "impulse-to-image mapping." One-to-one mapping refers to mapping a single nerve impulse signal to a single image data signal, and vice versa. Examples of various methods for generating impulse-image mappings are described in commonly owned U.S. Pat. No. 11,395,620, which is incorporated herein by reference in its entirety. The aforementioned patent also describes examples for storing and maintaining impulse-image mappings, and applying impulse-image mappings to convert nerve impulses to image data. These examples for generating, storing, maintaining, and applying impulse-image mappings are also applicable to embodiments of the present disclosure.

[0054] In certain embodiments, processing subsystem 12 further operates to process received image data representing an image of a scene or object, convert the image data into a sequence of nerve impulses, and provide the nerve impulses to region 43, thereby causing subject 40 to visually perceive the scene or object as if the subject 40 were viewing the scene with his or her eyes 44. Processing subsystem 12 processes the received image data by applying impulse-to-image mapping to the image data (the impulse-to-image mapping is bidirectional, converting nerve impulses to image data and vice versa). In certain embodiments, processing subsystem 12 provides nerve impulses to region 43 via nerve 46 by inducing neurotransmission of the nerve impulses. In certain embodiments, processing subsystem 12 converts the image data into signals (e.g., electrical signals) corresponding to the nerve impulses and transmits the converted signals to a microdevice to provide the nerve impulses to nerve 46. The microdevice may be, for example, one or more microelectrodes or microtransducers implanted within the subject 40 (e.g., in or relative to a portion of a nerve 46 or brain 42) to induce the transmission of nerve impulses corresponding to the transduced signal.

[0055] As described in more detail below, the image data received and processed by the processing subsystem 12 for conversion into nerve impulses may be image data captured by an imaging device (such as a camera) that is electrically connected to the processing subsystem 12, or may be image data obtained from computerized storage (i.e., memory) that is coupled, connected, or otherwise associated with the processing subsystem 12.

[0056] The aforementioned U.S. Patent No. 11,395,620 also describes examples of applying impulse-to-image mapping to image data to convert the image data into neural impulses, which examples may also be applied to embodiments of the present disclosure.

[0057] Continuing to refer to FIG. 1 , the communicative coupling between the processing subsystem 12 and the region 43 can be achieved by a machine-subject interface arrangement 18 (hereinafter interchangeably referred to as “interface”). The machine-subject interface arrangement 18 enables the processing subsystem 12 (and in this example, the processing device 13a) to communicate with the region 43 of the brain 42. In certain embodiments, the interface 18 can include two interface portions: a first interface portion 18a and a second interface portion 18b. The first interface portion 18a (also referred to as the electronics interface portion 18a) is connected to the processing device 13a. The second interface portion 18b, also referred to as the subject interface portion 18b, can be connected or coupled to the region 43 of the brain 42. The two portions 18a, 18b are interconnected via a connecting portion 20, which in certain embodiments can provide a wired connection between the two portions 18a, 18b and in other embodiments can provide a wireless connection between the two portions 18a, 18b.

[0058] Various arrangements for achieving a communicative coupling between the processing subsystem 12 and region 43 are contemplated herein, and some of these arrangements are described in more detail below. The arrangements described herein require some type of surgical implantation, which can be an invasive or semi-invasive approach. For example, an invasive approach can include implantation by surgically accessing the subject's optic nerve and / or optic nerve region through the subject's skull (i.e., surgically opening the skull). Surgery performed on the brain, particularly the visual cortex and optic nerve, has been commonplace for many years, and the necessary implantation procedures could be performed by trained human surgeons and / or robotic surgeons (such as those used by Neuralink Corporation, San Francisco, USA). A semi-invasive approach can include, for example, implantation via the nasal cavity via the sphenoid sinus to access the optic nerve or optic chiasm. Before describing some arrangements, it should be noted that the arrangements described herein are merely exemplary and non-exhaustively represent only a portion of the possible placement options for the processing subsystem 12. Other placement options may be possible, as will be apparent to those skilled in the art.

[0059] In one example arrangement according to certain non-limiting embodiments, the processing subsystem 12 communicates with the optic nerve 46 by branching off from the optic nerve 46 via the interface 18. In such an arrangement, the subject interface portion 18b can be surgically implanted in or relative to a segment (section) of the optic nerve 46, which in certain non-limiting implementations can be achieved by first surgically severing the optic nerve 46 to form cut ends of the optic nerve 46 and then connecting the subject interface portion 18b to those cut ends. In such an arrangement, the processing subsystem 12 preferably remains outside the brain 42 of the subject 40. When the processing subsystem 12 is external to the subject 40, the subject interface portion 18b, along with the entire connecting portion 20 or the segment of the connecting portion 20 connecting to the subject interface portion 18b, is surgically implanted in or relative to the optic nerve 46. When only the segment of the connecting portion 20 connecting to the subject interface portion 18b is surgically implanted, the remaining segment of the connecting portion 20 connecting to the electronics interface portion 18a is external to the subject 40. Preferably, the segment of the optic nerve 46 into which the subject interface portion 18b is surgically implanted is the optic chiasm 48, the portion of the brain 42 where the optic nerves 46 cross over one another.

[0060] In another exemplary arrangement, processing subsystem 12 is located external to the subject, and subject interface portion 18b, along with the entire coupling portion 20 or the segment of coupling portion 20 that connects to subject interface portion 18b, is surgically implanted in or relative to region 43. If only the segment of coupling portion 20 that connects to subject interface portion 18b is surgically implanted, the remaining segment of coupling portion 20 that connects to electronics interface portion 18a is external to subject 40. An example of such an arrangement is shown schematically in FIG.

[0061] In yet another exemplary arrangement according to certain non-limiting embodiments, the processing device 13a itself, along with the entire interface 18, can be implanted in or relative to the region 43. In another exemplary arrangement according to non-limiting embodiments, the processing device 13a is implanted in or relative to a segment of the optic nerve 46. FIG. 3 schematically illustrates such an arrangement, where implantation can be achieved, for example, by first cutting the optic nerve 46 to form cut ends 50a, 50b of the optic nerve 46, then placing the processing device 13a at the cut site and connecting the cut ends 50a, 50b of the optic nerve 46 to the processing device 13a via the interface 18. In such an arrangement, the segment of the optic nerve 46 into which the processing device 13a is implanted is preferably, but not necessarily, the optic chiasm 48, and the optic nerve 46 is surgically cut at the optic chiasm 48 (to form the cut ends 50a, 50b). It should be noted that in embodiments where the processing device 13a or interface 18 is surgically implanted into the optic nerve 46, care must be taken to ensure that the cutting ends 50a, 50b interfacing with the processing device 13a correspond to the same nerve.

[0062] Non-invasive arrangements are also contemplated herein. For example, interface 18 can be provided by an optical magnetic field sensor arrangement or a non-contact modulation arrangement, e.g., using optical, magnetic, or ultrasonic techniques. In such an arrangement, interface 18 (and its associated components) and processing device 13a (and all subcomponents of processing subsystem 12) are entirely external to brain 42. External interface 18 receives brain signals in region 43 via non-contact or non-invasive contact means and provides the received brain signals to processing device 13a.

[0063] It should be noted that the processing subsystem 12 can use a variety of techniques to obtain nerve impulses (and electrical signals representing them) from the target nerves 46, provide nerve impulses (converted from image data) to the nerves 46, and induce transmission of the provided nerve impulses by the nerves 46. Such techniques may typically be through the use of microdevices, such as microelectrodes or microtransducers, to measure (receive) the nerve impulses and form electrical signals in response thereto, and / or stimulate the nerves 46 with electrical signals to induce transmission of corresponding nerve impulses. There has been extensive research, development, and experimentation regarding the connection and interfacing of computer processing devices with the brain, tissue, and nerves through implants or other invasive or semi-invasive means. One example of such research is the University of Luxembourg's 2019 paper "CONNECT-Developing nervous system-on-a-chip" (available at https: / / wwwfr.uni.lu / lcsb / research / developmental_and_cellular_biology / news / connect_developing_nervous_system_on_a_chip), which describes the cultivation of individual nervous system components and their connection within a microfluidic chip (an integrated circuit).

[0064] An example of research and experimentation in the field of brain-machine interfaces is described in a 2011 article in Procedia Computer Science by Stefano Vassanelli of the Neurochip Laboratory at the University of Padua, Italy, entitled "Brain-Chip Interfaces: The Present and The Future." In one example, a computerized processing device is interfaced to neurons using metal microelectrodes or oxide-insulated electrical microtransducers (e.g., electrolyte-oxide-semiconductor field-effect transistors (EOSFETs) or electrolyte-oxide-semiconductor capacitors (EOSCs)) to record (i.e., measure) or stimulate the neurons' electrical activity. In another example, large-scale, high-resolution recordings (i.e., measurements) from individual neurons are obtained using a processing device that employs or is coupled to a microchip featuring a large-scale multi-transistor array (MTA). In yet another example, a microchip featuring a large-scale MTA is used to interface with cells in vitro by placing the MTA in contact with brain tissue, where the signal corresponding to the nerve impulse is, in one example, in the form of a local-field-potential (LFP).

[0065] An example of a brain-machine interface device is the Neuralink device developed by Neuralink Corporation (San Francisco, USA), which contains an ASIC that digitizes information obtained from neurons via microelectrodes.

[0066] With the above in mind, the following paragraphs provide a high-level description of an interface 18 that can be used to connect / interface the processing subsystem 12 to the subject 40 so as to provide a machine-brain interface, in accordance with a non-limiting exemplary embodiment of the present invention.

[0067] With continued reference to FIGS. 1-3, please also refer to FIG. 4, which shows a schematic diagram of interface 18 according to a non-limiting embodiment of the present invention. Here, target interface portion 18b includes electrode array 22 having a plurality of electrodes 23 positioned at or relative to optic nerve 46 (e.g., at or relative to optic chiasm 48). Electrodes 23 are preferably microelectrodes, such as EOSFETs or EOSCs. In embodiments in which processing subsystem 12 operates to convert nerve impulses into image data, electrode array 22 operates to measure nerve impulses transmitted by optic nerve 46, form (in response to the measurements) electrical signals related to (and representative of) the nerve impulses, and provide those signals to processing subsystem 12 (in the illustrated example, the signals are provided to processing device 13a) so that the processing device can collect the nerve impulses and process the electrical signals corresponding to (i.e., representative of) the nerve impulses. In the illustrated embodiment, coupling portion 20 may be implemented as a wire or cable that provides a physical transmission medium along which the electrical signals can propagate to processing device 13a. In certain embodiments, interface 18 may use a transducer (preferably a microtransducer as described above) as part of subject interface portion 18b instead of or in addition to electrode array 22. The transducer may be used in conjunction with processing device 13a to convert nerve impulses into digital image data. For example, the transducer may generate an electrical signal in response to receiving (measuring) nerve impulses transmitted by optic nerve 46. The generated electrical signal corresponds to (i.e., represents) the nerve impulse and is provided to processing device 13a for processing using impulse-to-image mapping.

[0068] In embodiments in which the processing subsystem 12 operates to convert image data into nerve impulses and transmit the nerve impulses via the optic nerve 46 to the brain 42, where the nerve impulses are interpreted by the brain 42 as images / vision, transmission of the nerve impulses may be achieved by stimulating one or more neurons of the optic nerve 46 with a microdevice, such as an electrode array 22 (or transducer). Broadly speaking, in such embodiments, the processing subsystem 12 may convert (using impulse-to-image mapping) the image data into nerve impulses (or electrical signals representing the nerve impulses) that are transmitted by the nerve 46. The processing subsystem 12 then provides the nerve impulses to the nerve 46 to induce neurotransmission of the nerve impulses (or provides electrical impulses to the nerve 46 to induce neurotransmission of the nerve impulses represented by the electrical impulses). In certain embodiments, the induction of nerve transmission can be achieved by the processing subsystem 12 providing electrical signals to the electrode array 22 (or transducer), which stimulates neurons in the optic nerve 46 in accordance with the electrical signals, thereby inducing the transmission of corresponding nerve impulses.

[0069] 5 illustrates another embodiment using wireless signal transmission to provide electrical signals to a microdevice, here represented as an electrode array 22. Here, the processing subsystem 12 is connected to a transmit (Tx) unit 24 via a wire or cable 25, and the electrode array 22 is connected to a receive (Rx) unit 26 via a wire or cable 27. In the illustrated example, the wire or cable 25 is connected to a processing device 13a. The Tx unit 24 includes transmitting circuitry and components for transmitting electrical signals formed by the processing subsystem 12 to the Rx unit 26 via a wireless interface. The Rx unit 26 includes receiving circuitry and components that receive the electrical signals and provide the received signals to the electrode array 22, which stimulates a nerve 46 to induce the nerve 46 to transmit a nerve impulse corresponding to the electrical signal.

[0070] In certain embodiments, the wireless transmission may be RF signal transmission. In such embodiments, the transmitting circuitry and components of the Tx unit 24 may include, for example, one or more antennas, digital-to-analog conversion circuitry, signal modulators, filters, amplifiers, and other signal transmitting electronic circuitry and components, and the receiving circuitry and components of the Rx unit 26 may include, for example, one or more antennas, filters, amplifiers, demodulators, and other signal receiving circuitry and components. In other embodiments, the wireless transmission may be inductive signal transmission, in which case the Tx unit 24 and the Rx unit 26 operate to transmit and receive, respectively, using inductive signal transmission means. In such embodiments, for example, the Tx unit 24 may include an inductive coil, and the Rx unit 26 may include an inductive receiver.

[0071] As mentioned above, in certain embodiments, interface 18 may provide non-contact or non-invasive contact between processing subsystem 12 and region 43. For example, interface 18 may include, for example, an optical-based magnetic field sensor arrangement or a non-contact modulation arrangement using, for example, optical, magnetic, magnetic resonance imaging (MRI), or ultrasound techniques.

[0072] In certain embodiments, where processing device 13a is implemented as a biological processor or biological processing element cultured or grown within a subject, interface 18 is the processing device 13a itself.

[0073] It should be noted that in certain embodiments, interface configuration 18 can include multiple interfaces. For example, a first interface can be used to achieve conversion of image data to neural impulses. The first interface can use an electrode array 22 or a microtransducer (e.g., implemented as an EOSC) connected or coupled to processing subsystem 12 (e.g., processing device 13a) via a wired connection (e.g., as shown in FIG. 4) or a wireless connection (e.g., as shown in FIG. 5). A second interface can be used to achieve conversion of neural impulses to image data. The second interface can use an electrode array 22 and / or a microtransducer (e.g., implemented as an EOSFET) connected or coupled to processing subsystem 12 (e.g., processing device 13a) via a wired connection (e.g., as shown in FIG. 4). In other embodiments, the second interface can use non-contact or non-invasive contact means (e.g., an optical magnetic field sensor configuration or a non-contact modulation configuration).

[0074] As explained above, in certain embodiments, processing subsystem 12 further operates to convert image data representing a scene or object into a sequence of neural impulses and then provide the neural impulses to region 43, thereby causing subject 40 to visually perceive the scene or object as if the subject 40 were viewing the scene with his or her eyes 44. However, implementing such embodiments can be technically challenging and invasive, as it may require some form of implantation into subject 40, e.g., implantation of processing device 13 a and / or implantation of a microdevice (which receives neural impulses or electrical signals representative thereof from processing subsystem 12 and induces transmission of neural impulses by nerves 46 connected to the microdevice). Accordingly, a particular feature of certain embodiments of the present disclosure is providing a non-invasive alternative for displaying images to a subject by using a display device 60 that operates to selectively (and controllably) display images to the subject based on images generated from the subject's viewing of the scene. According to such an embodiment, processing subsystem 12 (e.g., processing device 13a) operates to receive, during a sample period concurrent with display device 60 operating in non-display mode, signals related to neural impulses transmitted by one or more nerves 46 in response to one or more eyes 44 viewing scene 80. Processing subsystem 12 processes the received signals by applying impulse-to-image mapping, thereby generating a perceived image (i.e., converting neural impulses into image data for the perceived image), where the perceived image represents the visual perception of scene 80 by region 43 of brain 42. Processing subsystem 12 further operates to provide, during a display period concurrent with display device 60 operating in display mode, an image for display (referred to herein as a “display image”) to display device 60 based on the generated perceived image. Thus, during the display period, subject 40 can view the display image displayed by display device 60. As described below, the display image may include elements of the perceived image.

[0075] In one example, the displayed image is the perceived image, i.e., the displayed image is an unaltered version of the perceived image, which may be useful in various situations, such as, for example, calibrating and / or testing system 10.

[0076] In another example, the displayed image is a modified version of the perceived image. For example, as described in previous sections herein, the displayed image may be a perceived image that has been modified to change one or more image parameters or characteristics of one or more image elements (such as pixels), such as one or more of color, brightness, contrast, and size, and / or that has been modified to crop the perceived image (e.g., by removing pixel data). The displayed image may also be a perceived image (either the raw perceived image or a modified version of the perceived image as described above) that has been extended to include an image of an object so that the object appears superimposed on the scene in the displayed image.

[0077] The modification of the perceived image may be performed by processing subsystem 12, for example, by adding and / or deleting and / or changing elements (e.g., pixel information) within the image data of the perceived image. Further details of how processing subsystem 12 can modify the perceived image and the methods by which the perceived image can be modified by processing subsystem 12 are described in the aforementioned U.S. Patent No. 11,395,620.

[0078] In this disclosure, when subject 40 views a display image, which is a perceived image that has been altered in some way, e.g., by processing subsystem 12, to include an image representing the object, the object seen in the display image is referred to as a “virtual object.” The term “virtual object” is used herein because the object is virtual in the sense that it is not part of the scene viewed by the subject during the preceding sample period (and therefore is not part of the scene on which the perceived image is based); therefore, from the subject's perspective, the object is a “virtual” part of the display image. This virtual object may be based on any real-world object, such as, for example, a computer input device such as a computer keyboard or a computer mouse; an interactive display screen; a heads-up display (HUD) such as a heads-up display (HUD) in an automobile or aircraft cockpit; an electronic device such as a smartphone, a tablet, a digital radio receiver, a speaker system; or a home appliance such as an oven, microwave, dishwasher, or laundry appliance (e.g., a washing machine, a dryer). For example, image data describing / representing one of these aforementioned real-world objects may be combined with the perceived image, e.g., by processing subsystem 12, to form the display image. As described below, such image data can be obtained from a variety of sources, including, but not limited to, memory devices and imaging devices. For example, such image data may be stored in a device, such as a memory, associated with processing subsystem 12. As another example, a device, such as an imaging device (e.g., a camera), associated with processing subsystem 12 may capture one or more images of a real-world object to generate such image data.

[0079] 6 shows an example of a display image 90 projected by the display device 60 during a display period, as seen from the perspective of a subject 40 viewing the display device 60. Here, the display image 90 is a perceived image (an image of the scene 80) that has been extended to include a virtual object (here, represented schematically as a shaded four-pointed star 92). As a result, when the subject 40 views the display image 90, the subject 40 sees the virtual object 92 superimposed on the scene 80.

[0080] 6 also shows a portion (schematically represented as a freeform shape 94) that provides a bounding area surrounding part or all of object 92. Details of portion 94 are described in more detail below in connection with other aspects of the present disclosure that support interaction with virtual objects and / or manipulation of real-world objects associated with the virtual objects. Such aspects are particularly useful in augmented reality type applications.

[0081] In particular embodiments, processing subsystem 12 operates to controllably and repeatedly switch display device 60 between a display mode, in which display device 60 projects (i.e., displays) display images, and a non-display mode, in which display device 60 does not display images, such that display device 60 alternates between projecting display images and not projecting images to subject 40. In other words, by switching display device 60 between the display mode and the non-display mode, processing subsystem 12 effectively switches display device 60 between an on state and an off state, where in the on state (display mode), display device 60 projects one or more display images, and in the off state (non-display mode), display device 60 does not project images, thereby allowing the subject to view a real-world scene.

[0082] 1, the controlled switching of display device 60 is performed by control unit 13b. Furthermore, the control unit preferably also manages the timing associated with switching between display and non-display modes, as well as various period-related parameters, such as the length of the sample period and the length of the display period. However, as explained above, it should be noted that other divisions of control and timing functions are within the scope of the present invention.

[0083] In certain embodiments, for example, when display device 60 is implemented as a surface-mounted display screen or in an eyeglass form factor that occupies a small portion of an eyeglass lens, display device 60 may permanently obscure a small portion of a subject's field of view (FOV) of scene 80 when positioned between scene 80 and one or more eyes 44. In such embodiments, display device 60 obscures (i.e., blocks) a small portion of the FOV regardless of the operational state of display device 60. Preferably, the obscured portion is small enough that one or more eyes 44 of subject 40 can still faithfully form an image of scene 80. Note that obscuring the scene when display device 60 is operating in a display mode can be important to prevent subject 40 from simultaneously viewing the real scene and the displayed image, thereby avoiding visual confusion for that portion of subject 40.

[0084] It should be noted that in embodiments in which display device 60 is implemented as a projection display, the background onto which the image projection device projects an image can be positioned in any suitable spatial location relative to the subject, as long as the image projection device is capable of projecting an image onto the background. For example, in one arrangement, the background can be positioned adjacent to the scene (e.g., beside or behind the subject), requiring the subject to change their gaze direction and / or head or body position to change their view between the scene and the background. In another arrangement, the background can be positioned facing one or more eyes of the subject. In such an arrangement, the background can be interposed between the scene and the subject's one or more eyes, such that the background preferably partially covers a small portion of the subject's FOV relative to the scene. According to certain embodiments, when the background is positioned such that it is interposed between the scene and the subject's one or more eyes, the background itself can also be incorporated as part of the scene and thereby become part of the perceived image. In such embodiments, processing subsystem 12 can modify the perceived image to remove or adjust the opacity level of elements of the perceived image that correspond to the background.

[0085] In other non-limiting embodiments, display device 60 is an at least partially transparent display such that some or all of scene 80 (i.e., at least a portion) is at least partially viewable by one or more eyes 44 through display device 60 when display device 60 is operating in a non-display mode. Various types of partially transparent displays (i.e., “see-through displays”) are known in the art. Such displays employ various display technologies, such as holographic displays, LCD panels that use twisted nematic liquid crystals with crossed polarizers to achieve transparency, and LED displays that use layers of glass on both sides of addressable LEDs. In certain embodiments, display device 60 can be implemented as a HUD or HMD, for example, as goggles or in an eyeglass form factor. Such HUD and HMD implementations can advantageously employ light guides or substrates that make the HUD and HMD implementations partially transparent (see-through) or non-transparent (non-see-through). Various light guide technologies that provide see-through and non-see-through displays are known in the art, including, for example, HoloLens from Microsoft Corporation (Redmond, WA) and lightguide optical element (LOE) from Lumus Ltd. of Israel.

[0086] In certain embodiments, the display device 60 may overlay the scene 80 only when the display device 60 is operating in a display mode. Overlaying the display device 60 over the scene 80 can be achieved in various ways. In one non-limiting example, overlaying the display device 60 over the scene 80 can be achieved using mechanical movement of the display device 60, such as by disposing a mechanical arm or mount in a mechanical drive relationship with the display device 60, which moves the display device 60 in and out of the optical path between the scene and one or more eyes in synchronization with the display and non-display modes of operation of the display device. In another non-limiting example, overlaying the display device 60 over the scene 80 can be achieved by disposing a movable shutter or baffle that moves in and out of the optical path between the display device and the scene in synchronization with the display and non-display modes of operation of the display device. In yet another non-limiting example, a switchable polarizing filter can be placed between the display device and the scene. The polarizing filters may be operable to a first state that blocks incident light from a scene when the display device 60 is operating in a display mode, and to a second state that allows incident light from a scene to pass through the display device to one or more eyes 44 when the display device 60 is operating in a non-display mode. In all of the foregoing examples, control of the mechanical movement and / or state switching may be provided by the processing subsystem 12.

[0087] In certain embodiments, the sample and display periods may be non-overlapping and adjacent periods, such as shown in the timing diagram of FIG. 7, such that a sample period begins when a display period ends, and vice versa. Alternatively, there may be a small delay between the end of one period (e.g., a sample period) and the beginning of the next period (e.g., a display period). In certain embodiments, operation of display device 60 in the non-display mode is coincident with a non-display period that includes at least one sample period as a sub-period (or sub-interval) of the non-display period. FIG. 8 shows a timing diagram of one cycle of non-display and display mode operation of display device 60 in such an embodiment. Here, when processing subsystem 12 switches display device 60 to the non-display mode, one or more eyes 44 capture light from a scene, thereby transmitting neural impulses carrying image information of the scene to region 43 of brain 42 during the non-display period. The neural impulses corresponding to the light captured by one or more eyes 44 during the sample period sub-interval of the non-display period are converted into image data by processing subsystem 12 (by applying impulse-to-image mapping). In other words, the processing subsystem 12 transforms only selected samples of nerve impulses transmitted during non-display periods.

[0088] Preferably, the sample period is relatively short compared to the display period. For example, the display period may be an order of magnitude or more longer than the sample period. For example, the sample period may be on the order of a few milliseconds, while the display period may be on the order of a few seconds. This discrepancy in length may be due to the fact that in the human visual system, when a series of images is captured by the eye, the brain typically ignores individual images that do not fit into the series. For example, research has shown that the average human brain takes only 13 milliseconds to process neural impulses corresponding to snapshots of a scene and form a picture of the entire scene (https: / / news.mit.edu / 2014 / in-the-blink-of-an-eye-0116). Therefore, the brain may discard or ignore one or more snapshots that deviate from the series of snapshots. However, as described below, there may be situations in which the sample period and the display period are the same length or nearly equal. There may also be situations in which the sample period is shorter than the display period.

[0089] Thus, according to an embodiment of the present invention, processing subsystem 12 (e.g., control unit 13b) switches display device 60 to a non-display mode to allow the eyes to view (i.e., capture an image of) scene 80 during a relatively short sample period, and then switches display device 60 to a display mode to project a display image generated based on viewing the scene during the previous sample period. As explained above, the display image is generated by processing subsystem 12 by converting nerve impulses transmitted by nerves 46 to region 43 in response to one or more eyes 44 viewing scene 80 during the sample period into image data (i.e., a perceived image). This image data (i.e., a perceived image) may optionally be temporarily stored in memory (e.g., storage media 16a and / or 16b, or any other data storage medium associated with processing subsystem 12). This temporary storage allows processing subsystem 12 to collect all image data needed to construct a perceived image of the scene (thereby providing a sort of buffer). Processing subsystem 12 can then perform operations on the image data (i.e., the perceived image) by modifying the image, for example, by changing pixel values ​​in the image data by combining the perceived image with the image of the object. The image of the object can be obtained from any suitable electronic device, such as, for example, a camera / imaging device associated with processing subsystem 12 and memory associated with processing subsystem 12 (e.g., storage media 16a and / or 16b). In certain embodiments, the image of the object can be a brain-generated image, such as an imagined image, which is stored in memory associated with processing subsystem 12. Examples of techniques for forming brain-generated images, e.g., imagined images, are described in commonly owned U.S. patent application Ser. No. 17 / 863,480, filed July 13, 2022, the entire disclosure of which is incorporated herein by reference.

[0090] It should also be noted that in certain preferred embodiments, the sample period coincides with the non-blink period so that nerve impulses collected or transmitted during a blink are excluded from the nerve impulses to be converted into image data to form the perceived image. In certain embodiments, the sample period may be a discontinuous period with a gap or interruption corresponding to the length of a blink period. In other embodiments, a guard interval may be provided around the sample period (or multiple sample periods) to ensure that nerve impulses associated with a blink are not converted by the processing subsystem 12. In one group of non-limiting embodiments, the detection of blinks and associated blink periods is performed by a detection device, such as any suitable detection device known in the art. Non-limiting examples of devices capable of detecting blinks and associated blink periods include an eye tracker that detects movement of the eye and surrounding muscles, and a blink sensor that uses an infrared sensor to determine eye closure. In another non-limiting embodiment, the processing subsystem 12 performs blink detection by processing the collected nerve impulses. Processing subsystem 12 may process the collected nerve impulses and detect blinks by utilizing the fact that nerve impulses generated and transmitted during a blink correspond to dark or blank images. Thus, for example, processing subsystem 12 may determine which of one or more collected nerve impulses correspond to nerve transmissions during a blink by converting the collected nerve impulses into image data and analyzing the image data generated from the collected nerve impulses to identify image data representing black or blank images. Once processing subsystem 12 identifies nerve impulses that generate image data representing black or blank images, processing subsystem 12 may flag or mark those collected nerve impulses and may avoid using those flagged nerve impulses when generating a perceived image.

[0091] Once processing subsystem 12 completes combining the perceived image with the image of the object, processing subsystem 12 provides the resulting display image to display device 60 (i.e., processing subsystem 12 sends image data for the display image to display device 60) and controls display device 60 to display (i.e., project) the display image during a display period (i.e., processing subsystem 12 switches display device 60 to display mode). In certain embodiments, such as embodiments in which display device 60 naturally occupies a large portion of the subject's FOV, it is preferable that when display device 60 is switched to display mode, display device 60 obscure the subject's field of view of the scene so that the subject cannot view the scene while viewing the display image on display device 60.

[0092] In particular embodiments, processing subsystem 12 may repeatedly switch display device 60 between display mode and non-display mode, where each time display device 60 is switched to non-display mode, subject 40 views scene 80 during a corresponding sample period, and processing subsystem 12 generates a perceptual image of the scene based on neural impulses collected during the sample period. When processing subsystem 12 switches display device 60 to display mode at the end of a sample period (i.e., when the next display period begins), processing subsystem 12 provides the display device with a display image that is based on the perceptual image generated during the previous sample period. In this way, the scene portion of the display image is updated over time to account for incremental changes in the scene perceived by one or more eyes 44 of subject 40.

[0093] It should be noted that converting nerve impulses to image data can be a power-intensive and / or computationally intensive task. Therefore, it can be advantageous to optimize (or nearly optimize) some of the control and timing parameters to reduce the power consumption and number of computational resources of the processing subsystem 12. The following paragraphs describe such control and timing optimization.

[0094] As discussed above, the processing subsystem 12 can be configured to store image data converted from neural impulses (in response to one or more eyes 44 viewing the scene during a sample period) in memory associated with the processing subsystem 12 (e.g., storage medium 16a and / or 16b). According to certain embodiments, the processing subsystem 12 is configured to switch the display device 60 to display mode once a sufficient amount of image data corresponding to a “complete” image of the scene has been converted from the neural impulses and stored in memory. The sufficiency criteria for a “complete” image can be parameterized; for example, the processing subsystem 12 can compare image density against a threshold criterion to determine whether a “complete” image has been generated. In certain embodiments, the processing subsystem 12 switches the display device 60 to display mode as soon as image data for a “complete” perceived image of the scene has been generated from the neural impulses. In certain embodiments, such as embodiments in which the processing subsystem 12 is configured to modify the perceived image to include virtual objects, the processing subsystem 12 may delay or buffer transmission of the displayed image data (e.g., the perceived image augmented to include virtual objects) to the display device 60 to account for the processing time required to generate the displayed image. For example, processing subsystem 12 may switch display device 60 to display mode as soon as processing subsystem 12 completes generating image data for a "complete" perceptual image, but processing subsystem 12 may not immediately send the perceptual image to display device 60, but instead may continue to process the perceptual image to augment the perceptual image with virtual objects, and only then send the display image (in this case the augmented perceptual image) to display device 60 for projection onto a subject.

[0095] The length of the non-display period can be set by the processing subsystem 12 so that each time the processing subsystem 12 switches the display device 60 to the non-display mode, the scene is not obscured by the display device 60 to the subject 40 for a period of time that allows the brain to process nerve impulses transmitted in response to one or more eyes 44 collecting light from the scene during the non-display period. For example, if the non-display period is set to 150 milliseconds and it takes the brain 42 13 milliseconds to process the nerve impulses to form an entire / complete scene image, the brain can form approximately 11 scene images during the non-display period. The processing subsystem 12 may select a nerve impulse corresponding to one of these formed scene images for processing and convert the selected nerve impulse into image data. The selection of nerve impulses to process can be performed by defining the length of the sample period, for example, by setting the sample period to be at least the time it takes the brain to form an image. In certain preferred embodiments, the sample period can include a buffer or margin to account for processing delays and / or minor disruptions on the nerves 46. For example, the processing subsystem 12 may define the sample period to be approximately 15 milliseconds, which is based on the 13 milliseconds it takes an average brain to form a complete scene image, plus a buffer of approximately 2 milliseconds.

[0096] In certain embodiments, processing subsystem 12 may impose a time limit for the completion of generation of image data for a “complete” sensory image. For example, if, at the end of an allowed time limit, processing subsystem 12 has not transformed received neural impulses sufficiently to form a complete sensory image, processing subsystem 12 may instead use a stored sensory image, e.g., from a previous (preferably immediately preceding) operating cycle of display device 60 in non-display mode. The stored sensory image may be stored in any memory associated with processing subsystem 12 (e.g., storage medium 16a and / or 16b) and may be retrieved from that memory by processing subsystem 12. Processing subsystem 12 may then use the retrieved sensory image to form a display image, e.g., by augmenting the retrieved sensory image to include virtual objects.

[0097] In particular embodiments, processing subsystem 12 may set the length of the display period so that the same perceptual image (e.g., generated from neural impulses collected during the previous or immediately preceding sample period) is used as the basis for the displayed image projected by display device 60 during the display period. The length of the display period may be set, for example, according to an optimization strategy to ensure that the underlying perceptual image is not a stale image, i.e., the perceived image still represents, to some extent, the current scene.

[0098] It should be noted that the timing and control parameters of system 10 can be case-specific and / or user-specific. For example, in a vehicular head-up display environment, such as in an aircraft cockpit or automobile driver's seat, processing subsystem 12 preferably switches between operational modes of display device 60 relatively quickly to account for rapidly changing scenes, thereby enabling a subject to seamlessly perceive the changing real scene and one or more virtual objects overlaid thereon. In such cases, the display period is preferably the same as or slightly longer (e.g., a small integer multiple longer) than the sample period. For example, the sample period may constitute the entire non-display period and be 15 milliseconds, or the display period may be longer than 15 milliseconds, e.g., up to 60 milliseconds. In such an example, the one or more virtual objects may include, for example, vehicle speed, vehicle altitude, vehicle fuel information (e.g., a fuel gauge), and navigation information (e.g., a compass, heading, etc.).

[0099] As another example, if the system is deployed in a situation where the scene changes slowly over time, e.g., as part of an office workstation where display device 60 is implemented as a wall- or table / desk-mounted electronic display such as an LCD screen, the display period may be significantly longer than the sample period. For example, in such an office workstation deployment, the sample period may be 15 milliseconds, a fraction of a 100 millisecond hide period, and the display period may be hundreds of milliseconds, or even a few seconds or minutes (e.g., depending on the rate at which the scene changes). In such an example, the one or more virtual objects may include, for example, objects related to or associated with the workstation or the person working at the workstation, such as a clock, a list of tasks for the person working at the workstation (i.e., a "to-do list"), reminders, notifications, and computer files.

[0100] All of the aforementioned timing and control parameters may be automatically adjusted, e.g., optimized, by processing subsystem 12 based on input from subject 40, or may be manually adjusted by subject 40. In certain embodiments, as mentioned in previous sections of this disclosure, it may be advantageous to configure processing subsystem 12 to provide the perceived image to display device 60 as a display image (i.e., not augmented with virtual objects) to enable calibration and / or testing and / or optimization of system 10, such as calibration and / or optimization of non-display periods, sample periods, and display periods. In certain embodiments, if the current timing and / or control parameters do not allow processing subsystem 12 to collect all of the image data necessary to construct a perceived image of the scene, processing subsystem 12 may provide a warning or notification to the subject indicating the inability to collect. In certain embodiments, in addition to providing such a warning or notification, processing subsystem 12 may keep display device 60 in a non-display mode or may operate display device 60 to display only the virtual image until an input command is received from subject 40. In other embodiments, processing subsystem 12 may automatically adjust timing and / or control parameters to ensure a complete perceptual image is constructed.

[0101] In certain embodiments, display device 60 is positioned to simultaneously display images to both eyes of subject 40. However, other embodiments are contemplated herein in which two display devices are provided such that a pair of display devices exists, with each display device projecting an image to a corresponding eye of the subject. For example, a system may include a first (left-eye) display device that projects images seen only by the subject's left eye and a second (right-eye) display device that projects images seen only by the subject's right eye. Such an embodiment is particularly suitable when the display devices are implemented in a goggle or eyeglass form factor, where the left-eye display device and the right-eye display device can be easily positioned in a spatial relationship to the corresponding left and right eyes of the subject. Note that in such a dual display device configuration, processing subsystem 12 may independently operate the display devices to switch between display and non-display modes. Furthermore, the display and non-display modes of the two display devices may be fully contemporaneous with one another or may be distinct and / or non-overlapping. For example, the first display device may be configured to operate in a display mode when the second display device is operating in a non-display mode, and may be further configured to operate in a non-display mode when the second display device is operating in a display mode, i.e., in this example, when one display device is operating in a display mode, the other display device is operating in a non-display mode.

[0102] It should be noted that, in general, the provision of two display devices as described above can be extended to any number of displays and a corresponding number of eyes, particularly for more than two eyes. This can be particularly useful when the system is configured to be deployed and used by animal subjects with more than two eyes, such as certain invertebrate species. It should be further noted that the display device for a given eye may include a set of displays that operate together as a single display.

[0103] It should be further noted that the processing subsystem 12 may be further configured to operate in a standby mode or bypass mode for a predetermined period of time, during which the processing subsystem 12 does not convert collected nerve impulses into image data. Furthermore, one or more display devices may be further configured to operate in a conventional display mode, in which the display device projects any suitable type of image other than the display image described above, such as an image provided from a memory or an external device such as an imaging device (e.g., a camera). For example, the processing subsystem 12 may cause the display device 60 to intermittently operate in the conventional display mode while switching between the display mode and the non-display mode. For example, the processing subsystem 12 may switch the display device to the display mode to project a display image, then switch the display device to the conventional display mode so that the display device projects an image that is not the display image, and then switch the display device to the non-display mode so that the display device does not project an image. Obviously, the above-described operation in the conventional display mode is also applicable to dual display device configurations.

[0104] As mentioned above, there may be several situations in which it is useful to provide a display image that is a modified version of a perceived image that has been extended to include one or more virtual objects. One particularly useful set of scenarios is when interaction with a virtual object (e.g., by a subject) and manipulation of a real-world object associated with the virtual object (e.g., initiated by the subject or by an artificial intelligence (AI) platform) are desired. The following paragraphs describe embodiments that support interaction with virtual objects and manipulation of real-world objects. In such embodiments, processing subsystem 12 operates to identify data records (e.g., having metadata) associated with one or more elements of image data for a display image, where the display image is based on the perceived image, as described above, and thus derived from neural impulses transmitted in response to one or more eyes viewing a real-world scene. These one or more elements of image data associated with the identified data records are also associated with the spatial location of a portion of the display image that includes at least a portion of the virtual object. The portion of the display image that includes at least a portion of the object provides a bounding region that surrounds some or all of the object.

[0105] Referring again to Figure 6, this figure illustrates a non-limiting example in which a portion (schematically represented as a freeform shape 94) completely surrounds a virtual object 92. However, it should be noted that in some cases, the portion 94 may not completely surround the virtual object 92. For example, the portion 94 may only surround one or more portions of the virtual object 92, such that one or more portions of the virtual object 94 lie outside the bounding region.

[0106] The spatial location can be any location in space within portion 94 or sufficiently close to (i.e., near) portion 94, although it may be preferable for the spatial location to be a location within portion 94. In certain embodiments, the spatial location can be any location in two-dimensional space within or near portion 94. As such, such a spatial location can be defined by a pair of spatial coordinates, e.g., an x-coordinate and a y-coordinate corresponding to the horizontal and vertical dimensions, respectively, of displayed image 90. In other embodiments, the spatial location can be any location in three-dimensional space within or near portion 94. As such, such a spatial location can be defined by three spatial coordinates, e.g., x-, y-, and z-coordinates corresponding to the horizontal, vertical, and depth dimensions, respectively, of displayed image 90 as seen by subject 40.

[0107] The data record, optionally together with other similarly structured data records, may form a set of data records and may be stored in a storage medium (e.g., memory or database, etc.) associated with system 10, such as storage medium 16a and / or 16b. Each data record in the set of stored data records may store various attributes associated with image data for a displayed image, and in particular attributes associated with components of the image data related to a virtual object. As described further below, each data record may also store other attributes, such as the spatial coordinates (in two-dimensional or three-dimensional space) of the virtual object, as well as actions associated with the virtual object.

[0108] Processing subsystem 12 operates, for example, to identify, from a set of data records, a data record that is associated with (i.e., corresponds to) the spatial location of a portion (e.g., portion 94) of display image 90 that includes at least a portion of virtual object 92. Processing subsystem 12 may identify the data record based in part on image data for the display image and metadata in the data record associated with the image data. This metadata may include, for example, the spatial location (coordinates) of the virtual object within the display image, which may be stored as an attribute in the data record.

[0109] The identification of the data record by the processing subsystem 12 may be in response to a detected event associated with the portion 94 of the display image 90 that includes at least a portion of the virtual object 92. In particular embodiments, the detected event may be detected by an algorithm, e.g., an AI algorithm, executed by one of the processors of the processing subsystem 12. For example, the detected event may be a general event, such as the mere presence of a virtual object in the display image or the location of a virtual object at a particular location in the display image. In particular embodiments, the detected event is a detected action, which, without loss of generality, may be a subject-initiated action (i.e., subject 40 initiating an action) with respect to the portion 94 of the display image 90 that includes at least a portion of the virtual object 92.

[0110] As used herein, the term “subject-initiated action” refers to any action whose origin can be traced back to a subject. Examples of subject-initiated actions may include, but are not limited to, gesturing (performed with the subject's hand or other part of the subject's body, or with a real object held or manipulated by the subject) at a portion 94 of the displayed image 90 that includes at least a portion of the virtual object 92, pointing (performed with a part of the subject's body, or with a real object held or manipulated by the subject, such as a stick or laser pointer) at or to a portion 94 of the displayed image 90 that includes at least a portion of the virtual object 92, and voice-activated commands related to features of the virtual object 92.

[0111] Detection of subject-initiated actions can be performed by processing subsystem 12 or components associated with processing subsystem 12. By way of a non-limiting example, processing subsystem 12 may detect such actions by processing collected nerve impulses transmitted in response to the subject viewing the occurring action. For example, if an action is initiated by subject 40 pointing at portion 94 with a pointing object (e.g., a body part of the subject or a pointer device), processing subsystem 12 may receive signals (collected nerve impulses) transmitted in response to subject 40 viewing (with one or more eyes 44) portion 94 being pointed at by the pointing object. Processing subsystem 12 may then process the received signals (by applying impulse-to-image mapping) to transform the signals into a new image (i.e., generate new image data). Processing subsystem 12 may then compare elements of the image data of the new image with elements of image data of display image 90 corresponding to the virtual object to determine whether the pointing action is at a spatial location that includes at least a portion of virtual object 92. For example, processing subsystem 12 may compare each pixel of image data of the new image with each pixel of image data of display image 90 to determine whether the two pixels (one pixel from the new image and one pixel from display image 90) are at the same or nearby coordinates. For example, if processing subsystem 12 determines that one or more pixels of the new image are at the same or nearby coordinates as one or more pixels of display image 90 that correspond to a virtual object, processing subsystem 12 may make a determination that there is spatial overlap between the identified pixels of the new image and the pixels of the virtual object in display image 90, and may then further determine that a target-initiated action has been performed.

[0112] In another example, detection of a subject-initiated action can be performed in conjunction with an eye-tracking subsystem (i.e., an "eye tracker"). An eye tracker can be implemented as one or more cameras or any other sensor device capable of detecting and tracking gaze and eye movements and / or specific eye-related characteristics, such as blinking, all of which are well known in the art. An exemplary embodiment utilizing an eye tracker is shown in FIG. 1 . Here, eye tracker 70 can optionally be disposed in association with processing subsystem 12 to detect the gaze direction of one or more eyes 44, and processing subsystem 12 can use the detected gaze direction to detect a subject-initiated action. For example, subject 40 may rapidly shift their gaze a certain number of times between virtual object 92 (appearing overlaid on the scene within the display image) and other portions of display image 90, which can indicate a subject-initiated action. The processing subsystem 12 may be configured to compare the number of detected gaze shifts (detected by the eye tracker 70) with a gaze shift threshold (i.e., a preset value) and may detect a subject-initiated action if the number of detected gaze shifts is within a margin (which may be zero) of the threshold. The gaze shift threshold and margin may be pre-programmed into the processing subsystem 12.

[0113] As another example, eye tracker 70 may detect blinks and other blink-related features (such as blink duration and blink intervals), and the subject may initiate an action by blinking according to a set number of times, and possibly according to a set interval between the number of blinks and / or blink duration. For example, processing subsystem 12 may compare the number of detected blinks to a blink count threshold, and / or another detected blink interval to a blink interval threshold, and / or compare the duration of one or more detected blinks to a blink duration threshold. The processing subsystem 12 may detect a subject-initiated action when the detected number of blinks is within a margin (which may be zero) of a blink-number threshold, and / or when the detected interval between blinks is within a margin (which may be zero) of a blink-interval threshold, and / or when the length of one or more detected blinks is within a margin (which may be zero) of a blink-length threshold. The blink-number threshold, the blink-interval threshold, the blink-length threshold, and the associated margins may all be pre-programmed into the processing subsystem 12. As mentioned in a previous section herein, the eye tracker 70 can advantageously be used to provide blink information to the processing subsystem 12. The processing subsystem 12 can use this blink information to adjust timing and / or control parameters, particularly the sample period and / or the non-display period, to ensure that neural impulses transmitted during the blink period are not used to form a perceived image.

[0114] In yet another example, the display device 60 may be implemented as a touchscreen display device configured to detect touches in response to touch input received from the subject 40 via one or more fingers or a suitable input object, such as a stylus device. Here, the subject-initiated action can be detected by the display device 60 and relayed to the processing subsystem 12. Touchscreen display devices, and methods and techniques for detecting touch actions initiated on touchscreen display devices, are well known in the art but will be briefly described here. Some touchscreen display devices use capacitive technology, employing capacitive touchscreens constructed from multiple layers of glass and plastic and coated with a conductive material, such as indium tin oxide or copper. This conductive material reacts when contacted by another electrical conductor, such as a finger or a stylus device. When an electrical conductor touches the display screen, an electrical circuit is completed at the point of contact, changing the charge at that location and detecting a "touch." Other touchscreen display devices use resistive technology, employing resistive touchscreens. In this case, a protective layer of glass or hard plastic is covered with a resistive metal layer that conducts an electric charge. The protective and metal layers are separated by a spacer, and when the protective layer is pressed firmly (e.g., with a finger), the two layers come into contact, changing the charge at that location, thereby providing "touch" detection.

[0115] When a touchscreen display is used, processing subsystem 12 receives a "touch" detection and identifies a data record associated with the location of the "touch" detection, which is a spatial location of the displayed image containing the virtual object.

[0116] It will be apparent to those skilled in the art that the above example for detecting a subject-initiated action is merely one example of how such an action may be detected, and that other methods / techniques may be employed by system 10 or components associated with system 10.

[0117] In response to detecting a subject-initiated action relative to portion 94 of display image 90 that includes at least a portion of virtual object 92, processing subsystem 12 identifies (e.g., in a database) the correct data record associated with one or more elements of image data of display image 90 that are associated with the spatial location of that portion of the display image that includes at least a portion of the virtual object. As a result, when subject 40 initiates an action relative to portion 94, one or more image data elements of the display image that correspond to portion 94 are identified by processing subsystem 12, and data records associated with the identified one or more image data elements are also identified by processing subsystem 12. Processing subsystem 12 may identify the correct data record based on the spatial location (coordinates) of the detected subject-initiated action, for example, by identifying a data record that includes the spatial location based on spatial location metadata in one or more data records.

[0118] Once the processing subsystem 12 identifies the correct data record, attributes of or associated with the image data are also obtained / identified by the processing subsystem 12. The attributes may include, for example, one or more response actions that the processing subsystem 12 can initiate in response to identifying the data record, the spatial location of the virtual object, the two-dimensional or three-dimensional spatial location / coordinates of the portion 94 of the displayed image that includes the virtual object, a property or characteristic of the virtual object, a feature or parameter of the displayed image, such as the size (e.g., in pixels) of the displayed image and / or the size (e.g., in pixels) of the virtual object in the displayed image and / or the ratio of the size of the virtual object in the displayed image to the size of the displayed image (or vice versa), etc.

[0119] According to certain embodiments, the location of a virtual object relative to scene 80 in display image 90 can be arbitrary and, for example, can be selected ad hoc by processing subsystem 12 when processing subsystem 12 generates display image 90. In other embodiments, the location of a virtual object relative to scene 80 in display image 90 can be based on a property or characteristic of the virtual object, such as the type of virtual object. For example, a particular type of virtual object may be displayed in a particular region of the display image, while another type of virtual object may be displayed in another region of the display image. In such embodiments, the location of the virtual object relative to the scene in the display image can be preprogrammed into processing subsystem 12. For example, if the virtual object is associated with or represents a real-world object, such as an electronic device, processing subsystem 12 may display the virtual object in a location in the display image that is preferred by the subject. For example, if the virtual object is an icon representing an appliance (e.g., a laundry appliance, such as a microwave, dishwasher, washing machine, or dryer), processing subsystem 12 may provide image data of the virtual object to display device 60 so that the virtual object appears in a corner of the display image or in another convenient location in the display image.

[0120] In other embodiments, the location of the virtual object relative to the scene 80 in the displayed image 90 may be based on input data, such as input from an eye tracking subsystem (e.g., eye tracker 70). For example, if the gaze direction of one or more eyes 44 is directed toward the bottom right corner of the display device 60, the eye tracker 70 may detect this gaze direction and provide a signal input to the processing subsystem 12 to cause the virtual object to be displayed in the bottom right corner of the display device 60.

[0121] In particular embodiments, processing subsystem 12 may modify the display image, including elements of the display image that correspond to virtual objects, to adjust display parameters of the display image. For example, processing subsystem 12 may modify the display image to adjust / change the position of a virtual object within the display image. As another example, processing subsystem 12 may modify the display image to increase or decrease the size of the virtual object (or other portion of the display image) relative to other portions of the display image.

[0122] As mentioned above, processing subsystem 12 may initiate one or more responsive actions in response to identifying the data record. By way of example, the one or more responsive actions may include activating and / or manipulating a real-world object with which virtual object 92 is associated or on which virtual object 92 is based. As noted above, real-world objects may include, but are not limited to, interactive display screens such as a computer keyboard and a computer mouse; aircraft cockpits; HUDs such as those that may be located in the cockpits of automobiles or aircraft (e.g., automobiles, buses, trains, trucks, large vehicles and heavy equipment (e.g., construction vehicles and excavation vehicles), military vehicles, etc.); control or operating panels for aircraft or automobiles (including the aforementioned types of vehicles); electronic devices / appliances such as smartphones, tablets, digital radio receivers, speaker systems, home appliances such as ovens, microwave ovens, dishwashers, or laundry appliances (e.g., washing machines, dryers), etc. Thus, for example, if virtual object 92 is associated with a washing machine appliance, the one or more responsive actions may include, for example, manipulating the washing machine to start a wash cycle. In such an example, the virtual object 92 may be, for example, an icon representing the subject's washing machine, and an interaction with the icon (e.g., pointing or gesturing at the icon, initiating a voice command, or any other triggering mechanism performed by the subject) may trigger a wash cycle.

[0123] It should be noted that, to enable operation of any electronic device by system 10 according to embodiments of the present disclosure, such electronic device must be in electronic or data communication with system 10, such that communication messages, such as commands, operational messages, and instructions, can be exchanged between system 10 and one or more electronic devices. In one non-limiting set of examples, the electronic devices are network-enabled devices that communicate with system 10 over a communication network. For example, as is well known in the art, electronic devices can be made "smart" electronic devices by incorporating networking devices into them. Other electronic devices may be pre-configured as network-enabled devices, e.g., pre-configured with wireless network communication hardware, such as Bluetooth-enabled hardware.

[0124] FIG. 9 schematically illustrates a networked system environment in which system 10, according to an embodiment of the present disclosure, may be deployed. In the non-limiting embodiment illustrated in FIG. 9, processing device 13a may be coupled to a transceiver (Tx / Rx) unit 30, which provides a communication / network interface for transmitting and receiving data (i.e., exchanging data) with one or more network devices 38 over a network 36 (which may be one or more communication networks, such as a cellular network, a local area network, the Internet, etc.). Also illustrated in FIG. 9 is an external storage medium 32 coupled to (e.g., electronically connected to) processing device 13a and a server system 34 (having one or more processors and one or more storage media) communicatively coupled to processing device 13a over network 36. External storage medium 32 may be used to store and provide various types of data to processing device 13a, such as image data representing objects for overlaying on a scene in a displayed image. Server system 34 may also be used to store and provide various types of data to processing device 13a (similar to external storage medium 32), and may be configured to perform some (or all) of the processing functions of processing device 13a. Thus, in certain embodiments, processing subsystem 12 may comprise server system 34.

[0125] It should be noted that a given response action may be selected from a plurality of response actions, where each response action from the plurality of actions is associated with a corresponding data record from the plurality of data records, and where each data record from the plurality of data records is associated with a corresponding one or more elements of image data of the display image 90 that are associated with a corresponding spatial location of a portion of the display image that includes at least a portion of the virtual object. As a result, for example, a given virtual object may have a plurality of locations, where each location is associated with a corresponding one or more image data elements and a corresponding response action. As an example, consider a virtual object as an icon representing a target washing machine. One region / location of the virtual object (icon) may correspond to a first response action, such as starting a wash cycle, while another region / location of the virtual object (icon) may correspond to a second response action, such as ending the wash cycle.

[0126] As another non-limiting example, the one or more responsive actions may include displaying one or more new virtual objects to the subject as part of the displayed image, where each new virtual object is associated with a corresponding responsive action. For example, virtual object 92 itself may be a virtual computer keyboard, displayed to subject 40 as a computer keyboard overlaid on a scene in the displayed image. Subject 40 may interact with the virtual keyboard to compose a message. For example, if subject 40 wishes to compose the word "hello," subject may initiate an action on a portion of the virtual keyboard containing the letter "h." Upon detecting this subject-initiated action, processing subsystem 12 identifies data records associated with one or more elements of image data for displayed image 90 that are associated with the spatial location of the portion of the displayed image containing the virtual letter "h." Subject 40 may subsequently initiate sequential actions on corresponding portions of the virtual keyboard containing the letters "e," "l," "l," and "o," and in response to each detected subject-initiated action, processing subsystem 12 identifies data records associated with one or more elements of image data for displayed image 90 that are associated with the spatial location of the portion of the displayed image containing the corresponding virtual letter.

[0127] As discussed above, a storage medium (e.g., a memory or a database) associated with system 10, such as storage medium 16a and / or 16b, can be configured to store a set of data records (i.e., a plurality of data records). According to certain embodiments, the plurality of data records includes multiple subsets of data records, each subset associated with a different virtual object of the plurality of virtual objects. For example, the plurality of data records may include two subsets of data records, i.e., a first subset of data records and a second subset of data records. The first subset of data records may include two data records, each associated with a first virtual object (e.g., virtual icons corresponding to home appliances), and the second subset of data records may include one data record associated with a second virtual object (e.g., a virtual keyboard).

[0128] In certain scenarios, it may be advantageous or beneficial for a subject to receive some type of feedback from system 10 when system 10 accepts a subject-initiated action and / or when system 10 initiates a responsive action. In certain non-limiting embodiments, system 10, e.g., processing subsystem 12, may provide feedback in the form of an alert or notification to the subject. For example, in response to an action being triggered on a virtual object in a displayed image, processing subsystem 12 may provide an alert or notification to the subject, e.g., in the form of an auditory alert (e.g., a chime or other sound). In another non-limiting example, the alert or notification may be in the form of a message, e.g., a short message service (SMS) or other suitable text-based message, or in the form of an email. In yet another non-limiting example, the alert or notification may be a flag or marker applied to an identified data record, which may enable processing / handling at an appropriate time, e.g., later, by, e.g., processing subsystem 12.

[0129] According to particular embodiments, the feedback may be physical, which may be, for example, visual feedback or another type of sensory feedback initiated by processing subsystem 12. For example, in one group of non-limiting embodiments, the feedback may be in the form of visual feedback within the displayed image itself. For example, in response to an action being invoked on a virtual object within the displayed image, processing subsystem 12 may operate display device 60 to display one or more images illustrating an action corresponding to the invoked action. For example, continuing with the virtual keyboard example above, if subject 40 wishes to form the word "hello," processing subsystem 12 may, in response to the subject initiating an action on a portion of the virtual keyboard containing the associated letter, provide image data to display device 60 representing an avatar or character typing on the virtual keyboard, thereby allowing the subject to view the avatar / character typing on the virtual keyboard.

[0130] In another non-limiting embodiment, system 10 uses haptic technology to provide non-visual sensory feedback to target 40. Haptic technology is a type of technique for providing physical feedback to a user / target, applying force, vibration, or movement to the user / target to create a tactile experience. Referring again to FIG. 1 , in certain embodiments, for example, a haptic interface 75 may optionally be disposed in association with processing subsystem 12 and target 40. Haptic interface 75 is configured to provide haptic feedback to target 40 in response to receiving an input corresponding to an object-initiated action and / or in response to initiating one or more responsive actions. Haptic interface devices are well known in the art and may include, for example, gloves, watches, headsets, vests, or other wearable apparel equipped with haptic feedback devices that produce vibrations or pressure (or other haptic feedback) that are sensed by target 40 through touch or feel. As one non-limiting example implementation, haptic interface 75 may be implemented as a haptic glove configured to be worn on the hand of subject 40 and providing pressure or vibration feedback to one or more fingers of the hand wearing the haptic glove. In such an example, subject 40 may initiate an action on a virtual object in a displayed image by "touching" the virtual object with a finger of subject 40, and haptic interface 75 may vibrate or apply pressure to the finger in response to the action initiated by the subject.

[0131] Although embodiments supporting interaction with virtual objects and manipulation of real-world objects have been described above in the context of system 10 having display device 60 that projects a display image that is a perceptual image that is augmented to include one or more virtual objects, according to certain embodiments, virtual object overlay and interaction may be achieved without the use of a display device.

[0132] According to one exemplary embodiment, to present a virtual object to a subject as overlaid on a scene, processing subsystem 12 may receive image data representing the object, process the received image data (using impulse-to-image mapping) to convert the image data into one or more neural impulses conveying image information, and then provide those neural impulses to visual processing region 43 of brain 42 of subject 40 while subject 40 views a real scene (e.g., scene 80) with one or more of subject's eyes 44, thereby causing subject 40 to visually perceive the object (represented by the received image data) as a virtual object overlaid on the real scene viewed by subject's one or more eyes 44. Alternatively, according to another exemplary embodiment, processing subsystem 12 may convert collected neural impulses conveyed in response to subject 40 viewing a real scene (e.g., scene 80) into image data (using impulse-to-image mapping), and then modify this image data to include the object. The processing subsystem 12 then converts the modified image data into neural impulses and provides those neural impulses to the visual processing region 43 of the brain 42 of the subject 40 while the subject 40 views a real scene (e.g., scene 80) with one or more eyes 44, thereby causing the object (represented by the received image data) to be visually perceived by the subject 40 as a virtual object superimposed on the real scene viewed by the subject's one or more eyes 44.

[0133] Processing subsystem 12 may then identify data records associated with one or more elements of image data representing the object, as described above. These one or more elements of image data associated with the identified data records are also associated with the spatial location of the virtual object relative to the object. As described above, the data records preferably include attributes of or associated with the image data, such as, for example, one or more responsive actions that processing subsystem 12 may initiate in response to identifying the data records, characteristics or features of the virtual object, display features or parameters such as the size of the virtual object within the target's FOV, and the location at which the virtual object is located within the target's FOV. Processing subsystem 12 may then initiate one or more responsive actions associated with one or more virtual objects in response to identifying the data records, as described above.

[0134] Identifying the data record may be performed in response to detecting a subject-initiated action in or on a virtual object, as described above. Additionally, in embodiments without a display device, a haptic interface may be used that provides haptic feedback to the subject in response to the subject-initiated action and / or in response to one or more responsive actions.

[0135] As mentioned above, in human subjects, the visual processing region 43 is commonly referred to as the visual cortex. The visual processing region 43 is also commonly referred to as the visual cortex in many other non-human animal species, such as canines, felines, non-human primates, and rodents. In human subjects and many other vertebrates, the visual cortex is a portion of the temporal lobe that processes visual information. In animal species that do not have a cerebral cortex or visual cortex (e.g., reptiles, birds, non-mammalian marine / aquatic species, etc.), the term "visual processing region" refers to one or more equivalent portions of the brain that perform visual processing. Thus, while embodiments of the present invention are of particular use when applied within the context of human vision, embodiments of the present disclosure may be equally applicable to the vision of non-human animal subjects, including, but not limited to, other primates (e.g., monkeys, gorillas, etc.), canines, felines, reptiles, birds, and non-mammalian marine / aquatic species.

[0136] Implementation of the method and / or system of embodiments of the present invention may involve performing or completing selected tasks manually, automatically, or a combination thereof. Furthermore, depending on the actual equipment and facilities of the method and / or system embodiments of the present invention, selected tasks may be implemented by hardware, software, firmware, or a combination thereof using an operating system.

[0137] For example, hardware for performing selected tasks according to embodiments of the present invention may be implemented as a chip or circuit. As software, selected tasks according to embodiments of the present invention may be implemented as a plurality of software instructions executed by a computer using any suitable operating system. In exemplary embodiments of the present invention, one or more tasks according to exemplary embodiments of the methods and / or systems described herein are performed by a data processor, such as a computing platform, for executing a plurality of instructions. Optionally, the data processor includes volatile memory for storing instructions and / or data and / or non-volatile storage (e.g., a non-transitory storage medium such as a magnetic hard disk and / or removable media) for storing instructions and / or data. Optionally, a network connection is also provided. Optionally, a display and / or user input devices, such as a keyboard, mouse, etc., are also provided.

[0138] For example, any combination of one or more non-transitory computer-readable (storage) media may be utilized in accordance with the above-described embodiments of the present invention. The non-transitory computer-readable (storage) medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media may include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. As used herein, a computer-readable storage medium may be any tangible medium capable of holding or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

[0139] A computer-readable signal medium may include a propagated data signal containing computer-readable program code, for example embodied in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium but may be any computer-readable medium that can communicate, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0140] As can be understood with reference to the above paragraphs and the referenced drawings, various embodiments of computer-implemented methods are provided herein, some of which may be performed by various embodiments of the devices and systems described herein, and some of which may be performed according to instructions stored on non-transitory computer-readable storage media described herein. Nevertheless, as will be apparent to those skilled in the art with reference to the embodiments described herein, some embodiments of the computer-implemented methods provided herein may be performed by other devices or systems and may be performed according to instructions stored on computer-readable storage media other than those described herein. References to systems and computer-readable storage media with respect to the computer-implemented methods below are for illustrative purposes and are not intended to limit any of such systems and such non-transitory computer-readable storage media with respect to the computer-implemented method embodiments described above. Similarly, any references to the computer-implemented methods below with respect to systems and computer-readable storage media are provided for illustrative purposes and are not intended to limit any of such computer-implemented methods disclosed herein.

[0141] The block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the block diagrams may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing one or more specified logical functions.

[0142] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over existing technology in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0143] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly indicates otherwise. For example, a reference to a single nerve can also refer to both nerves of a nerve pair. Further, a reference to both nerves of a nerve pair can also refer to a single nerve, unless the context clearly indicates otherwise.

[0144] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0145] It is understood that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination, or as suitable with any other described embodiment of the invention. Certain features described in the context of various embodiments should not be considered essential features of those embodiments, except to the extent that the embodiment is inoperable without those elements.

[0146] The above processes (including portions thereof) can be implemented by software, hardware, and combinations thereof. These processes, or portions thereof, can be implemented by computers, computer-based devices, workstations, processors, microprocessors, other electronic searching tools, and their associated memories and other non-transitory storage devices. The processes, or portions thereof, can also be embodied in programmable non-transitory storage media, such as compact discs (CDs) or other machine-readable magnetic, optical, or other disks, or other computer-usable storage media including magnetic, optical, solid-state storage, or other electronic signal sources.

[0147] Processes (methods) and systems (including components thereof) have been described herein with reference to specific hardware and software as examples. The processes (methods) are described as examples, and certain steps and their order may be omitted and / or changed by those of ordinary skill in the art in order to tailor these embodiments for practice without undue experimentation. The processes (methods) and systems have been described in a sufficient manner so that those of ordinary skill in the art can readily adapt other hardware and software, as necessary, to tailor any of the embodiments for practice without undue experimentation and using conventional techniques.

[0148] Where the appended claims are drafted without multiple dependencies, this is done solely to accommodate formality requirements in jurisdictions that do not recognize such multiple dependencies. It should be noted, however, that all possible combinations of features that could be implied by making a claim multiple dependent are expressly contemplated and should be considered part of the present invention.

[0149] While the present invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

Claims

1. 1. A system for use in a subject having a brain including an area responsible for visual perception, comprising: a display device operable in a display mode and a non-display mode; a processing subsystem configured to communicate with at least one nerve associated with the subject's eye, receiving a signal related to a nerve impulse transmitted by the at least one nerve in response to viewing at least a portion of a scene with the eye during a sample period, the sample period being contemporaneous with operation of the display device in the non-display mode; processing the received signals to generate a perceptual image representative of the visual perception of the scene by the region of the brain; and a processing subsystem configured to provide the display device with an image for display based on the perceived image during a display period that is concurrent with operation of the display device in the display mode.

2. 10. The system of claim 1, wherein the processing subsystem is further configured to switch the display device between the non-display mode and the display mode, and wherein the display device does not display images when operating in the non-display mode.

3. 10. The system of claim 1, wherein the display device is an at least partially transparent display such that at least a portion of the scene is viewable by the eye through the display device when the display device is operating in the non-display mode.

4. 2. The system of claim 1, wherein when the display device is operating in the display mode, the display device displays an image visible to the eye, and when the display device is operating in the non-display mode, the display device does not display an image.

5. The system of claim 1 , wherein the display device is configured to operate in the non-display mode for a plurality of sampling periods and to operate in the display mode for a plurality of display periods.

6. 6. The system of claim 5, wherein the processing subsystem is further configured to controllably switch the display device so that it alternates between operating in the non-display mode and operating in the display mode, the processing subsystem being configured, when the display device operates in the non-display mode, to receive signals related to nerve impulses transmitted by the at least one nerve in response to the eye viewing at least a portion of the scene during a corresponding one of the sample periods, and the processing subsystem being configured, when the display device operates in the display mode during a corresponding one of the display periods subsequent to the corresponding one of the sample periods, to provide the display device with a corresponding display image based on a corresponding perceived image generated by the processing subsystem as a result of processing the received signals related to nerve impulses transmitted by the at least one nerve in response to the eye viewing the at least a portion of the scene during the corresponding one of the sample periods.

7. The system of claim 1 , wherein the display device comprises at least one of a surface-mounted display, a head-mounted display, a head-up display, or a holographic display.

8. The system of claim 1 , further comprising an interface arrangement that enables the processing subsystem to communicate with the at least one nerve.

9. The system of claim 8 , wherein the interface configuration is external to the object.

10. The system of claim 1 , wherein the image is the perceived image.

11. The system of claim 1 , wherein the image is a modified version of the perceived image.

12. The system of claim 1 , wherein the image is the perceived image extended to include images of objects overlaid on the perceived image.

13. The processing subsystem includes: identifying a data record associated with one or more elements of image data of the image, the one or more elements of image data associated with a spatial location of a portion of the image that includes at least a portion of the object; 13. The system of claim 12, further configured to initiate at least one responsive action in response to said identifying, said at least one responsive action being related to said object in said image.

14. 10. The system of claim 1, wherein the subject has a pair of eyes, and the display device is positioned to project an image for viewing by both of the eyes of the subject.

15. 2. The system of claim 1, wherein the eye of the subject is a first eye of the subject, the display device is arranged to project an image for viewing by the first eye, and the system further comprises a second display device arranged to project an image for viewing by a second eye of the subject, the second display device being operable in a display mode and a non-display mode.

16. 16. The system of claim 15, wherein the display device is configured to operate in a display mode when the second display device is operating in a non-display mode and to operate in a non-display mode when the second display device is operating in a display mode.

17. 1. A system for use in a subject having a brain including an area responsible for visual perception, comprising: a display device configured to be positioned in a spatial relationship with the object; a processing subsystem configured to communicate with at least one nerve associated with the subject's eye, receiving a signal related to a nerve impulse transmitted by the at least one nerve in response to the eye viewing a scene; processing the received signals to generate a perceptual image representative of the visual perception of the scene by the region of the brain; generating the displayed image by combining the perceived image and an image of an object such that the object appears superimposed on the scene in the displayed image; providing the display image to the display device for viewing by the subject; and a processing subsystem configured to: identify a data record associated with one or more elements of image data of the display image, the one or more elements of image data associated with a spatial location of a portion of the display image that includes at least a portion of the object.

18. 20. The system of claim 17, wherein the processing subsystem is further configured to initiate at least one responsive action associated with the object in response to identifying the data record.

19. The system of claim 18 , wherein the at least one responsive action comprises at least one of activating or manipulating at least one electronic device associated with the object.

20. 20. The system of claim 18, wherein the at least one response action is selected from a plurality of response actions, each response action of the plurality of response actions being associated with a corresponding data record of a plurality of data records, each data record of the plurality of data records being associated with a corresponding one or more elements of the image data.

21. The system of claim 17 , wherein the processing subsystem is configured to identify the data record in response to detecting an event associated with the portion of the displayed image that includes at least a portion of the object.

22. 20. The system of claim 17, wherein the processing subsystem is further configured to initiate feedback provided to the subject in response to detecting an event associated with the portion of the displayed image that includes at least a portion of the object.

23. 20. The system of claim 17, further comprising a haptic interface associated with the object, wherein the processing subsystem is further configured to actuate the haptic interface to provide haptic feedback to the object in response to detecting an event associated with the portion of the displayed image that includes at least a portion of the object.

24. 20. The system of claim 17, wherein the object is one of a plurality of objects, the system further comprising at least one storage medium for maintaining a plurality of data records including the data record, the plurality of data records including a plurality of subsets of data records, each of the subsets being associated with a corresponding object of the plurality of objects.

25. 1. A method for use in a subject having a brain containing an area responsible for visual perception, comprising: Disposing a processing subsystem configured to be in communication with i) at least one nerve associated with an eye of the subject, and ii) a display device disposed in a spatial relationship with the subject and configured to operate in a display mode and a non-display mode; receiving, by the processing subsystem, a signal related to a nerve impulse transmitted by the at least one nerve in response to viewing at least a portion of a scene with the eye during a sample period, the sample period being contemporaneous with operation of the display device in the non-display mode; processing, by the processing subsystem, the received signals to generate a perceptual image representative of the visual perception of the scene by the region of the brain; and providing, by the processing subsystem, to the display device, during a display period contemporaneous with operation of the display device in the display mode, an image for display based on the perceptual image.

26. 1. A method for use in a subject having a brain containing an area responsible for visual perception, comprising: Disposing a processing subsystem configured to be in communication with i) at least one nerve associated with an eye of the subject, and ii) a display device disposed in a spatial relationship with the subject and configured to operate in a display mode and a non-display mode; receiving, by the processing subsystem, a signal related to a nerve impulse transmitted by the at least one nerve in response to the eye viewing a scene; processing, by the processing subsystem, the received signals to generate a perceptual image representative of the visual perception of the scene by the region of the brain; generating, by the processing subsystem, the display image by combining the perceptual image and an image of an object such that the object appears superimposed on the scene in the display image; providing, by the processing subsystem, the display image to the display device for viewing by the subject; and identifying, by the processing subsystem, a data record associated with one or more elements of image data of the display image, the one or more elements of image data being associated with a spatial location of a portion of the display image that includes at least a portion of the object.

Images