Method for tracking visual attention, computing device and computer readable storage medium

Abstract

The invention discloses a method for tracking visual attention, a computing device and a computer readable storage medium. The method comprises the following steps: filtering graphic data of a screen object to obtain a basic image; identifying an object of interest within the base image; generating at least one visual stimulus by processing at least a portion of the base image including the object of interest to extract a high spatial frequency (HSF) component of the base image, each stimulus having a characteristic modulation applied to the HSF component of the base image, the HSF component having a spatial frequency above a predetermined high spatial frequency threshold; displaying the or each visual stimulus via a graphical user interface (GUI) of the display; receiving a neural signal of a user from a neural signal capture device; and determining a point of interest of the user in the viewed image based on a neural signal when the user views the or each visual stimulus, the neural signal comprising information associated with a characteristic modulation of the visual stimulus directed by the visual attention of the user.

Description

[0001] This patent application is a divisional application of a patent application with an international application date of November 6, 2020, national application number 202080080126.8, and invention name “Visual Brain-Computer Interface”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application entitled “Brain-Computer Interface” and having Serial No. 62 / 938,098, filed on November 20, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present invention relates to the operation of brain-computer interfaces that include visual sensing. Background Art

[0005] In a visual brain-computer interface (BCI), neural responses to a target stimulus, typically among multiple generated visual stimuli presented to a user, are used to infer (or "decode") which stimulus is essentially the object of attention at any given time. The object of attention can then be associated with an action that is selectable or controllable by the user.

[0006] Various known techniques can be used to obtain neural responses. One convenient method relies on surface electroencephalography (EEG), which is non-invasive, has fine-grained temporal resolution, and is based on a well-understood empirical basis. Surface EEG can measure the changes in diffuse potentials on the surface of a subject's skull (i.e., scalp) in real time. These potential changes are generally referred to as electroencephalographic signals or EEG signals.

[0007] In a typical BCI, visual stimuli are presented in a display generated by a display device. Examples of suitable display devices (some of which are listed in Fig. 9 908 ) include television screens and computer monitors 902 , projectors 910 , virtual reality headsets 906 , interactive whiteboards and display screens of tablet computers 904 , smart phones, smart glasses 908 , etc. The visual stimuli 911 , 911 ′, 912 , 912 ′, 914 , 914 ′, 916 , 918 may form part of a generated graphical user interface (GUI) or they may be presented as augmented reality (AR) or mixed reality graphical objects 916 superimposed with a base image: the base image may be just the user's actual field of view (as in the case of a mixed reality display function projected onto a transparent display of a set of smart glasses) or a digital image corresponding to the user's field of view but captured in real time by an optical capture device (which in turn may capture images corresponding to the user's field of view among other possible views).

[0008] Inferring which, if any, of multiple visual stimuli is the object of attention at any given time is fraught with difficulty. For example, when a user is presented with multiple stimuli, such as numbers displayed on an on-screen keyboard, it has proven nearly impossible to infer directly from brain activity which one is in attention at a given time. The user perceives a number, such as the number 5, as being in attention, so the brain must contain information that distinguishes that number from the others, but current methods are unable to extract that information. That is, current methods can infer with difficulty that a stimulus has been perceived, but they cannot determine which particular stimulus is in attention using brain activity alone.

[0009] To overcome this problem and provide sufficient contrast between the stimulus and the background (and between stimuli), it is known to configure the stimuli used by the visual BCI to flicker or pulse (e.g., a large surface of a pixel switches from black to white and a large surface of a pixel switches from white to black) so that each stimulus has a distinguishable distribution of characteristics over time. The flickering stimulus causes a measurable electrical response. Specific techniques monitor different electrical responses, such as steady-state visual evoked potentials (SSVEPs) and P-300 event-related potentials. In a typical implementation, the stimulus flickers at a rate of more than 6 Hz. Therefore, such a visual BCI relies on a method that includes displaying various stimuli discretely (rather than continuously) and usually at different points in time. It is found that brain activity associated with attention paid to a given stimulus corresponds to (i.e., correlates with) one or more aspects of the temporal distribution of the stimulus, such as the frequency of the stimulus flickering and / or the duty cycle of the stimulus alternating between the flickering state and the static state.

[0010] The decoding of the neural signal therefore relies on the fact that when a stimulus is turned on, it will trigger a characteristic pattern of neural responses in the brain that can be determined from the electrical signals picked up by the electrodes of an EEG device, such as the electrodes of an EEG helmet, namely the SSVEP. This pattern of neural data may be very similar or even identical for various digits, but it is time-locked to the digit being perceived: only one digit can pulse at any one time, so that a correlation with the neural response to the pulse and the time at which that digit pulsed can be determined as an indication that that digit is the object of interest.

[0011] By displaying each digit at a different point in time, turning the digit on and off at different rates, applying different duty cycles, and / or applying stimulation only at different points in time, the BCI algorithm can determine which stimulation, when turned on, is most likely to trigger a given neural response, allowing the system to determine the target of attention.

[0012] Visual BCIs have improved significantly in recent years, making it increasingly practical to accurately decode a user's attention in real time. However, the constant flickering of stimuli - sometimes, all of them on a whole screen when there are many - is an inherent limitation to using this technology on a large scale. In fact, flickering can cause discomfort and mental fatigue, and, if persistent, can also cause physiological responses such as headaches. Furthermore, the flickering effect can hinder the user's ability to focus on a specific target, as well as the system's ability to quickly and accurately determine the object of attention.

[0013] Figure 3A The influence of peripheral vision is shown. Subject 305 is shown as watching display screen 302, and this display screen 302 displays multiple numbers 310, 312 in keyboard. When subject tries to pay attention to the number "5" 310 in the on-screen keyboard discussed above, other (that is, peripheral) numbers (such as "3" 312) act as distractors, and their existence and the fact that they present a flickering effect instantly attract the user's attention. The display of peripheral numbers causes interference to the user's visual system. This interference in turn hinders the performance of BCI. Therefore, there is a need for distinguishing screen targets from their display stimuli to determine which one the user is paying attention to, and for distinguishing the object of attention (target) from the object (distractor) around the target with a certain speed and accuracy.

[0014] Conventionally, visual stimulation would occupy a large portion of the screen surface, either filled with high energy uniform light (bright white shapes) or a rough checkerboard. These large surfaces would remain dedicated to the visual BCI system and could not be used for any other purpose other than visual stimulation. These large stimulation surfaces are inconsistent with the fine discrete integration of the visual BCI system and are not conducive to the display device, e.g. Fig. 9 There is a limitation in the degree of freedom of design of the user interface in the display device shown in .

[0015] Known systems in the medical or related research fields typically include a head-mounted device with attachment locations for receiving individual sensors / electrodes. Electronic circuits are then connected to the electrodes as well as to a housing that is connected to the acquisition chain (i.e., an assembly of connected components for acquiring EEG signals). Thus, an EEG device typically consists of three different elements that the operator / exhibitor must assemble each time it is used. Likewise, the nature of an EEG device is such that technical assistance is desirable, if not required.

[0016] Furthermore, user acceptability of EEG devices (and their electrodes) has aesthetic limitations, as well as limitations in comfort and ease of use. In many cases, these limitations are effectively significant barriers to adoption of EEG technology. Examples of applications where comfort and the need for technical assistance for prolonged use hinder adoption include applications such as video games, training (e.g., for health and safety or flight simulation), sleep aids, etc.

[0017] Therefore, it is desirable to provide brain-computer interfaces that address the above challenges. Summary of the invention

[0018] The present disclosure relates to brain-computer interfaces in which visual stimuli are presented on a graphical interface such that they are neurally decodable and provide an improved user experience.

[0019] In certain aspects, the present disclosure describes systems and methods for improving the accuracy and speed of determining objects of interest in a field of objects or as specific regions within a single large target. Image data for all objects is processed to extract a version consisting only of high spatial frequency (HSF) components for each object.

[0020] The present disclosure relates to techniques for acquiring (potential) objects of interest within a user's field of view (typically but not always on a display presented to the user), extracting components related to visual properties of those objects (e.g., their edges), and applying modulation to the high spatial frequency components of those visual properties. Thus, flickering visual stimuli used to elicit neural responses, such as visual evoked potentials (VEPs), can be delivered only via the HSF version of the object. The modulation causes the object to flicker or otherwise change visually, so that the modulation acts as a stimulus for the associated neural response. In turn, the neural response can be measured and decoded to determine which object of interest is the focus of the user's attention.

[0021] In certain aspects, the image data may also be processed to extract another version of the object consisting only of low spatial frequency (LSF) components. In the case of extracting the LSF version, the modulated HSF version may be superimposed on the LSF version (which does not flicker).

[0022] In one aspect, the present disclosure includes a closed-loop feedback system in which a user stares at a screen and its objects, neural activity is captured as signals using electrodes of the helmet, and the proportion of HSF detected from the neural activity and associated with each object will change as the user's object of interest changes. This is somewhat equivalent to flickering the objects at different rates and duty cycles, but because the filtering makes the flickering display objects those that essentially cause an HSF response (e.g., HSF versions), the interference presented is much less. If the object is peripheral, the flickering of its HSF version will naturally be suppressed by human visual behavior. However, the object of interest whose HSF version flickers will induce an easily recognizable neural response. As a result, interference is significantly eliminated, making the experience more comfortable and the identification of the object of interest more accurate and timely.

[0023] The present disclosure also extends the technique to allow for perceptible modulation associated with a more general range of potential objects of interest. The extended technique is applicable to objects with a wider range of natural illumination and a wider range of spatial textures. Homogeneous and smooth objects can become neurally decodable stimuli even if they do not have sufficient HSF components to allow effective direct modulation.

[0024] According to another aspect, the present disclosure relates to a method for tracking visual attention, the method comprising: obtaining a basic image; generating at least one visual stimulus, each stimulus having a characteristic modulation, the characteristic modulation being applied to a high spatial frequency (HSF) component of the visual stimulus; displaying the visual stimulus or each visual stimulus via a graphical user interface (GUI) of a display; receiving a user's neural signal from a neural signal capture device; and when the user views the visual stimulus or each visual stimulus, determining the user's focus point in the viewed image based on the neural signal, the neural signal including information associated with the characteristic modulation of the visual stimulus to which the user's visual attention is directed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To easily identify the discussion of any particular element or act, the highest digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0026] Figure 1 An electronic architecture for receiving and processing EEG signals according to the present disclosure is shown;

[0027] Figure 2 A system incorporating a brain-computer interface (BCI) according to the present disclosure is shown;

[0028] Figure 3A shows the foveal region and peripheral vision as discussed in this disclosure;

[0029] Figure 3BIt shows that the user focuses on the object and all the screen objects including the focused object flash at different rates and times;

[0030] Figure 4 is an example of a flicker control mode in which objects are each assigned a flicker pattern that differs in start time, frequency, and duty cycle;

[0031] Figure 5A and Figure 5B shows how a screen display object can be filtered to produce an HSF version and an LSF version and then the versions are superimposed;

[0032] Figure 6 shows an exemplary embodiment in which the HSF version of a screen object flashes while the LSF version remains stationary;

[0033] Figure 7 Another exemplary embodiment is shown in which a screen object remains stationary while an overlay object is separated into an HSF version and an LSF version, and the HSF overlay version flashes while the LSF version remains stationary;

[0034] Fig. 8A , Figure 8B and Figure 8C Further aspects of the operation of the method of the present disclosure are shown;

[0035] Fig. 9 Various examples of display devices suitable for the methods of the present disclosure are shown;

[0036] Fig.10 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some example embodiments;

[0037] Fig.11 is a diagrammatic representation of a machine in the form of a computer system according to some example embodiments, within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed;

[0038] Fig.12 shows the main functional blocks in the operation method of the BCI according to the present disclosure; and

[0039] Fig.13 As shown above, Fig.12 The described processing of elementary images to generate visual stimuli. DETAILED DESCRIPTION

[0040] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that embody the illustrative embodiments of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various embodiments of the subject of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the subject of the present invention can be put into practice without these specific details. Generally, known instruction examples, protocols, structures and techniques are not necessarily shown in detail.

[0041] BCIs that exploit neural signals associated with vision can be used to determine which objects on the screen the user is focusing on.

[0042] It has been found that objects of interest in the foveal visual area are associated with a high HSF signal component. Similarly, objects in the peripheral visual area have been found to be associated with a high LSF signal component. Thus, for example, by applying this observation, a visual stimulus can be completely visible centrally but become virtually invisible in peripheral vision (where it would be visible peripherally under normal circumstances).

[0043] By enhancing those differences of HSF signal components and LSF signal components with the help of various filtering methods, the accuracy and speed of BCI can be improved.

[0044] Figure 1 An example of an electronic architecture for receiving and processing EEG signals by means of the EEG device 100 according to the present disclosure is shown.

[0045] To measure diffuse potentials on the skull surface of a subject 110 , EEG device 100 includes a portable device 102 (ie, a mask or cap), analog-to-digital conversion (ADC) circuitry 104 , and a microcontroller 106 . Figure 1 The portable device 102 comprises one or more electrodes 108, typically 1 to 128 electrodes, advantageously 2 to 64 electrodes, advantageously 4 to 16 electrodes.

[0046] Each electrode 108 may include a sensor for detecting electrical signals generated by the subject's neuronal activity and electronic circuitry for pre-processing (e.g., filtering and / or amplifying) the detected signals prior to analog-to-digital conversion: such electrodes are referred to as "active" electrodes. Figure 1 , active electrodes 108 are shown being used, with the sensor in physical proximity to the subject's scalp. The electrodes may be adapted for use with a conductive gel or other conductive liquid (referred to as "wet" electrodes) or without such a liquid (i.e., "dry" electrodes).

[0047] Each ADC circuit 104 is configured to convert signals of a given number of active electrodes 108 , such as 1 to 128 electrodes.

[0048] The ADC circuit 104 is controlled by the microcontroller 106 and communicates with the microcontroller 106, for example, via the protocol SPI (“Serial Peripheral Interface”). The microcontroller 106 packages the received data for transmission, for example, via Bluetooth, Wi-Fi (“Wireless Fidelity”) or Li-Fi (“Light Fidelity”) to an external processing unit (not shown), such as a computer, a mobile phone, a virtual reality headset, an automotive or aviation computer system such as a car computer or an aircraft computer system.

[0049] In some embodiments, each active electrode 108 is powered by a battery ( Figure 1 The battery is conveniently provided in the housing of the portable device 102.

[0050] In some embodiments, each active electrode 108 measures a corresponding potential value, subtracts the potential measured by the reference electrode (Ei=Vi-Vref) from the corresponding potential value, and digitizes the difference with the aid of the ADC circuit 104 and then sends it by the microcontroller 106.

[0051] In certain embodiments, the method of the present disclosure introduces a target object for display in a graphical user interface of a display device. The target object includes a control item, and the control item is in turn associated with an action selectable by a user.

[0052] Figure 2 A system incorporating a brain-computer interface (BCI) according to the present disclosure is shown. The system incorporates a neural response device 206, such as Figure 1 The EEG apparatus 100 is shown. In the system, images are displayed on a display of a display device 202. A subject 204 watches the images on the display and focuses on a target object 210.

[0053] In an embodiment, the display device 202 displays at least the target object 210 as a graphical object having a changing temporal characteristic that is different from the temporal characteristics of other display objects and / or the background in the display. The changing temporal characteristic can be, for example, a constant or time-locked flickering effect that changes the appearance of the target object at a rate greater than 6 Hz. In the case where more than one graphical object is a potential target object (i.e., in the case where the viewing subject is provided with a choice of target objects to focus attention on), each object is associated with a discrete spatial and / or temporal code.

[0054] The neural response device 206 detects neural responses (i.e., tiny electrical potentials indicative of brain activity in the visual cortex) associated with attention focused on the target object; thus, the visual perception of the changing temporal characteristics of the target object(s) acts as a stimulus in the subject's brain, generating a specific brain response consistent with the code associated with the target object being attended to. The detected neural response (e.g., electrical potential) is then converted into a signal and transmitted to the processing device 208 for decoding. Examples of neural responses include visual evoked potentials (VEPs), which are commonly used in neuroscience research. The term VEP includes conventional SSVEPs as mentioned above, in which the stimulus oscillates at a specific frequency, as well as other methods such as code-modulated VEPs, in which the stimulus is subjected to a variable or pseudo-random temporal code.

[0055] Processing device 208 executes instructions to interpret received neural signals to determine feedback indicating in real time the target object having the current (visual) focus of attention. Decoding the information in the neural response signal relies on the correspondence between that information and one or more aspects of the temporal distribution of the target object (i.e., stimulus).

[0056] In some embodiments, the processing device may conveniently generate image data for presentation on the display device 202 that includes a temporally varying target object.

[0057] Feedback can be conveniently presented visually on a display screen. For example, a display device can display an icon, cursor, crosshair, or other graphical object or effect as being close to a target object (or overlapping with the object or at least partially blocking the object), thereby highlighting an object that appears to be the current focus of visual attention. Obviously, the visual display of such feedback has a reflexive cognitive effect on the perception of the target object, thereby amplifying the brain response. This positive feedback (wherein an obvious target object is confirmed as an expected target object by virtue of a long period of amplified attention) is referred to as "neural synchronization" in this article.

[0058] Figure 3B shows that using Figure 1 and Figure 2 A neural response device in order to distinguish multiple target objects. Figure 3BThe neural response device worn by the user (i.e., viewer) 305 in FIG. 3 is an electrode helmet for an EEG device. Here, the user wearing the helmet views a screen 302 displaying multiple target objects (numbers in an on-screen keyboard) that flash at significantly different times, frequencies, and / or duty cycles. The electrode helmet can convey signals derived from neural activity. Here, the user is focusing on the number 5, 310, where at time t1, the number 3, 312, flashes; at time t2, the number 4, 314, flashes; at time t3, the number 5, 310', flashes; and at time t4, the number 6, 316, flashes. The neural activity as conveyed by the helmet signal will be significantly different at t3 than at other time points. This is because the user is focusing on the number 5, 310, which flashes at t3, 310'. However, in order to distinguish this signal appearing at t3 from the signal at other times, all objects on the screen must flash at significantly different times. As a result, the screen becomes active with the flashing objects, resulting in an uncomfortable viewing experience.

[0059] Figure 3B The system in can use display signal mode, such as Figure 4 An example pattern is shown in which screen objects will flash at different frequencies and duty cycles at different points in time.

[0060] Challenges One method of determining the object of interest (target) from surrounding objects (distractors) with speed and accuracy relies on the characteristics of the human visual system.

[0061] Studies of how human visual perception operates have shown that when staring at a screen with multiple objects and focusing on one of these objects, the human visual system will pick up both high spatial frequencies (HSF) and low spatial frequencies (LSF). Evidence suggests that the human visual system is primarily sensitive to the HSF component of the particular display region of focus (e.g., the object the user is gazing at): this corresponds to the central region of the subject's retina filled with cones, called the fovea. This can be seen in the Figure 3A , where the foveal area of ​​the display 318 where vision is clearest contrasts with the peripheral area 304.

[0062] In contrast, for peripheral objects, the human visual system is primarily sensitive to their LSF components. In other words, the picked-up neural signal will be substantially influenced by both the HSF component from the object in focus and the LSF component from the peripheral objects. However, since all objects induce a certain proportion of both HSF and LSF, processing the neural signal to determine the object of focus may be hampered by the LSF noise contributed by the peripheral objects. This tends to reduce the accuracy and timeliness of identifying the object of focus.

[0063] The basic science of this approach involves the difference in how our eye-brain system processes stimuli from objects of attention and peripheral objects. This separation between foveal (center of visual field) and peripheral vision is described in the literature in terms of specific frequency channels from the retina to the visual cortex, where foveal vision is mainly driven by HSF channels that convey visual details, while peripheral vision is mainly driven by LSF channels that convey coarse visual information such as the global shape of an object without details. These two types of information are associated with separate neural pathways, different functions, and different impacts on unconscious and conscious perception.

[0064] Spatial frequency is usually calculated based on the number of cycles per degree. It depends mainly on three parameters: the density of pixels per inch (dpi), also known as pixels per inch (ppi); the distance between the user's eyes and the monitor; and the cutoff frequency of the spatial filtering. Spatial frequency filtering can be used so that the stimulus signal retains only the HSF characteristics, or conversely only the LSF characteristics. The spatial frequency filtering used in the context of visual BCI can conveniently perform high-pass filtering on values ​​exceeding 7 cycles per degree and low-pass filtering on values ​​below 3 cycles per degree. In some cases, the lower threshold of the low-pass filtering can produce an output of uniform flat tones in some cases (such "low-pass filtering" is still an effective filtering). In contrast, the maximum value of the high-pass filtering threshold is limited by the resolution of the display system and ultimately limited by the visual physiological ability of the subject. In any case, regardless of the low-pass and high-pass other thresholds, the present disclosure operates without knowing the specific values ​​of frequency filtering and / or transformation. The main principle is to separate spatial frequency components to optimize visual BCI.

[0065] The human visual system is tuned to process multiple stimuli in parallel at different locations in the visual field, usually unconsciously or subconsciously. Therefore, peripheral object stimuli will continue to trigger neural responses in the user's brain, even if they appear at the periphery of the visual field. As a result, this causes competition among multiple stimuli and makes specific neural decoding of the object of attention (target) more difficult.

[0066] In prior art neural capture systems, due to the operation of the human visual system, the picked-up neural signal will be substantially influenced by both the HSF component from the object of interest and the LSF component from the surrounding objects. However, since all objects induce a certain proportion of both HSF and LSF, processing the neural signal to determine the object of interest may be hampered by the LSF noise contributed by the surrounding objects. This tends to reduce the accuracy and timeliness of identifying the object of interest.

[0067] Consider Again Figure 3BThe on-screen keyboard, the flickering peripheral signals 312, 314, 316 will induce LSF neural activity in the viewer, which will be captured and processed in parallel with the signals inducing HSF neural activity in the viewer stimulated by the flickering number 5, 310. Therefore, these peripheral objects can be considered as distractors, and the LSF signals they induce can be considered as noise. One result of this noise is that it takes longer for the system to accurately determine the object of interest.

[0068] In the method of the present disclosure, multiple objects are displayed in a manner that each object is divided into a version consisting only of the LSF component of the object and a version consisting only of the HSF component. In one example, the flickering visual stimulus used to induce SSVEP is transmitted only through the HSF version of the object. The flickering HSF version is superimposed on the LSF version (which does not flicker).

[0069] Figure 5A Four different screen display objects are shown, a number 501, a photograph 504, a cluster of various filler objects 507, and a cluster of multiple thin strokes 510, each with a different orientation. By filtering these objects, an HSF version of each object (represented as 502, 505, 508, and 511, respectively) and an LSF version of each object (represented as 503, 506, 509, and 512, respectively) can be generated. Keeping in mind that the human visual system is naturally more sensitive to HSF signals from the object of interest and LSF signals from surrounding objects, if the display is separated into HSF and LSF versions, and only the HSF version is flickered while the LSF version is displayed constantly, the surrounding object "noise" generated by the LSF can be suppressed by keeping the LSF version of each object constant and only flickering the HSF version of each object. This effectively reduces the LSF noise from surrounding objects and reduces the LSF signal from the object of interest. Since the human visual system naturally associates HSF with foveal objects (those in focus), keeping the LSF version constant will have essentially no effect on the neural signals stimulated by the objects in focus, but will have reduced LSF interference from peripheral objects. It has been observed that when the screen display object is a plurality of fine strokes, the HSF version 511 of the base object 510 effectively provides a particularly high possible spatial frequency relative to the base object (while remaining fully visible), while the corresponding LSF version 512 of the base object 510 provides a particularly low possible spatial frequency (because it is a monochromatic uniform background).

[0070] Figure 5BThe screen display result that appears when the LSF version and the HSF version of the target object are superimposed and displayed is shown. Because only one version is flickering (e.g., the HSF version), the apparent flicker of the object is softer for surrounding objects, while the flicker of the object of interest is obvious because the human visual system is more sensitive to HSF-induced stimulation from the object of interest.

[0071] Figure 6 is an exemplary illustration of one implementation of the disclosed subject matter. Here, all target objects 601 are filtered to create their own HSF and LSF versions.

[0072] exist Figure 6 In the embodiment of the present invention, the target object 601 is a different graphical element in the graphical interface presented on the display of the display device. Examples of target objects include glyphs, photo images, and user interface elements. The display device may include a processing unit (not shown), a modulator subsystem 605, and a display driver subsystem 606.

[0073] For each target object 601, the processing unit of the display device operates to apply spatial frequency filtering 602 (or spatial frequency transform) to generate an HSF version and an LSF version (represented as 603 and 604, respectively) of the target object 601. In other embodiments, the target objects 601 can be filtered to create only an HSF version of each object. Therefore, only an HSF version is generated. HSF versions of target objects in the form of multiple fine strokes are particularly suitable for such embodiments.

[0074] The modulator subsystem 605 processes and transmits the HSF version 603 and LSF version 604 of each object to the display driver subsystem 606. The HSF version 603 and LSF version 604 of each object are processed differently: the LSF version signal 604 is encoded to produce a static (e.g., non-flickering) display, while the HSF version signal 603 is temporally modulated (e.g., flickering at different times with optional different frequencies and / or duty cycle characteristics). The temporal modulation effect is typically a periodic change in the visual characteristics of the object, such as brightness, hue, color components, etc., and may be abrupt (switching between an on state and an off state) or may include a more gradual transition. An example of temporal modulation includes a flickering effect, in which the brightness characteristics of a group of pixels in a target object switch between two distinguishable brightness levels. Conveniently, the modulation effect encodes the optical characteristics to which the user's brain responds (when viewing the object) in a detectable and decodable manner. The processing may also include increasing the contrast level in the HSF version signal and / or reducing the contrast level in the LSF version signal.

[0075] In some embodiments, the modulation effect can be random or pseudo-random in nature: the change of the visual characteristics of the object, such as flickering, is performed at random times, for example, following a pseudo-random time pattern. Conveniently, a pseudo-random time pattern is used rather than a strictly random pattern to reduce the temporal overlap between different time patterns associated with different objects.

[0076] The display driver subsystem 606 receives the processed HSF and LSF versions of each object (either as separate signals or as a combined signal of the modulated HSF and LSF versions) from the modulator subsystem 605. As a result, the signal driving the display includes screen objects where the signal associated with the LSF produces a constant display, while the signal associated with the HSF produces a flickering effect (or other temporal modulation effect).

[0077] In some embodiments, as described above, the modulator subsystem 605 may process only the HSF version of each object and transmit it to the display driver subsystem 606: so that only the HSF version is received at the display driver subsystem 606. In such a case, the modulator subsystem 605 applies high-pass filtering to generate only the HSF version of each object. In certain other embodiments, the modulator subsystem may apply one or more spatial frequency filters (or spatial frequency transforms) to generate both the HSF version and the LSF version of the target object, but only transmit the HSF version of each object to the display driver subsystem 606. In each case, the HSF version is temporally modulated as described above, so that the display driver subsystem drives the display with a signal including a screen object having a temporally modulated HSF component. In another embodiment, the display driver subsystem 606 displays the generated HSF version 603 in addition to the (entire) target object 601. In such a case, the contrast in the modulated HSF version can be increased to improve the visibility of the visual stimulus on the object of interest.

[0078] The user's attention to the displayed (HSF modulated) object can then be detected (via capture 622 and decoding of the neural response 607). As a result, the temporal modulation effect of peripherally viewed objects is significantly reduced. Figure 1 and Figure 2When a user (i.e., subject) of a neuro-responsive device 620 of the type described in the discussion of is viewing the screen, only the object of interest will flicker vibrantly, while surrounding objects that are also temporally modulated will contribute less noise (because their modulation only appears in HSF components that are out of view for the user focusing on the object of interest). This enables the system to quickly and accurately determine which object the viewer is currently focusing on. The modulated HSF components that are barely visible in situations other than the object of interest make the stimulus barely visible to an external viewer, especially when viewed from a distance (e.g., a distance greater than twice the distance between a typical active user and the display device). This allows for discreet (e.g., more private) interaction between the user and the object(s) of interest.

[0079] In some alternative embodiments, the target object (or "screen object") is a distinct graphical element presented on a display of a display device, and an overlay object is generated to correspond to one or more screen objects. It is the overlay object rather than the screen object itself that is now filtered to produce the HSF version and the LSF version.

[0080] Figure 7 is an exemplary illustration of another embodiment of the disclosed subject matter that implements such a scheme. Figure 6 As in FIG. 1 , the display device may include a processing unit (not shown), a modulator subsystem 605, and a display driver subsystem 606: Figure 7 The display device is also provided with an overlay subsystem 704 .

[0081] Here, the screen object 601 will have a display signal transmitted to the overlay subsystem 704 .

[0082] In some embodiments, each screen object will have an overlay object associated with it. Figure 7 As shown, only some of the screen objects have associated overlay objects 701. The graphic overlay objects may be, for example, geometric shapes surrounding the corresponding screen objects.

[0083] In certain embodiments, for example Figure 7 In the illustrated embodiment, the overlay object 701 is now filtered rather than the screen object itself to produce an HSF version and an LSF version (again represented as 603 and 604, respectively). For each overlay object 701, the processing unit of the display device operates to apply a spatial frequency filter 702 (or a spatial frequency transform) to generate an HSF version 603 and an LSF version 604 of the overlay object 701.

[0084] The modulator subsystem 605 modulates the HSF version 603 and the LSF version 604 of each overlay object, respectively. The modulator subsystem 605 then transmits the modulated HSF version and LSF version of each overlay object to the overlay-folding subsystem 704. The modulator subsystem 605 may optionally transmit the modulated HSF version and LSF version of each overlay object as a single overlayed overlay object or as separate versions.

[0085] The overlay compositing subsystem 704 that receives the processed graphic overlay object processes the screen object 601 and the modulated overlay object to generate an overlay display signal based on the screen object and the overlay object.

[0086] The overlay subsystem 704 then transmits the processed overlay display signal to the display driver subsystem 606 to drive the display.

[0087] Individual modulation can reflect Figure 6 The modulator subsystem 605 is configured to process the overlay object 701 so that the LSF version 604 is static (e.g., constantly displayed) while the HSF version 603 is modulated to produce different flickering patterns for distinguishing the overlay objects. When the subjects focus on the screen object 601, they see the flickering overlay object 701 very clearly, while the surrounding screen objects and their flickering HSF overlay objects are naturally suppressed by the human visual system. Again, the overlay object 701 associated with the focused target screen object 601 is easily distinguished from other overlay objects, so that the system can quickly and accurately determine which screen object the user is focusing on.

[0088] The above schemes for processing the target object itself to apply modulation to the HSF version of the object or processing the superimposed object to be visually superimposed on the target object to apply modulation to the HSF version of the superimposed object become less effective when the HSF component of the target object / superimposed object provides insufficient stimulus impact. For example, the target object may be visually smooth so that there are not enough high-contrast sharp edges or patches to generate a large HSF component. In such a case, the amount of modulation based on the HSF version of the object will not be visually noticeable to the viewing subject.

[0089] Fig. 8A , Figure 8B and Figure 8COperation of a method of another aspect of the present disclosure is illustrated. In certain embodiments, a base image (e.g., a 2D pixel surface) comprising image data is processed for display in a graphical user interface of a display: the base image is processed to identify (and optionally classify) an object of interest within the base image. The base image and / or a portion of the base image comprising the identified object of interest is logically associated with one or more control items (the control items are associated with user-selectable actions).

[0090] Image data of a base image and / or a portion of the base image including the identified object of interest is processed to generate low spatial frequency map data and high spatial frequency map data. In some embodiments, the processing is achieved by applying spatial frequency filtering: features with spatial frequencies below a predetermined low spatial frequency threshold are retained in the low spatial frequency map data, while features with spatial frequencies above a predetermined high spatial frequency threshold are retained in the high spatial frequency map data. The predetermined high spatial frequency threshold and the predetermined low spatial frequency threshold may be the same. Alternatively, the corresponding spatial frequency thresholds may be different from each other, so that features with spatial frequencies between the two thresholds may be included in or excluded from the two sets of map data.

[0091] In certain embodiments, the distribution integral of the HSF components in the HSF map data is calculated for each of a plurality of patches across the HSF map data, the size of the patch being based on the foveal area of ​​the retina (the area of ​​the fovea sets the largest area over which the HSF components will have an effect). The integral for a given patch is compared to a predetermined spatial frequency threshold T1. The threshold T1 is a cutoff value that is set to a value that is considered to indicate that sufficient HSF components are present to provide a significant stimulus effect (i.e., in terms of the intensity of the HSF components and / or their spatial distribution over the base image).

[0092] In the event that the integral of a given patch exceeds a predetermined spatial frequency threshold T1, temporal modulation is applied to the HSF component of that patch, and the first modified (i.e., mixed) HSF and temporal signal (with or without the LSF component) is fed to the display. Examples of signal mixing include additive mixing, multiplication and subtraction, and other conventional signal multiplexing techniques.

[0093] In some embodiments, the technique used for blending is similar to conventional overlay blending modes used in the fields of digital image editing and computer graphics. Conveniently, in the modulated on-state, the topmost image is superimposed on the background image: portions of the top image where the underlying background image is light become lighter, and portions of the background image where it is dark become darker.

[0094] In the case where the integral of a given patch does not exceed a predetermined spatial frequency threshold T1, it is determined whether HSF noise is mixed in. Figure 8B and Figure 8C In some embodiments, it is determined whether the average brightness (and / or contrast) of each pixel of the basic image falls within a predetermined range: for example, a brightness range from (50-T2)% to (50+T2)%. The predetermined range is set to correspond to the range of brightness and / or contrast where HSF noise may be effective, that is, to maintain a low ratio between the brightness of the on state and the off state, while allowing low peripheral saliency and ensuring a good neural response.

[0095] If the average brightness of the pixels of the base image does fall within the predetermined range, the processor is configured to process the image data for the area (i.e., patch) where the HSF is below the predetermined threshold Tl to (at least partially) desaturate the local tones of the pixels in the area, thereby generating a desaturation map, and is configured to apply a high spatial frequency filtered noise patch or Gabor variant patch to the area where the HSF is below the predetermined threshold T1 to generate an HSF noise map for the area. Desaturation of the pixels is achieved by calculating the median of the RGB channels, which will statistically provide a brightness value close to 50% gray. Temporal (and spatial) modulation is applied to (one or more) HSF noise patches, and the desaturation map data and the second modified (i.e., mixed) HSF and temporal modulation are fed to the display. Therefore, after applying the temporal modulation, the brightness of these areas will remain substantially the same, while the stimulation caused by the temporal modulation is smooth (i.e., not visually disturbing), while promoting improved decoding of the focus target / foveal target.

[0096] If the average brightness of the pixels of the base image does not fall within the predetermined range (i.e., [50-T2:50+T2]), an alternative approach is needed in that region / patch. Too large or insufficient average brightness does not allow temporal modulation to be applied directly to the HSF portion of the base image. In some embodiments (by Figure 8C As shown in option (A) in FIG. 1 ), pixels of high and low values ​​are replaced by preset intermediate values ​​(thus discarding some original pixel information from the base image) to generate a gray overlay map. HSF noise is generated, and temporal modulation can then be applied to the HSF noise. The combined gray overlay map and the temporally modulated HSF noise are fed to the display. In some examples, the generated HSF noise is arranged to "balance" the brightness between the "off" state and the "on" state, generated at 50% black and 50% white, so that, for example, the HSF noise is actually also "gray".

[0097] In an alternative approach for pixels of the base image with out-of-range brightness (by Figure 8COption (B) in FIG. 1 shows an example of a method for displaying a grayscale image of a display device, wherein the pixels in the region are directly overlaid with pixel values ​​containing HSF noise. A temporal modulation may then be applied to the HSF noise. The combined gray overlay map and the temporally modulated HSF noise are fed to the display. In another embodiment, a contrast variation is applied to the LSF component so that when the HSF component and the LSF component are mixed with the additive mixture, the resulting visual stimulus is light balanced.

[0098] It should be understood that the spatial frequency thresholds and brightness thresholds discussed above can be conveniently dependent on (and set according to) many different factors, such as the subject's own visual ability, display device distance, display device pixel density, display device display size, etc.

[0099] Fig.10 is a block diagram illustrating an example software architecture 1006 that may be used in conjunction with the various hardware architectures described herein. Fig.10 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1006 may be implemented in a system such as Fig.11 1100, which includes, among other things, a processor 1104, a memory 1106, and an input / output (I / O) component 1118. A representative hardware layer 1052 is shown and may represent, for example, Fig.11 The machine 1100 of the embodiment of the present invention. The representative hardware layer 1052 includes a processing unit 1054 having associated executable instructions 1004. The executable instructions 1004 represent executable instructions of the software architecture 1006, including implementations of the methods, modules, etc. described herein. The hardware layer 1052 also includes a memory and / or storage module, shown as a memory / storage device 1056, which also has executable instructions 1004. The hardware layer 1052 may also include other hardware 1058, such as specialized hardware for interfacing with EEG electrodes and / or for interfacing with a display device.

[0100] exist Fig.10In the example architecture of , software architecture 1006 can be conceptualized as a stack of layers, in which each layer provides specific functionality. For example, software architecture 1006 may include layers such as operating system 1002, library 1020, framework or middleware 1018, application 1016, and presentation layer 1014. In operation, application 1016 and / or other components within a layer may call application programming interface (API) call 1008 through the software stack and receive a response as message 1010. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 1018, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

[0101] The operating system 1002 can manage hardware resources and provide public services. The operating system 1002 can include, for example, a kernel 1022, a service 1024, and a driver 1026. The kernel 1022 can be used as an abstraction layer between the hardware and other software layers. For example, the kernel 1022 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 1024 can provide other public services to other software layers. The driver 1026 can be responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, depending on the hardware configuration, the driver 1026 can include a display driver, an EEG device driver, a camera driver, drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0102] The library 1020 may provide a common infrastructure that may be used by the application 1016 and / or other components and / or layers. The library 1020 generally provides functions that enable other software modules to perform tasks in an easier manner than directly interfacing with the underlying operating system 1002 functions (e.g., the kernel 1022, the service 1024, and / or the driver 1026). The library 1020 may include a system library 1044 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the library 1020 may include an API library 1046, such as a media library (e.g., a library that supports the rendering and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that may be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that may provide various relational database functions), a network library (e.g., WebKit that may provide network browsing functions), and the like. The library 1020 may also include a variety of other libraries 1048 to provide many other APIs to the applications 1016 and other software components / modules.

[0103] The framework 1018 (also sometimes referred to as middleware) provides a higher-level common infrastructure that can be used by the applications 1016 and / or other software components / modules. For example, the framework / middleware 1018 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 1018 can provide a wide range of other APIs that can be used by the applications 1016 and / or other software components / modules, some of which can be specific to a particular operating system or platform.

[0104] The applications 1016 include built-in applications 1038 and / or third-party applications 1040 .

[0105] Applications 1016 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 1022, services 1024, and / or drivers 1026), libraries 1020, or framework / middleware 1018. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer such as presentation layer 1014. In these systems, the application / module "logic" may be separated from aspects of the application / module that interact with the user.

[0106] Fig.11 1 is a block diagram illustrating components of a machine 1100 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.11A schematic representation of a machine 1100 in the example form of a computer system is shown, in which instructions 1110 (e.g., software, programs, applications, applets, apps, or other executable codes) for causing the machine 1100 to perform any one or more of the methods discussed herein can be executed. Thus, the instructions 1110 can be used to implement the modules or components described herein. The instructions 1110 transform the general unprogrammed machine 1100 into a specific machine that is programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, the machine 1100 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1110 that specify actions to be taken by the machine 1100, either sequentially or otherwise. In addition, while only a single machine 1100 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1110 to perform any one or more of the methodologies discussed herein.

[0107] The machine 1100 may include a processor 1104, a memory 1106, and an input / output (I / O) component 1118 that may be configured to communicate with each other, for example, via a bus 1102. In an example embodiment, the processor 1104 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1108 and a processor 1112 that may execute instructions 1110. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.11Multiple processors are shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with single cores, multiple processors with multiple cores, or any combination thereof.

[0108] The memory 1106 may include a memory 1114, such as a main memory, static memory, or other memory storage device, and a storage unit 1116, both of which are accessible by the processor 1104, for example, via the bus 1102. The storage unit 1116 and the memory 1114 store instructions 1110 that implement any one or more of the methods or functions described herein. The instructions 1110 may also reside, completely or partially, within the memory 1114, within the storage unit 1116, within at least one of the processors 1104 (e.g., within a cache memory of the processor), or any suitable combination thereof during execution thereof by the machine 1100. Thus, the memory 1114, the storage unit 1116, and the memory of the processor 1104 are examples of machine-readable media.

[0109] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions 1110. The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media that can store instructions (e.g., instructions 1110) for execution by a machine (e.g., machine 1100), such that the instructions, when executed by one or more processors (e.g., processor 1104) of machine 1100, cause machine 1100 to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network including multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0110] Input / output (I / O) components 1118 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific input / output (I / O) components 1118 included in a particular machine will depend on the type of machine. For example, user interface machines and portable machines such as mobile phones will likely include a touch input device or other such input mechanism, while headless server machines will likely not include such a touch input device. It will be appreciated that input / output (I / O) components 1118 may include Fig.11 Many other components not shown.

[0111] The input / output (I / O) components 1118 are grouped according to function only for the purpose of simplifying the following discussion, and the grouping is by no means limiting. In various example embodiments, the input / output (I / O) components 1118 may include output components 1126 and input components 1128. The output components 1126 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1128 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0112] In further example embodiments, the input / output (I / O) component 1118 may include various other components such as a biometric component 1130, a motion component 1134, an environmental component 1136, or a positioning component 1138. For example, the biometric component 1130 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves such as output from an EEG device), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1134 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component 1136 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1138 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0113] Various technologies may be used to implement communications. Input / output (I / O) components 1118 may include communications components 1140 operable to couple machine 1100 to network 1132 or device 1120 via coupler 1124 and coupler 1122, respectively. For example, communications components 1140 may include a network interface component or other suitable device to interface with network 1132. In other examples, communications components 1140 may include wired communications components, wireless communications components, cellular communications components, near field communications (NFC) components, Parts (e.g. Low energy consumption), Components and other communication components that provide communication via other forms. Device 1120 can be another machine or any peripheral device of various peripheral devices (e.g., a peripheral device coupled via a universal serial bus (USB)). In the case where the EEG device or display device is not integrated with the machine 1100, the device 1120 can be an EEG device and / or a display device.

[0114] Fig.12 A processing device (eg, Figure 2 The main functional blocks in the operating method of the processing device 208 in the processing device 208). In block 1202, the processing device obtains a basic image. In block 1204, the processing device generates at least one visual stimulus, each stimulus having a characteristic modulation, which is applied to a high spatial frequency HSF component of the visual stimulus. In block 1206, the processing device controls the display of the visual stimulus or each visual stimulus via a graphical user interface GUI of the display. In block 1208, the processing device receives a neural signal of the user from a neural signal capture device. In block 1210, when the user views the visual stimulus or each visual stimulus, the processing device determines the user's focus point in the viewed image based on the neural signal, and the neural signal includes information associated with the characteristic modulation of the visual stimulus to which the user's visual attention is directed.

[0115] Fig.13 As shown above, Figure 6 The base image is processed as described to generate visual stimuli.The base image 1320 is subjected to image recognition 1310 (using, for example, a convolutional neural network) and the image is segmented 1308 to identify the boundary between the object(s) of interest (ie, target object 601) and the background.

[0116] for Figure 6 For an object of interest in FIG. 1 , image data of the object is processed to derive an HSF component 1306 and an LSF component 1304. Processing of the image data includes applying a spatial frequency filter or transform.

[0117] The resulting HSF component is then modulated to encode the optical properties to which the user's brain responds (when viewing the object) in a detectable and decodable manner. The modulated HSF component is mixed with the LSF component to generate a hybrid image object 1302. The user's attention to the hybrid image object 1302 can then be detected (through the capture and decoding of the neural response).

[0118] Once it becomes possible to identify objects of interest through the use of overlaid image objects, use cases such as “zooming” on a specific object / face in a surveillance video simply require focusing attention on that face / object and actioning the real object in an AR scene (e.g., extracting additional information associated with that object), with stimuli presented to the user in a headset (e.g., smart glasses) overlaid on top of the user's view of the real object.

[0119] and Figure 2The positive neural synchronization feedback loop described in connection with the BCI in the embodiment of the present invention can be used to confirm the user's intention, such as initiating actions such as zooming, information request, browsing / navigation of virtual UI elements, annotation of objects, activation / selection of objects (e.g., through a virtual device interface on a real wall) in a mixed reality setting.

[0120] Although described through a number of detailed exemplary embodiments, the portable device for acquiring electroencephalographic signals according to the present disclosure includes various variations, modifications and improvements that are obvious to those skilled in the art, and it should be understood that these various variations, modifications and improvements fall within the scope of the subject matter of the present disclosure as defined in the appended claims.

[0121] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. These embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention", only for convenience and if more than one disclosure or inventive concept is in fact disclosed, it is not intended to voluntarily limit the scope of the present application to any single disclosure or inventive concept.

[0122] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered to have a limiting meaning, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents to which such claims are assigned.

[0123] As used herein, the term "or" may be interpreted as inclusive or exclusive meaning. In addition, multiple instances may be provided for resources, operations, or structures described as single instances herein. In addition, the boundaries between various resources, operations, modules, engines, and data stores are arbitrary to some extent, and specific operations are shown in the context of specific illustrative configurations. Other allocations of functions are foreseen and may fall within the scope of various embodiments of the present disclosure. Typically, structures and functions presented as independent resources in the example configuration may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and the accompanying drawings are considered to be illustrative rather than restrictive.

[0124] Thus, the present disclosure describes systems and methods for improving the accuracy, speed performance, and visual comfort of BCIs.

[0125] According to one aspect of the present disclosure, the system is a closed-loop system, which includes: a display subsystem, which is operable to display one or more object images; a display drive subsystem, which is operable to transmit a display signal to the display subsystem; an HSF / LSF differentiation, filtering and processing subsystem, which is operable to substantially separate the object image into an HSF version and an LSF version, the HSF version and the LSF version respectively inducing a substantially HSF neural response and a substantially LSF neural response; the HSF / LSF differentiation, filtering and processing subsystem is operable to process the HSF version to turn on and off flicker, and to process the LSF version to not flicker; and an electrode helmet, which is operable to detect neural brain activity, generate an electrical signal representing the neural brain activity, and transmit the electrical signal to the HSF / LSF differentiation, filtering and processing subsystem, wherein the electrical signal is compared with a simultaneous display signal in the HSF / LSF differentiation, filtering and processing subsystem to associate the electrical signal with the corresponding simultaneous display signal.

[0126] The system may further comprise means for modulating said display signal of a screen object.

[0127] Alternatively or additionally, the system may further comprise means for modulating said display signal for screen object overlay.

[0128] According to another aspect of the present disclosure, a method for improving the accuracy, speed performance, and visual comfort of a BCI is provided, the method comprising: detecting a neural signal from a helmet electrode when the helmet electrode is worn on a user's head and the user is looking at an object on a display screen; associating the detected neural signal with a display signal for modulating the object; comparing the neural signal with the display signal; and identifying an object of interest, wherein the neural signal and the display signal are associated.

[0129] The method may further include identifying the object of interest, wherein the object of interest is a screen object.

[0130] Alternatively or additionally, the method may further include identifying the object of interest, wherein the object of interest is a screen object overlay.

[0131] Example

[0132] To better illustrate the systems and methods disclosed herein, a non-limiting list of examples is provided here:

[0133] 1. A method comprising:

[0134] filtering graphics data for one or more screen objects to generate a high spatial frequency (HSF) version of the or each screen object;

[0135] For each HSF version of the screen object, applying characteristic modulation;

[0136] generating a display signal comprising a visual stimulus corresponding to a modulated version of the screen object;

[0137] Displaying the display signal on a display screen;

[0138] When the user looks at the display screen, receiving a neural signal of the user from a neural signal capturing device;

[0139] for each visual stimulus, determining whether the neural signal received when the user is looking at the display screen includes a neural signature of the visual stimulus; and

[0140] When it is determined that the neural signal includes the neural signature of the visual stimulus, an object of interest of the user is identified in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0141] 2. A method according to Example 1, wherein the neural signal corresponds to neural oscillations measured in the visual cortex of the user's brain.

[0142] 3. A method according to Example 1 or 2, wherein the neural signature includes information associated with a characteristic modulation of the visual stimulus.

[0143] 4. The method according to any one of Examples 1, 2 or 3 further includes filtering the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and encoding a static display signal for each LSF version of the screen object.

[0144] 5. A method according to any one of Examples 1 to 4, wherein the graphics data of the one or more screen objects is filtered using at least one of spatial frequency filtering or spatial frequency transformation.

[0145] 6. A method according to any one of Examples 1 to 5, wherein the characteristic modulation is a characteristic time modulation.

[0146] 7. A method according to any one of Examples 1 to 6, wherein determining whether the received neural signal includes a digital signature of the visual stimulus comprises performing spectral analysis on the received neural signal, and determining whether the spectral characteristics of the received neural signal correspond to a spectrum associated with a characteristic modulation of the visual stimulus.

[0147] 8. A method according to any one of Examples 1 to 7, wherein the neural signal capture device includes an EEG helmet, the EEG helmet includes electrodes, and the EEG helmet is configured to be worn on the head of a user.

[0148] 9. A method according to any one of Examples 1 to 8, wherein the object of focus is the screen object itself.

[0149] 10. A method according to any one of Examples 1 to 9, wherein the object of interest is a display object displayed on the display screen, and the visual stimulus is a superimposed object that is different from the display object and displayed above the display object.

[0150] 11. A brain-computer interface system, comprising:

[0151] a display subsystem configured to present a display screen to a user;

[0152] a neural signal capturing device configured to capture neural signals associated with the user;

[0153] An interface device, operably coupled to the display subsystem and the neural signal capturing device, the interface device comprising:

[0154] Memory; and

[0155] a processor operably coupled to the memory and configured to:

[0156] filtering graphics data for one or more screen objects to generate a high spatial frequency (HSF) version of the or each screen object;

[0157] For each HSF version of the screen object, applying characteristic modulation;

[0158] generating a display signal comprising a visual stimulus corresponding to a modulated version of the screen object;

[0159] transmitting the display signal to the display subsystem for display on the display screen;

[0160] When the user looks at the display screen, receiving a neural signal of the user from the neural signal capturing device;

[0161] for each visual stimulus, determining whether the neural signal received when the user is looking at the display screen includes a neural signature of the visual stimulus; and

[0162] When it is determined that the neural signal includes the neural signature of the visual stimulus, an object of interest of the user is identified in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0163] 12. The brain-computer interface system of example 11, wherein the processor is further configured to:

[0164] Associating the object of interest with at least one control item from a set of control items;

[0165] Determining an action planned by the user based on the object of interest and the at least one control item; and

[0166] Implement the action planned by the user.

[0167] 13. The brain-computer interface system of Example 11 or Example 12, wherein the processor further comprises:

[0168] The display driving subsystem is operable to transmit a display signal to the display subsystem.

[0169] 14. A brain-computer interface system according to any one of Examples 11 to 13, wherein the neural signal capture device includes an electrode helmet, which is operable to detect neural brain activity, generate electrical signals representing the neural brain activity, and transmit the electrical signals to the interface device.

[0170] 15. A brain-computer interface system according to any one of Examples 11 to 14, wherein the object of focus is the screen object itself.

[0171] 16. A brain-computer interface system according to any one of Examples 11 to 14, wherein the object of interest is a display object displayed on the display screen, and the visual stimulus is a superimposed object that is different from the display object and displayed above the display object.

[0172] 17. A brain-computer interface system according to any one of Examples 11 to 16, wherein the processor is further configured to: filter the graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and encode a static display signal for each LSF version of the screen object.

[0173] 18. A computer-readable storage medium carrying instructions that, when executed by a computer, cause the computer to perform operations comprising:

[0174] filtering graphics data for one or more screen objects to generate a high spatial frequency (HSF) version of the or each screen object;

[0175] For each HSF version of the screen object, applying characteristic modulation;

[0176] generating a display signal comprising a visual stimulus corresponding to a modulated version of the screen object;

[0177] Displaying the display signal on a display screen;

[0178] When the user looks at the display screen, receiving a neural signal of the user from a neural signal capturing device;

[0179] for each visual stimulus, determining whether the neural signal received when the user is looking at the display screen includes a neural signature of the visual stimulus; and

[0180] When it is determined that the neural signal includes the neural signature of the visual stimulus, an object of interest of the user is identified in the display signal displayed on the display screen, the object of interest being a display object on the display screen that is consistent with the visual stimulus.

[0181] 19. A computer-readable storage medium as described in Example 18, wherein the neural signal corresponds to neural oscillations measured in a visual cortex of the user's brain.

[0182] 20. A computer-readable storage medium according to Example 18 or Example 19, wherein the neural signature includes information associated with a characteristic modulation of the visual stimulus.

[0183] 21. A computer-readable storage medium according to any one of Examples 18, 19 or 20, wherein the instructions further cause the computer to perform operations including: filtering graphic data of one or more screen objects to generate a low spatial frequency (LSF) version of the screen object or each screen object; and encoding a static display signal for each LSF version of the screen object.

[0184] 22. A computer-readable storage medium as described in any of Examples 18 to 21, wherein the graphics data of the one or more screen objects is filtered using at least one of spatial frequency filtering or spatial frequency transformation.

[0185] 23. A computer-readable storage medium according to any one of Examples 18 to 22, wherein the characteristic modulation is a characteristic time modulation.

[0186] 24. A computer-readable storage medium according to any one of Examples 18 to 23, wherein determining whether the received neural signal includes the digital signature of the visual stimulus comprises: performing spectral analysis on the received neural signal, and determining whether the spectral characteristics of the received neural signal correspond to a spectrum associated with a characteristic modulation of the visual stimulus.

[0187] 25. A method for tracking visual attention, the method comprising:

[0188] Get the basic image;

[0189] generating at least one visual stimulus, each stimulus having a characteristic modulation applied to a high spatial frequency (HSF) component of the visual stimulus;

[0190] displaying the or each visual stimulus via a graphical user interface GUI of a display;

[0191] receiving a user's neural signal from a neural signal capturing device; and

[0192] As the user views the or each visual stimulus, a point of focus of the user in the viewed image is determined based on the neural signal, the neural signal comprising information associated with a characteristic modulation of the visual stimulus to which the user's visual attention is directed.

[0193] 26. The method of example 25, wherein generating the stimulus comprises:

[0194] processing at least a portion of the base image to generate low spatial frequency (LSF) map data and HSF map data; and

[0195] It is determined whether the HSF map data satisfies a predetermined spatial frequency threshold T1.

[0196] 27. The method of example 26, wherein processing at least a portion of the base image comprises:

[0197] applying at least one of spatial frequency filtering or spatial frequency transform;

[0198] storing features of at least a portion of the base image having a spatial frequency below a predetermined low spatial frequency threshold in the low spatial frequency map data; and

[0199] Features of at least a portion of the base image having a spatial frequency above a predetermined high spatial frequency threshold are stored in the high spatial frequency map data.

[0200] 28. The method of example 26, wherein the HSF map data does not satisfy the predetermined spatial frequency threshold T1, and

[0201] Wherein, generating the stimulus further comprises:

[0202] generating enhanced HSF map data to introduce additional HSF stimuli into the visual stimulus; and

[0203] Temporal modulation is applied to the enhanced HSF map data.

[0204] 29. The method of example 28, wherein generating enhanced HSF map data comprises:

[0205] determining whether the brightness of at least a portion of the base image falls within a predetermined range; and

[0206] When the brightness of at least a portion of the base image falls within the predetermined range, a HSF noise map is generated to be used as the enhanced HSF map data.

[0207] 30. The method of example 29, wherein generating enhanced HSF map data when the brightness of at least a portion of the base image does not fall within the predetermined range comprises:

[0208] Generate a gray overlay map; and

[0209] generating an HSF noise map for use as the enhanced HSF map data, and

[0210] Wherein, displaying the visual stimulus or each visual stimulus comprises:

[0211] merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and

[0212] The mixed image is displayed via the GUI.

[0213] 31. The method of example 29, wherein generating enhanced HSF map data when the brightness of at least a portion of the base image does not fall within the predetermined range comprises:

[0214] Generate a gray overlay map;

[0215] generating a hard noise map; and

[0216] replacing the HSF map data with data from the hard noise map for use as the enhanced HSF map data, and

[0217] Wherein, displaying the visual stimulus or each visual stimulus comprises:

[0218] merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and

[0219] The mixed image is displayed via the GUI.

[0220] 32. The method of example 25, wherein generating the stimulus comprises:

[0221] Processing the basic image into segmented display objects;

[0222] For each segmented display object, filtering the segmented display object into a high spatial frequency HSF component and a low spatial frequency LSF component; and

[0223] Temporal modulation is applied to the HSF components of the segmented display object.

[0224] 33. The method of example 25, wherein generating the stimulus comprises:

[0225] Processing the basic image into segmented display objects;

[0226] For each segmented display object, generating an overlay object;

[0227] For each overlay object, generating an HSF overlay object component and an LSF overlay object component; and

[0228] A temporal modulation is applied to the HSF superposition object component of the superposition object.

[0229] 34. The method of example 33, wherein generating the HSF overlay object component and the LSF overlay object component comprises filtering the HSF component and the LSF component of the overlay object, respectively.

[0230] 35. A method according to Example 25, wherein determining the focus point includes associating the received neural signal with a display signal used to modulate the object.

[0231] 36. A method according to Example 25, wherein at least one visual stimulus corresponds to at least one control item, the control item or each control item being associated with one or more user-selectable actions.

[0232] 37. The method of example 36, further comprising:

[0233] determining the user's intended action based on the focus point and the at least one control item; and

[0234] Implement the action planned by the user.

[0235] 38. The method of example 25, wherein obtaining a base image comprises:

[0236] An image capture device is operated to capture the base image.

[0237] 39. The method of example 25, wherein obtaining a base image comprises:

[0238] Image data generated by an application executed in the processing device is received, and the received data is used as the basic image.

[0239] 40. The method of example 25, wherein displaying the or each visual stimulus comprises:

[0240] combining the or each visual stimulus with the base image to provide a mixed image; and

[0241] The mixed image is displayed via the GUI.

[0242] 41. A method according to Example 38, wherein the display is a transparent display, through which a field of view aligned with the field of view of the image capture device is presented to a user, and wherein displaying the or each visual stimulus includes displaying the or each visual stimulus within the field of view.

[0243] 42. An apparatus comprising:

[0244] a display configured to present a control interface to a user;

[0245] a neural signal capturing device configured to capture neural signals associated with the user;

[0246] An interface device, operably coupled to the display and the neural signal capturing device, the interface device comprising:

[0247] Memory; and

[0248] a processor operably coupled to the memory and configured to:

[0249] receiving the neural signal from the neural signal capturing device;

[0250] generating a stimulus and presenting the stimulus to the user via the control interface, the stimulus comprising a set of controls associated with a set of actions;

[0251] determining a focus of the user based on the neural signal, the focus point being associated with at least one control item in the set of controls;

[0252] determining the user's intended action based on the focus point and the at least one control item; and

[0253] Implement the action planned by the user.

[0254] 43. A computing device, the computing device comprising:

[0255] Processor; and

[0256] a memory storing instructions that, when executed by the processor, configure the device to:

[0257] Get the basic image;

[0258] generating at least one visual stimulus, each stimulus having a characteristic modulation applied to a high spatial frequency (HSF) component of the visual stimulus;

[0259] displaying the or each visual stimulus via a graphical user interface GUI of a display;

[0260] receiving a user's neural signal from a neural signal capturing device; and

[0261] As the user views the or each visual stimulus, a point of focus of the user in the viewed image is determined based on the neural signal, the neural signal comprising information associated with a characteristic modulation of the visual stimulus to which the user's visual attention is directed.

[0262] 44. The computing device of example 43, wherein generating the stimulus comprises:

[0263] processing at least a portion of the base image to generate low spatial frequency (LSF) map data and HSF map data; and

[0264] It is determined whether the HSF map data satisfies a predetermined spatial frequency threshold T1.

[0265] 45. The computing device of example 43, wherein processing at least a portion of the base image comprises:

[0266] applying at least one of spatial frequency filtering or spatial frequency transform;

[0267] storing features of at least a portion of the base image having a spatial frequency below a predetermined low spatial frequency threshold in the low spatial frequency map data; and

[0268] Features of at least a portion of the base image having a spatial frequency above a predetermined high spatial frequency threshold are stored in the high spatial frequency map data.

[0269] 46. ​​The computing device of example 43, wherein the HSF map data does not satisfy the predetermined spatial frequency threshold T1, and

[0270] Wherein, generating the stimulus further comprises:

[0271] generating enhanced HSF map data to introduce additional HSF stimuli into the visual stimulus; and

[0272] Temporal modulation is applied to the enhanced HSF map data.

[0273] 47. The computing device of example 46, wherein generating the enhanced HSF map data comprises:

[0274] determining whether the brightness of at least a portion of the base image falls within a predetermined range; and

[0275] When the brightness of at least a portion of the base image falls within the predetermined range, a HSF noise map is generated to be used as the enhanced HSF map data.

[0276] 48. The computing device of example 47, wherein generating enhanced HSF map data when the brightness of at least a portion of the base image does not fall within the predetermined range comprises:

[0277] Generate a gray overlay map; and

[0278] generating an HSF noise map for use as the enhanced HSF map data, and

[0279] Wherein, displaying the visual stimulus or each visual stimulus comprises:

[0280] merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and

[0281] The mixed image is displayed via the GUI.

[0282] 49. The computing device of example 47, wherein generating enhanced HSF map data when the brightness of at least a portion of the base image does not fall within the predetermined range comprises:

[0283] Generate a gray overlay map;

[0284] generating a hard noise map; and

[0285] replacing the HSF map data with data from the hard noise map for use as the enhanced HSF map data, and

[0286] Wherein, displaying the visual stimulus or each visual stimulus comprises:

[0287] merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and

[0288] The mixed image is displayed via the GUI.

[0289] 50. The computing device of example 43, wherein generating the stimulus comprises:

[0290] Processing the basic image into segmented display objects;

[0291] For each segmented display object, filtering the segmented display object into a high spatial frequency HSF component and a low spatial frequency LSF component; and

[0292] Temporal modulation is applied to the HSF components of the segmented display object.

[0293] 51. The computing device of example 43, wherein generating the stimulus comprises:

[0294] Processing the basic image into segmented display objects;

[0295] For each segmented display object, generating an overlay object;

[0296] For each overlay object, generating an HSF overlay object component and an LSF overlay object component; and

[0297] A temporal modulation is applied to the HSF superposition object component of the superposition object.

[0298] 52. A computing device according to example 51, wherein generating the HSF overlay object component and the LSF overlay object component includes filtering the HSF component and the LSF component of the overlay object respectively.

[0299] 53. A computing device according to example 43, wherein determining the focus point includes associating the received neural signal with a display signal used to modulate the object.

[0300] 54. A computing device according to example 43, wherein at least one visual stimulus corresponds to at least one control item, the control item or each control item being associated with one or more user-selectable actions.

[0301] 55. The computing device of example 54, wherein the instructions further configure the device to:

[0302] determining the user's intended action based on the focus point and the at least one control item; and

[0303] Implement the action planned by the user.

[0304] 56. The computing device of example 43, wherein obtaining the base image comprises:

[0305] An image capture device is operated to capture the base image.

[0306] 57. The computing device of example 43, wherein obtaining the base image comprises:

[0307] Image data generated by an application executed in the processing device is received, and the received data is used as the basic image.

[0308] 58. The computing device of example 43, wherein displaying the or each visual stimulus comprises:

[0309] combining the or each visual stimulus with the base image to provide a mixed image; and

[0310] The mixed image is displayed via the GUI.

[0311] 59. A computing device according to Example 56, wherein the display is a transparent display, through which a field of view aligned with the field of view of the image capture device is presented to a user, and wherein displaying the or each visual stimulus includes displaying the or each visual stimulus within the field of view.

[0312] 60. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising:

[0313] Get the basic image;

[0314] generating at least one visual stimulus, each stimulus having a characteristic modulation applied to a high spatial frequency (HSF) component of the visual stimulus;

[0315] displaying the or each visual stimulus via a graphical user interface GUI of a display;

[0316] receiving a user's neural signal from a neural signal capturing device; and

[0317] As the user views the or each visual stimulus, a point of focus of the user in the viewed image is determined based on the neural signal, the neural signal comprising information associated with a characteristic modulation of the visual stimulus to which the user's visual attention is directed.

[0318] 61. The computer-readable storage medium of example 60, wherein generating the stimulus comprises:

[0319] processing at least a portion of the base image to generate low spatial frequency (LSF) map data and HSF map data; and

[0320] It is determined whether the HSF map data satisfies a predetermined spatial frequency threshold T1.

[0321] 62. The computer-readable storage medium of example 60, wherein processing at least a portion of the base image comprises:

[0322] applying at least one of spatial frequency filtering or spatial frequency transform;

[0323] storing features of at least a portion of the base image having a spatial frequency below a predetermined low spatial frequency threshold in the low spatial frequency map data; and

[0324] Features of at least a portion of the base image having a spatial frequency above a predetermined high spatial frequency threshold are stored in the high spatial frequency map data.

[0325] 63. The computer readable storage medium of example 60, wherein the HSF map data does not satisfy the predetermined spatial frequency threshold T1, and

[0326] Wherein, generating the stimulus further comprises:

[0327] generating enhanced HSF map data to introduce additional HSF stimuli into the visual stimulus; and

[0328] Temporal modulation is applied to the enhanced HSF map data.

[0329] 64. The computer readable storage medium of example 63, wherein generating enhanced HSF map data comprises:

[0330] determining whether the brightness of at least a portion of the base image falls within a predetermined range; and

[0331] When the brightness of at least a portion of the base image falls within the predetermined range, a HSF noise map is generated to be used as the enhanced HSF map data.

[0332] 65. The computer readable storage medium of example 64, wherein generating enhanced HSF map data when a brightness of at least a portion of the base image does not fall within the predetermined range comprises:

[0333] Generate a gray overlay map; and

[0334] generating an HSF noise map for use as the enhanced HSF map data, and

[0335] Wherein, displaying the visual stimulus or each visual stimulus comprises:

[0336] merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and

[0337] The mixed image is displayed via the GUI.

[0338] 66. The computer readable storage medium of example 64, wherein generating enhanced HSF map data when a brightness of at least a portion of the base image does not fall within the predetermined range comprises:

[0339] Generate a gray overlay map;

[0340] generating a hard noise map; and

[0341] replacing the HSF map data with data from the hard noise map for use as the enhanced HSF map data, and

[0342] Wherein, displaying the visual stimulus or each visual stimulus comprises:

[0343] merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and

[0344] The mixed image is displayed via the GUI.

[0345] 67. The computer-readable storage medium of example 60, wherein generating the stimulus comprises:

[0346] Processing the basic image into segmented display objects;

[0347] For each segmented display object, filtering the segmented display object into a high spatial frequency HSF component and a low spatial frequency LSF component; and

[0348] Temporal modulation is applied to the HSF components of the segmented display object.

[0349] 68. The computer-readable storage medium of example 60, wherein generating the stimulus comprises:

[0350] Processing the basic image into segmented display objects;

[0351] For each segmented display object, generating an overlay object;

[0352] For each overlay object, generating an HSF overlay object component and an LSF overlay object component; and

[0353] A temporal modulation is applied to the HSF superposition object component of the superposition object.

[0354] 69. The computer-readable storage medium of example 68, wherein generating the HSF overlay object component and the LSF overlay object component comprises filtering the HSF component and the LSF component of the overlay object, respectively.

[0355] 70. The computer-readable storage medium of example 60, wherein determining the focus point comprises associating the received neural signal with a display signal for modulating the object.

[0356] 71. A computer-readable storage medium according to example 60, wherein at least one visual stimulus corresponds to at least one control item, and the control item or each control item is associated with one or more user-selectable actions.

[0357] 72. The computer-readable storage medium of example 71, wherein the instructions further configure the computer to:

[0358] determining the user's intended action based on the focus point and the at least one control item; and

[0359] Implement the action planned by the user.

[0360] 73. The computer readable storage medium of example 60, wherein obtaining the base image comprises:

[0361] An image capture device is operated to capture the base image.

[0362] 74. The computer readable storage medium of example 60, wherein obtaining the base image comprises:

[0363] Image data generated by an application executed in the processing device is received, and the received data is used as the basic image.

[0364] 75. The computer readable storage medium of example 60, wherein displaying the or each visual stimulus comprises:

[0365] combining the or each visual stimulus with the base image to provide a mixed image; and

[0366] The mixed image is displayed via the GUI.

[0367] 76. Computer-readable storage medium according to Example 73, wherein the display is a transparent display, through which a field of view aligned with the field of view of the image capture device is presented to a user, and wherein displaying the or each visual stimulus includes displaying the or each visual stimulus within the field of view.

[0368] Other particular and preferred aspects of the present disclosure are set out in the accompanying independent and dependent claims.It will be appreciated that features of the dependent claims may be combined with features of the independent claims in combinations other than those explicitly set out in the claims.

Claims

1. A method for tracking visual attention, the method comprising: filtering the graphic data of the screen object to obtain a basic image; identifying an object of interest within the base image; generating at least one visual stimulus by processing at least a portion of the base image including the object of interest to extract a high spatial frequency (HSF) component of the base image, each stimulus having a characteristic modulation applied to the HSF component of the base image, the HSF component having a spatial frequency above a predetermined high spatial frequency threshold; displaying the or each visual stimulus via a graphical user interface GUI of a display; receiving a neural signal of a user from a neural signal capturing device; as well as As the user views the or each visual stimulus, a point of focus of the user in the viewed image is determined based on the neural signal, the neural signal comprising information associated with a characteristic modulation of the visual stimulus to which the user's visual attention is directed.

2. The method according to claim 1, wherein: Generating the at least one visual stimulus comprises: At least a portion of the base image including the object of interest is processed to generate a low spatial frequency (LSF) component of the base image, the LSF component having a spatial frequency below a predetermined low spatial frequency threshold.

3. The method according to claim 2, wherein: Processing at least a portion of the base image including the object of interest to extract the HSF component and the LSF component comprises: applying at least one of a spatial frequency filter or a spatial frequency transform to at least a portion of the base image including the object of interest; and The LSF component and the HSF component are stored as LSF map data and HSF map data, respectively.

4. The method according to claim 3, wherein: Generating the stimulus further comprises: determining whether the HSF map data satisfies a predetermined spatial frequency threshold T1, and if the HSF map data does not satisfy the predetermined spatial frequency threshold: generating enhanced HSF map data to introduce additional HSF stimuli into the visual stimulus; and Temporal modulation is applied to the enhanced HSF map data.

5. The method according to claim 4, wherein: Generating enhanced HSF map data includes: determining whether the brightness of at least a portion of the base image including the object of interest falls within a predetermined range; and When the brightness of at least a portion of the base image including the object of interest falls within the predetermined range, a HSF noise map is generated to be used as the enhanced HSF map data.

6. The method according to claim 5, wherein: Generating enhanced HSF map data when the brightness of at least a portion of the base image including the object of interest does not fall within the predetermined range comprises: Generate a gray overlay map; and generating an HSF noise map for use as the enhanced HSF map data, and Wherein, displaying the visual stimulus or each visual stimulus comprises: merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and The mixed image is displayed via the GUI.

7. The method according to claim 5, wherein: Generating enhanced HSF map data when the brightness of at least a portion of the base image including the object of interest does not fall within the predetermined range comprises: Generate a gray overlay map; generating a hard noise map; and replacing the HSF map data with data from the hard noise map for use as the enhanced HSF map data, and Wherein, displaying the visual stimulus or each visual stimulus comprises: merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and The mixed image is displayed via the GUI.

8. The method according to claim 1, wherein: Generating the stimulus comprises: Processing the basic image into segmented display objects; For each segmented display object, filtering the segmented display object into a high spatial frequency HSF component and a low spatial frequency LSF component; and Temporal modulation is applied to the HSF components of the segmented display object.

9. The method according to claim 1, wherein: Generating the stimulus comprises: Processing the basic image into segmented display objects; For each segmented display object, generating an overlay object; For each overlay object, generating an HSF overlay object component and an LSF overlay object component; and A temporal modulation is applied to the HSF superposition object component of the superposition object.

10. The method according to claim 9, wherein: Generating the HSF superimposed object component and the LSF superimposed object component includes filtering the HSF component and the LSF component of the superimposed object, respectively.

11. The method according to claim 9, wherein: Determining the focus point includes correlating the received neural signal with a display signal used to modulate the display object.

12. The method according to claim 1, wherein: At least one visual stimulus corresponds to at least one control item, the or each control item being associated with one or more user-selectable actions.

13. The method according to claim 12, further comprising: determining the user's intended action based on the focus point and the at least one control item; as well as Implement the action planned by the user.

14. The method according to claim 1, wherein: Getting the basic image includes: An image capture device is operated to capture the base image.

15. The method according to claim 1, wherein: Getting the basic image includes: Image data generated by an application executed in the processing device is received, and the received data is used as the basic image.

16. The method according to claim 1, wherein: Displaying the or each visual stimulus comprises: combining the or each visual stimulus with the base image to provide a mixed image; and The mixed image is displayed via the GUI.

17. The method according to claim 14, wherein: The display is a transparent display through which a field of view aligned with a field of view of the image capture device is presented to a user, and wherein displaying the or each visual stimulus comprises displaying the or each visual stimulus within the field of view.

18. A computing device, the computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the device to: filtering the graphic data of the screen object to obtain a basic image; identifying an object of interest within the base image; generating at least one visual stimulus by processing at least a portion of the base image including the object of interest to extract a high spatial frequency (HSF) component of the base image, each stimulus having a characteristic modulation applied to the HSF component of the base image, the HSF component having a spatial frequency above a predetermined high spatial frequency threshold; displaying the or each visual stimulus via a graphical user interface GUI of a display; receiving a neural signal of a user from a neural signal capturing device; as well as As the user views the or each visual stimulus, a point of focus of the user in the viewed image is determined based on the neural signal, the neural signal comprising information associated with a characteristic modulation of the visual stimulus to which the user's visual attention is directed.

19. The computing device of claim 18, wherein: Generating the at least one stimulus comprises: At least a portion of the base image including the object of interest is processed to generate a low spatial frequency (LSF) component of the base image, the LSF component having a spatial frequency below a predetermined low spatial frequency threshold.

20. The computing device of claim 19, wherein: Processing at least a portion of the base image including the object of interest to extract the HSF component and the LSF component comprises: applying at least one of a spatial frequency filter or a spatial frequency transform to at least a portion of the base image including the object of interest; and The LSF component and the HSF component are stored as LSF map data and HSF map data, respectively.

21. The computing device of claim 19, wherein: Generating the stimulus further comprises: determining whether the HSF map data satisfies a predetermined spatial frequency threshold T1, and if the HSF map data does not satisfy the predetermined spatial frequency threshold: generating enhanced HSF map data to introduce additional HSF stimuli into the visual stimulus; and Temporal modulation is applied to the enhanced HSF map data.

22. The computing device of claim 21, wherein: Generating enhanced HSF map data includes: determining whether the brightness of at least a portion of the base image including the object of interest falls within a predetermined range; and When the brightness of at least a portion of the base image including the object of interest falls within the predetermined range, a HSF noise map is generated to be used as the enhanced HSF map data.

23. The computing device of claim 22, wherein: Generating enhanced HSF map data when the brightness of at least a portion of the base image including the object of interest does not fall within the predetermined range comprises: Generate a gray overlay map; and generating an HSF noise map for use as the enhanced HSF map data, and Wherein, displaying the visual stimulus or each visual stimulus comprises: merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and The mixed image is displayed via the GUI.

24. The computing device of claim 22, wherein: Generating enhanced HSF map data when the brightness of at least a portion of the base image including the object of interest does not fall within the predetermined range comprises: Generate a gray overlay map; generating a hard noise map; and replacing the HSF map data with data from the hard noise map for use as the enhanced HSF map data, and Wherein, displaying the visual stimulus or each visual stimulus comprises: merging the enhanced HSF map data with the gray overlay map to provide a hybrid image; and The mixed image is displayed via the GUI.

25. The computing device of claim 18, wherein: Generating the stimulus comprises: Processing the basic image into segmented display objects; For each segmented display object, filtering the segmented display object into a high spatial frequency HSF component and a low spatial frequency LSF component; and Temporal modulation is applied to the HSF components of the segmented display object.

26. The computing device of claim 18, wherein: Generating the stimulus comprises: Processing the basic image into segmented display objects; For each segmented display object, generating an overlay object; For each overlay object, generating an HSF overlay object component and an LSF overlay object component; and A temporal modulation is applied to the HSF superposition object component of the superposition object.

27. The computing device of claim 26, wherein: Generating the HSF superimposed object component and the LSF superimposed object component includes filtering the HSF component and the LSF component of the superimposed object, respectively.

28. The computing device of claim 26, wherein: Determining the focus point includes correlating the received neural signal with a display signal used to modulate the display object.

29. The computing device of claim 18, wherein: At least one visual stimulus corresponds to at least one control item, the or each control item being associated with one or more user-selectable actions.

30. The computing device of claim 29, wherein: The instructions also configure the device to: determining the user's intended action based on the focus point and the at least one control item; and Implement the action planned by the user.

31. The computing device of claim 18, wherein: Getting a basic image includes: An image capture device is operated to capture the base image.

32. The computing device of claim 18, wherein: Getting a basic image includes: Image data generated by an application executed in the processing device is received, and the received data is used as the basic image.

33. The computing device of claim 18, wherein: Displaying the or each visual stimulus comprises: combining the or each visual stimulus with the base image to provide a mixed image; and The mixed image is displayed via the GUI.

34. The computing device of claim 31 , wherein: The display is a transparent display through which a field of view aligned with a field of view of the image capture device is presented to a user, and wherein displaying the or each visual stimulus comprises displaying the or each visual stimulus within the field of view.

35. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: filtering the graphic data of the screen object to obtain a basic image; identifying an object of interest within the base image; generating at least one visual stimulus by processing at least a portion of the base image including the object of interest to extract a high spatial frequency (HSF) component of the base image, each stimulus having a characteristic modulation applied to the HSF component of the base image, the HSF component having a spatial frequency above a predetermined high spatial frequency threshold; displaying the or each visual stimulus via a graphical user interface GUI of a display; receiving a user's neural signal from a neural signal capturing device; and As the user views the or each visual stimulus, a point of focus of the user in the viewed image is determined based on the neural signal, the neural signal comprising information associated with a characteristic modulation of the visual stimulus to which the user's visual attention is directed.

Images