Portable communication device

Abstract

The present disclosure relates to a portable communication device (301). The portable communication device comprises a controller (302) configured to receive (210) first and second images from one or more cameras (340) and to transmit images to a video display (314). The portable communication device also comprises an electroencephalogram (EEG) device (350) configured to capture EEG signals of a brain of a user, wherein the controller is configured to receive the EEG signals from the EEG device. The controller is configured for detecting (220) an object on the received first images, displaying (230) a word associated with the detected object on the video display, determining (510) a gaze point of one or more eyes of the user based on second images of cameras, wherein the second images relate to the one or more eyes of the user, verifying (520) the determined gaze point based on the EEG signals, and determining (540) a word or a function selected on the video display based on the gaze point.

Description

PORTABLE COMMUNICATION DEVICE

[0001] The present application relates to Australian Provisional Patent Application No. 2024900238, the content of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present invention relates generally to a communication device. The present invention also relates to a method for operating the communication device, and to a computer program product including a computer readable medium having recorded thereon a computer program for the communication device. Background

[0003] People with conditions such as cerebral palsy, autism spectrum disorder, amyotrophic lateral sclerosis, and the like may require assistance in communicating.

[0004] In a conventional arrangement, such people may use desktop-based communication device having an augmentative and alternative communication (AAC) system. The AAC system provides methods of assisting the user for generating sentences and speaking out the generated sentences. However, such a conventional arrangement lacks privacy, portability, and robustness. Summary

[0005] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0006] Disclosed are arrangements which seek to address the above problems by providing a portable device providing an augmentative and alternative communication (AAC) system. The portable communication device in accordance with the present disclosure is capable of capturing images from the point of view of a user. The portable communication device then detects objects within the captured images and presents the detected objects as words on a display to the user. The portable communication device then tracks the eye movements of the user and monitors the electroencephalogram (EEG) signals of the user. Based on the tracked eye movements and EEG signals, the portable communication device determines a word on the display that the user intends AH25(45458020)to select. A sentence can then be generated from the selection of a sequence of the words. A voice generator within the portable communication device then speaks the sentence.

[0007] Such a portable communication device can be worn by the user and provides a method for the user to communicate in a more independent manner.

[0008] According to an aspect of the present disclosure, there is provided a portable communication device, comprising: a controller configured to receive first and second images from one or more cameras and to transmit images to a video display; an electroencephalogram (EEG) device configured to capture EEG signals of a brain of a user, wherein the controller is configured to receive the EEG signals from the EEG device; wherein the controller is configured for: detecting an object on the received first images; displaying a word associated with the detected object on the video display; determining a gaze point of one or more eyes of the user based on second images of cameras, wherein the second images relate to the one or more eyes of the user; verifying the determined gaze point based on the EEG signals; determining a word or a function selected on the video display based on the gaze point.

[0009] According to another aspect of the present disclosure, there is provided a communication method comprising: detecting, by a controller, an object on first images captured by a first camera; displaying, by the controller, a word associated with the detected object on a video display; determining, by the controller, a gaze point of one or more eyes of a user based on second images captured by a second camera, wherein the second images relate to the one or more eyes of a user; verifying, by the controller, the determined gaze point based on electroencephalogram (EEG) signals received from an EEG device; and determining, by the controller, a word or a function selected on the video display based on the gaze point.

[0010] According to another aspect of the present disclosure, there is provided a computer program product including a computer readable medium having recorded thereon a computer program for implementing any one of the methods described above.

[0011] Other aspects are also disclosed. Brief Description of the Drawings

[0012] At least one embodiment of the present invention will now be described with reference to the drawings, in which: AH25(45458020)

[0013] Figs.1A and 1B show a portable communication device in accordance with the present disclosure;

[0014] Fig.2 is a flow diagram of a method of operating the portable communication device shown in Figs.1A and 1B;

[0015] Fig.3 is a flow diagram of a word display sub-process of the method shown in Fig.2;

[0016] Fig.4 shows an example of an embodiment of the sub-process shown in Fig.3;

[0017] Fig.5 is a flow diagram of a word selection sub-process of the method shown in Fig.2;

[0018] Fig.6 shows an example of the device of Figs.1A and 1B;

[0019] Fig.7 is a diagram for determining eye gaze; and

[0020] Fig.8 shows a method for determining brain wave spike. Detailed Description including Best Mode

[0021] Where reference is made in any one or more of the accompanying drawings to steps and / or features, which have the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

[0022] Figs.1A and 1B collectively form a schematic block diagram of a portable communication device 301 including embedded components, upon which the communication methods to be described are desirably practiced. The portable communication device 301 includes at least an electroencephalogram (EEG) device 350, cameras 340, and a video display 314. In one arrangement, the cameras 340 and video display 314 are components of an extended reality (XR) device. When the cameras 340 and video display 314 are parts of an XR device, then the controller 302 (see below) of the device 301 is configured to electrically communicate with the XR device. The controller 302 when connected to the XR device is then able to receive images (e.g., a video or separate images) from the cameras 340 and to transmit images (e.g., a video or separate images) to the video display 314.

[0023] Fig.6 shows one arrangement of the device 301. As shown in Fig.6, the cameras 340 and the video display 314 are disposed in the proximity of the eyes of the user. AH25(45458020)

[0024] One or more of the cameras 340 captures images of the eyes of a user wearing the portable communication device 301. For ease of reference, the cameras 340 capturing the images of the eyes of the user will be referred to herein as cameras 340EYES.

[0025] Another one or more of the cameras 340 captures images that are associated with the point of view of the user wearing the device 301. For ease of reference, the cameras 340 capturing the images that are associated with the point of view of the user will be referred to herein as cameras 340POV.

[0026] The video display 314 provides videos / images captured by cameras 340POV. The video display 314 also overlays the videos / images with additional data. The additional data includes words associated with objects that are detected on the videos / images.

[0027] The EEG device 350 is configured to be worn by the user on his / her head. The EEG device 350 then captures EEG signals of the brain of the user. In one arrangement, the EEG device 350 includes 22 electrodes with a sampling rate of 250 Hz. The EEG device 350 also includes a notch filter of 48 to 52 Hz and a band-pass filter of 0.2 to 60 Hz. The signals from the EEG device 350 are processed by method 800 (described hereinafter).

[0028] As seen in Fig.1A, the device 301 also comprises an embedded controller 302. The controller 302 has a processing unit (or processor) 305 which is bi-directionally coupled to an internal storage module 309. The storage module 309 may be formed from non-volatile semiconductor read only memory (ROM) 360 and semiconductor random access memory (RAM) 370, as seen in Fig.1B. The RAM 370 may be volatile, non-volatile or a combination of volatile and non-volatile memory.

[0029] The device 301 includes a display controller 307, which is connected to the video display 314, such as a liquid crystal display (LCD) panel or the like. The display controller 307 is configured for displaying graphical images on the video display 314 in accordance with instructions received from the embedded controller 302, to which the display controller 307 is connected.

[0030] The device 301 also includes user input devices 313 which are typically formed by keys, a keypad or like controls. In some implementations, the user input devices 313 may include a touch sensitive panel physically associated with the display 314 to collectively form a touch-screen. Such a touch-screen may thus operate as one form of graphical user interface (GUI) as opposed to a prompt or menu driven GUI typically used with keypad-display AH25(45458020)combinations. Other forms of user input devices may also be used, such as a microphone (not illustrated) for voice commands or a joystick / thumb wheel (not illustrated) for ease of navigation about menus.

[0031] As seen in Fig.1A, the device 301 also comprises a portable memory interface 306, which is coupled to the processor 305 via a connection 319. The portable memory interface 306 allows a complementary portable memory device 325 to be coupled to the electronic device 301 to act as a source or destination of data or to supplement the internal storage module 309. Examples of such interfaces permit coupling with portable memory devices such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, Personal Computer Memory Card International Association (PCMIA) cards, optical disks and magnetic disks.

[0032] The device 301 also has a communications interface 308 to permit coupling of the device 301 to a computer or communications network 320 via a connection 321. The connection 321 may be wired or wireless. For example, the connection 321 may be radio frequency or optical. An example of a wired connection includes Ethernet. Further, an example of wireless connection includes BluetoothTMtype local interconnection, Wi-Fi (including protocols based on the standards of the IEEE 802.11 family), Infrared Data Association (IrDa) and the like.

[0033] The methods described hereinafter may be implemented using the embedded controller 302, where the processes of Figs.2, 3, 5, and 8 may be implemented as one or more software application programs 333 executable within the embedded controller 302. The device 301 of Fig.1A implements the described methods. In particular, with reference to Fig.1B, the steps of the described methods are effected by instructions in the software 333 that are carried out within the controller 302. The software instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

[0034] The software 333 of the embedded controller 302 is typically stored in the non-volatile ROM 360 of the internal storage module 309. The software 333 stored in the ROM 360 can be updated when required from a computer readable medium. The software 333 can be loaded into and executed by the processor 305. In some instances, the processor 305 may execute software instructions that are located in RAM 370. Software instructions may be loaded into the AH25(45458020)RAM 370 by the processor 305 initiating a copy of one or more code modules from ROM 360 into RAM 370. Alternatively, the software instructions of one or more code modules may be pre- installed in a non-volatile region of RAM 370 by a manufacturer. After one or more code modules have been located in RAM 370, the processor 305 may execute software instructions of the one or more code modules.

[0035] The application program 333 is typically pre-installed and stored in the ROM 360 by a manufacturer, prior to distribution of the device 301. However, in some instances, the application programs 333 may be supplied to the user encoded on one or more CD-ROM (not shown) and read via the portable memory interface 306 of Fig.1A prior to storage in the internal storage module 309 or in the portable memory 325. In another alternative, the software application program 333 may be read by the processor 305 from the network 320, or loaded into the controller 302 or the portable storage medium 325 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that participates in providing instructions and / or data to the controller 302 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, flash memory, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the device 301. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the device 301 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. A computer readable medium having such software or computer program recorded on it is a computer program product.

[0036] The second part of the application programs 333 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 314 of Fig.1A. Through manipulation of the user input device 313 (e.g., the keypad) or the communication methods described in Figs, 2, 3, and 5, a user of the device 301 and the application programs 333 may manipulate the interface in a functionally adaptable manner to provide controlling commands and / or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via loudspeakers (not illustrated) and user voice commands input via the microphone (not illustrated). AH25(45458020)

[0037] Fig.1B illustrates in detail the embedded controller 302 having the processor 305 for executing the application programs 333 and the internal storage 309. The internal storage 309 comprises read only memory (ROM) 360 and random access memory (RAM) 370. The processor 305 is able to execute the application programs 333 stored in one or both of the connected memories 360 and 370. When the electronic device 301 is initially powered up, a system program resident in the ROM 360 is executed. The application program 333 permanently stored in the ROM 360 is sometimes referred to as “firmware”. Execution of the firmware by the processor 305 may fulfil various functions, including processor management, memory management, device management, storage management and user interface.

[0038] The processor 305 typically includes a number of functional modules including a control unit (CU) 351, an arithmetic logic unit (ALU) 352, a digital signal processor (DSP) 353 and a local or internal memory comprising a set of registers 354 which typically contain atomic data elements 356, 357, along with internal buffer or cache memory 355. One or more internal buses 359 interconnect these functional modules. The processor 305 typically also has one or more interfaces 358 for communicating with external devices via system bus 381, using a connection 361.

[0039] The application program 333 includes a sequence of instructions 362 though 363 that may include conditional branch and loop instructions. The program 333 may also include data, which is used in execution of the program 333. This data may be stored as part of the instruction or in a separate location 364 within the ROM 360 or RAM 370.

[0040] In general, the processor 305 is given a set of instructions, which are executed therein. This set of instructions may be organised into blocks, which perform specific tasks or handle specific events that occur in the device 301. Typically, the application program 333 waits for events and subsequently executes the block of code associated with that event. Events may be triggered in response to input from a user, via the user input devices 313 of Fig.1A or the communication methods described in Figs.2, 3, 5, and 8, as detected by the processor 305. Events may also be triggered in response to other sensors and interfaces in the electronic device 301.

[0041] The execution of a set of the instructions may require numeric variables to be read and modified. Such numeric variables are stored in the RAM 370. The disclosed method uses input variables 371 that are stored in known locations 372, 373 in the memory 370. The input variables 371 are processed to produce output variables 377 that are stored in known locations 378, 379 in the memory 370. Intermediate variables 374 may be stored in additional AH25(45458020)memory locations in locations 375, 376 of the memory 370. Alternatively, some intermediate variables may only exist in the registers 354 of the processor 305.

[0042] The execution of a sequence of instructions is achieved in the processor 305 by repeated application of a fetch-execute cycle. The control unit 351 of the processor 305 maintains a register called the program counter, which contains the address in ROM 360 or RAM 370 of the next instruction to be executed. At the start of the fetch execute cycle, the contents of the memory address indexed by the program counter is loaded into the control unit 351. The instruction thus loaded controls the subsequent operation of the processor 305, causing for example, data to be loaded from ROM memory 360 into processor registers 354, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to the location stored in another register and so on. At the end of the fetch execute cycle the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed this may involve incrementing the address contained in the program counter or loading the program counter with a new address in order to achieve a branch operation.

[0043] Each step or sub-process in the processes of the methods described below is associated with one or more segments of the application program 333, and is performed by repeated execution of a fetch-execute cycle in the processor 305 or similar programmatic operation of other independent processor blocks in the electronic device 301.

[0044] Fig.2 shows a flow diagram of a method 100 of operating the device 301 to provide the AAC system. The method 100 is a software application program 333 executable within the embedded controller 302, as described hereinbefore. In an alternative arrangement, certain steps of the method 100 are performed by a server in electrical communication with the device 301 via the network 320. For example, if certain steps require processing power, it may be faster to transmit the data to be processed to the server as the server would have higher processing power than the processor 305.

[0045] In one arrangement, the method 100 commences when a user of the device 301 presses a power on button.

[0046] The method 100 commences at a word display sub-process 200. Fig.3 shows a flow diagram of the sub-process 200, which will be described below. The sub-process 200 receives AH25(45458020)first images from the cameras 340POV, detects objects within the received first images, and displays the detected objects as words on the video display 314. The term “first images” is associated with the images captured by the cameras 340POV.

[0047] Fig.4 shows one example of such a display of words 410, 412, 420 associated with the detected objects. Fig.4 will be discussed in more detail when discussing the sub-process 200 hereinafter.

[0048] The method 100 then proceeds from the sub-process 200 to a function selection sub- process 400. Fig.5 shows a flow diagram of the sub-process 400. The sub-process 400 receives second images from the cameras 340EYES, and determines the gaze point of one or more eyes of the user based on the received second images and the EEG device 350. The term “second images” is associated with the images captured by the cameras 340EYES.

[0049] Based on the determined gaze point, the sub-process 400 determines the function (e.g., word 410, 412, 414, functions 420, 430) selected on the video display 314. Fig.4 shows words 410, 412, 414 and functions 420, 430 displayed on the video display 314. If the sub- process 400 determines that the user’s gaze point is related to any of the words 410, 412, or 414, then a word has been selected. If the sub-process 400 determines that the user’s gaze point is related to the speak function 420, then the speak function is selected. If the sub- process 400 determines that the user’s gaze point is related to the delete function 430, then the delete function is selected.

[0050] The sub-process 400 returns the function (i.e., word, speak function, delete function) selected to the method 100. The sub-process 400 will be described in more detail hereinafter.

[0051] Once the sub-process 400 is completed, control returns to the method 100. The method 100 then proceeds from sub-process 400 to step 120.

[0052] In step 120, the method 100 determines whether the speak function 420 (see Fig.4) is selected at the sub-process 400. If the speak function 420 is selected (YES), then the method 100 proceeds from step 120 to step 130. However, if the speak function 420 is not selected (NO), then the method 100 proceeds from step 120 to step 160.

[0053] The discussion now turns to step 160 and its subsequent steps before discussing step 130. AH25(45458020)

[0054] In step 160, the method 100 determines whether a word has been selected or deleted. If a word has been selected (SELECTED), then the method 100 proceeds from step 160 to step 170. If a word has been deleted (DELETED), then the method 100 proceeds from step 160 to step 180.

[0055] In step 170, the method 100 adds the selected word to a sentence. If there is no sentence pending, then a new sentence is created with the selected word being the first word of the sentence. The method 100 then proceeds from step 170 to the sub-process 400, to select the next function (i.e., add a word, delete a word, speak).

[0056] In step 180, the method 100 deletes a last word of a pending sentence. If there is no pending sentence, then the method 100 displays a notification on the video display 314 that the selected function is invalid as there is no pending sentence.

[0057] In one arrangement, the pending sentence 450 is shown on the video display 314 (see Fig.4) to enable the user to review the currently pending sentence.

[0058] The method 100 then proceeds from step 180 to the sub-process 400.

[0059] The discussion now turns to step 130.

[0060] In step 130, the method 100 verifies the pending sentence by reviewing the sentence for grammatical issues. For example, the user may have formed a sentence “I want eat apple.” The method 100 in step 130 may then amend this sentence to “I want to eat an apple.”

[0061] In one arrangement, such a verification is performed using a neural network model. There are commonly available neural network models that can be adapted for use for the method 100. The neural network models need to be provided with correct language uses and neural network models are then able to verify such sentences.

[0062] The method 100 then proceeds from step 130 to step 140.

[0063] In step 140, the method 100 generates a soundtrack for the sentence verified at step 130. The method 100 then proceeds from step 140 to step 150.

[0064] In step 150, the method 100 plays the generated soundtrack via the loudspeakers of the device 301. As described hereinbefore, in one alternative arrangement, steps 130 to 150 are performed by a server in electrical communication with the device 301 via the network 320. AH25(45458020)For example, the device 301 communicates with a server via the network 320 at the conclusion of step 120. In turn, the server performs steps 130 to 150. The alternative arrangement is advantageous as some of the computer processing is performed by a server that is external to the device 301.

[0065] The method 100 concludes at the conclusion of step 150. The method 100 may offer the user with an option to generate another sentence at the conclusion of step 150.

[0066] The discussion now turns to the sub-process 200. The sub-process 200 is a software application program 333 executable within the embedded controller 302, as described hereinbefore. In the alternative arrangement, certain steps of the sub-process 200 may be performed by the server that is in electrical communication with the device 301 via the network 320.

[0067] The sub-process 200 commences at step 210 by receiving first images from the cameras 340POV. The sub-process 200 then proceeds from step 210 to step 220.

[0068] In step 220, the sub-process 200 detects objects that are within the received first images. In one arrangement, the sub-process 200 uses YOLOv5 (object detection models) to detect objects within the received first images.

[0069] Fig.4 shows an example of the sub-process 200 detecting an apple and a cup.

[0070] The sub-process 200 then proceeds from step 220 to step 230.

[0071] In step 230, the sub-process 200 displays words relating to the detected objects on the video display 314. Fig.4 shows the word 412 for the detected apple and the word 410 for the detected cup.

[0072] Fig.4 also shows common words 414 (e.g., I, it, you, she, etc.) that may be required for generating a sentence and functions that are used for the AAC system (such as speak function 420 and delete function 430). The speak function 420 provides instructions to the device 301 to speak the sentence created by the user (see steps 120 to 150 of the method 100). The delete function 430 provides instructions to the device 301 to delete the last word from the pending sentence (see steps 160 and 180 of the method 100).

[0073] The sub-process 200 concludes at the conclusion of step 230. AH25(45458020)

[0074] The discussion now turns to the sub-process 400 shown in Fig.5. The sub-process 400 is a software application program 333 executable within the embedded controller 302, as described hereinbefore. In the alternative arrangement, certain steps of the sub-process 400 may be performed by the server that is in electrical communication with the device 301 via the network 320.

[0075] The sub-process 400 commences at step 505 by detecting the eyes of the user. Such a detection is performed on the second images received from the cameras 340EYES. In one arrangement, the eye detection is performed by an algorithm that detects two circles with contrast color in monochrome filtered image. The two circles include a large circle located outside of a small circle, representing the iris and pupil of an eye respectively. If the eyes are detected (YES), the sub-process 400 proceeds from step 505 to step 510. If the eyes are not detected (NO), step 505 is repeated.

[0076] In step 510, the sub-process 400 determines, using the second images received from the cameras 340EYES, the gaze point of the eyes of the user. The relative shape, position, and distortion of iris and pupil in the second images represents the direction of the gaze. The cross- point of the gaze directions of the two eyes is the gaze point.

[0077] Before a user uses the device 301, the device 301 performs a calibration process for the user. The calibration process includes presenting, on the video display 314, a sequence of points. The device 301 uses the cameras 340EYES to capture a sequence of images of the eyes of the user as the user sequentially looks at the sequence of points. Each of the captured images includes the shape, position, and distortion (referred to as reference parameters) of the iris and pupil of each eye. Accordingly, in step 510, the eyes captured in the second images are compared against the reference data to determine the gaze direction of the eyes of the user. The cross-point of the gaze directions of the two eyes are then used to determine the gaze point.

[0078] In one arrangement, 3 or more sequential points are used to obtain the reference data.

[0079] In one arrangement, the gaze point data is transformed mathematically to transform the determined eye gaze to a point on the video display 314. Fig.7 shows an eye 710 (denoted as point O) gazing toward point B or I on the video display 314. The camera 340 however captures the gaze of the eye at point A. Accordingly, to determine the eye gaze at the video display 314, the eye gaze is transformed using the equation: AH25(45458020)where K is the scaling factor of OA to OB. The parameter K is determined during the calibration of the device 301.

[0080] The sub-process 400 then proceeds from step 510 to step 520.

[0081] In step 520, the sub-process 400 verifies the determined gaze point. The verification is performed by flashing the word / function associated with the determined gaze together with the nearest 3 to 5 words / functions in a random manner. For example, if the gaze point is determined to be directed at the speak function 420, the flashing lights are generated at the speak function 420, the delete function 430, the word “No”, and their adjacent words. The flashing lights are generated in a random manner. Simultaneously, the controller 302 of the device 301 monitors the EEG data received from the EEG device 350. When the controller 302 detects a brain wave spike, the time at which the brain wave spike occurs is associated with the word / function at which the flashing light occurs at that particular time. For example, if a brain wave spike is detected at 1 second and, at this time, the speak function 420 was flashing, then the controller 302 determines that the user intends to select the speak function 420 based on the EEG data. In one arrangement, the controller 302 performs method 800 to determine a positive brain wave spike. The method 800 is described hereinafter in relation to Fig.8.

[0082] The sub-process 400 then proceed from step 520 to step 530.

[0083] In step 530, the sub-process 400 determines whether the gaze point is accurate. The determination is performed by comparing the determined gaze point at step 510 with the gaze point determined based on the EEG data at step 520. If both determined gaze points match (YES), then the sub-process 400 proceeds from step 530 to step 540. Otherwise (NO), the sub- process 400 proceeds from step 530 to step 510.

[0084] In step 540, the sub-process 400 determines the word / function associated with the determined gaze point. The sub-process 400 then returns the selected word / function to the method 100. The sub-process 400 concludes at the conclusion of step 450. AH25(45458020)

[0085] The discussion now turns to the method 800 for processing the signals of the EEG device 350. The method 800 is a software application program 333 executable within the embedded controller 302, as described hereinbefore. In the alternative arrangement, certain steps of the method 800 may be performed by the server that is in electrical communication with the device 301 via the network 320.

[0086] The method 800 commences at step 815 for processing the signals (i.e., input 810) of the EEG device 350. One or more signal processing methods is used at step 815. The signal processing methods include feature extraction, low-pass filter, noise feature based filter (e.g., Non-target Event Related Potentials (ERP) based filter), and feature selection.

[0087] Feature extraction transforms the input 810 into a set of numerical features. Examples of feature extraction that may be used includes statistical methods, Principal Component Analysis, Linear Discriminant Analysis (LDA), Autoencoders, and the like.

[0088] Low-pass filter is used to filter out aditional noise while keeping major features as determined by the Feature Extraction method.

[0089] Non-target ERP based filter functions to mitigate the noise that has high similarity to ERP features. Non-target ERP based filter first derives a spectrum based on the average of non-target epochs from the original signals of the EEG device 350. The derivation uses the following equations:AH25(45458020)

[0094] epochs non-target epochs in frequency domain and corresponds to and is deduced from .

[0095]

[0096] is performed in the time domain. of feature selection algorithm isVector Regression (Linear-

[0097] Once the signal from the EEG device 350 is processed in step 815, the method 800 proceeds to step 820. In step 820, the method 800 performs classification on the processed signal using a neural network, such as a convolutional recurrent neural network. Examples of classification algorithms includeMachine (SVM), Latent

[0098] Table 1 below shows an example RNN structure for the classification algorithm. AH25(45458020)

[0099]

[0100] The method 800 then proceeds from step 820 to step 825.

[0101] In step 825, the method 800 performs labelling. In this case, the labelling determines the probability of a positive brain wave spike (see step 520). The labelling algorithm uses the following equations: AH25(45458020)

[0102]

[0103]

[0104] run i (e.g., a period of time) belongs to block k (e.g., brain wave spike) being 1. is the sum of all 825 to step 830.

[0105] In occurs). TheIndustrial Applicability

[0106] The are applicable to the computer and data processing industries.

[0107] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

[0108] In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of”. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings. AH25(45458020)

[0107] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

[0108] In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of”. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings.

Claims

CLAIMS:

1. A portable communication device, comprising: a controller configured to receive first and second images from one or more cameras and to transmit images to a video display; an electroencephalogram (EEG) device configured to capture EEG signals of a brain of a user, wherein the controller is configured to receive the EEG signals from the EEG device; wherein the controller is configured for: detecting an object on the received first images; displaying a word associated with the detected object on the video display; determining a gaze point of one or more eyes of the user based on second images of cameras, wherein the second images relate to the one or more eyes of the user; verifying the determined gaze point based on the EEG signals; determining a word or a function selected on the video display based on the gaze point.

2. The device of claim 1, wherein the cameras and the video display are components of an extended reality (XR) device.

3. The device of claim 2, wherein the XR device is in communication with the controller.

4. The device of any one of claims 1 to 3, wherein the controller is further configured for: generating a sentence by selecting words on the video display.

5. The device of claim 4, wherein the controller is further configured for: verifying the sentence; generating a soundtrack for the sentence; and playing the generated soundtrack.

6. The device of any one of claims 1 to 5, wherein the controller is further configured for: displaying common words on the video display for generating the sentence.

7. The device of any one of claims 1 to 6, wherein the EEG signals are processed to determine a brain wave spike. AH25(45458020)8. The device of claim 7, wherein the processing of the EEG signals is performed using a neural network.

9. A communication method comprising: detecting, by a controller, an object on first images captured by a first camera; displaying, by the controller, a word associated with the detected object on a video display; determining, by the controller, a gaze point of one or more eyes of a user based on second images captured by a second camera, wherein the second images relate to the one or more eyes of a user; verifying, by the controller, the determined gaze point based on electroencephalogram (EEG) signals received from an EEG device; and determining, by the controller, a word or a function selected on the video display based on the gaze point.

10. The method of claim 9, further comprising: generating a sentence by selecting words on the video display.

11. The method of claim 10, further comprising: verifying the sentence; generating a soundtrack for the sentence; and playing the generated soundtrack.

12. The method of any one of claims 9 to 10, further comprising: displaying common words on the video display for generating the sentence.

13. The method of any one of claims 9 to 12, comprising processing the EEG signals to determine a brain wave spike.

14. The method of claim 13, wherein the processing of the EEG signals is performed using a neural network.

15. A computer readable medium comprising a computer program executable by a controller, wherein, when the computer program is executed by the controller, the controller performs a communication method, the communication method comprising: AH25(45458020)detecting, by the controller, an object on first images captured by a first camera; displaying, by the controller, a word associated with the detected object on a video display; determining, by the controller, a gaze point of one or more eyes of a user based on second images captured by a second camera, wherein the second images relate to the one or more eyes of a user; verifying, by the controller, the determined gaze point based on electroencephalogram (EEG) signals received from an EEG device; and determining, by the controller, a word or a function selected on the video display based on the gaze point.

16. The computer readable medium of claim 15, the method further comprising: generating a sentence by selecting words on the video display.

17. The computer readable medium of claim 16, the method further comprising: verifying the sentence; generating a soundtrack for the sentence; and playing the generated soundtrack.

18. The computer readable medium of any one of claims 15 to 17, the method further comprising: displaying common words on the video display for generating the sentence.

19. The computer readable medium of any one of claims 15 to 18, wherein the method comprises processing the EEG signals to determine a brain wave spike.

20. The computer readable medium of claim 19, wherein the processing of the EEG signals is performed using a neural network. AH25(45458020)

Images