Portable communication device

Abstract

The present disclosure relates to a portable communication device (301). The portable communication device comprises a controller (302) configured to receive (210) a first image and a second image from one or more cameras (340) and to transmit the images to a video display (314). The portable communication device further comprises an electroencephalogram (EEG) device (350) configured to capture an EEG signal of a brain of a user, wherein the controller is configured to receive the EEG signal from the EEG device. The controller is configured to detect (220) an object on the received first image, to display (230) a word associated with the detected object on the video display, to determine (510) a gaze point of one or more eyes of the user based on the second image of the camera, wherein the second image is related to the one or more eyes of the user, to verify (520) the determined gaze point based on the EEG signal, and to determine (540) a selected word or function on the video display based on the gaze point.

Description

Citation of relevant applications

[0001] This application relates to Australian Provisional Patent Application No. 2024900238, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention generally relates to a communication device. It also relates to a method for operating the communication device, and a computer program product comprising a computer-readable medium having a computer program for the communication device recorded thereon. Background Technology

[0003] People with conditions such as cerebral palsy, autism spectrum disorder, or amyotrophic lateral sclerosis may need assistance to communicate.

[0004] In a typical setup, these individuals can use desktop-based communication devices with an augmentative and alternative communication (AAC) system. The AAC system provides methods to assist users in generating and reading aloud generated sentences. However, such a typical setup lacks privacy, portability, and robustness. Summary of the Invention

[0005] The object of the present invention is to substantially overcome or at least improve one or more disadvantages of the existing arrangement.

[0006] This invention discloses an arrangement aimed at solving the aforementioned problems by providing a portable device that offers an Assistive and Alternative Communication (AAC) system. The portable communication device according to this disclosure is capable of capturing an image from the user's viewpoint. The portable communication device then detects objects within the captured image and presents the detected objects as words to the user on a display. The portable communication device then tracks the user's eye movements and monitors the user's electroencephalogram (EEG) signals. Based on the tracked eye movements and EEG signals, the portable communication device determines the word the user wishes to select on the display. A sentence can then be generated based on the selection of a sequence of words. Finally, a speech generator within the portable communication device speaks the sentence.

[0007] This portable communication device can be worn by users and provides a way for users to communicate in a more independent manner.

[0008] According to one aspect of this disclosure, a portable communication device is provided, comprising: a controller configured to receive a first image and a second image from one or more cameras and transmit the images to a video display; an electroencephalogram (EEG) device configured to capture EEG signals from a user's brain, wherein the controller is configured to receive EEG signals from the EEG device; wherein the controller is configured to: detect an object in the received first image; display a word associated with the detected object on the video display; determine a gaze point of one or more of the user's eyes based on the second image from the camera, wherein the second image is associated with one or more of the user's eyes; verify the determined gaze point based on the EEG signals; and determine a word or function selected on the video display based on the gaze point.

[0009] According to another aspect of this disclosure, a communication method is provided, comprising: detecting an object in a first image captured by a first camera via a controller; displaying a word associated with the detected object on a video display via the controller; determining a gaze point of one or more eyes of a user based on a second image captured by a second camera, wherein the second image is associated with one or more eyes of the user; verifying the determined gaze point based on EEG signals received from an electroencephalogram (EEG) device via the controller; and determining a word or function selected on the video display via the controller based on the gaze point.

[0010] According to another aspect of this disclosure, a computer program product is provided, comprising a computer-readable medium on which a computer program for implementing any of the above methods is recorded.

[0011] This disclosure also discloses other aspects. Attached Figure Description

[0012] At least one embodiment of the present invention will now be described with reference to the accompanying drawings, wherein: Figure 1A and Figure 1B A portable communication device according to this disclosure is shown; Figure 2 It is an operation Figure 1A and Figure 1B A flowchart illustrating the method for using the portable communication device is shown. Figure 3 yes Figure 2 The flowchart of the word display sub-procedure of the method shown is as follows; Figure 4 It shows Figure 3 Examples of embodiments of the subprocess shown; Figure 5 yes Figure 2 The flowchart of the word selection sub-process of the method shown is as follows; Figure 6 Showing Figure 1A and Figure 1B Example of the device shown; Figure 7 It is a diagram used to determine eye fixation; and Figure 8 The method used to determine brainwave spikes is shown. Detailed Implementation

[0013] If any step and / or feature with the same reference numerals is mentioned in any one or more of the accompanying drawings, then unless otherwise intended to the contrary, such step and / or feature are used for the purposes of this specification and have the same function(one or more) or operation(one or more).

[0014] Figure 1A and Figure 1B A schematic block diagram of a portable communication device 301 is shown, comprising embedded components on which the communication method to be described is deliberately implemented. The portable communication device 301 includes at least an electroencephalogram (EEG) device 350, a camera 340, and a video display 314. In one arrangement, the camera 340 and the video display 314 are components of an extended reality (XR) device. When the camera 340 and the video display 314 are part of the XR device, a controller 302 of the device 301 (see below) is configured to communicate electrically with the XR device. Upon connection to the XR device, the controller 302 is then able to receive images (e.g., video or individual images) from the camera 340 and transmit the images (e.g., video or individual images) to the video display 314.

[0015] Figure 6 One arrangement of device 301 is shown. For example... Figure 6 As shown, the camera 340 and the video display 314 are positioned near the user's eyes.

[0016] One or more of the cameras 340 capture images of the eyes of a user wearing a portable communication device 301. For ease of reference, the camera 340 that captures images of the user's eyes will be referred to herein as camera 340EYES.

[0017] One or more other cameras in camera 340 capture images associated with the point of view of the user wearing device 301. For ease of reference, the camera 340 that captures images associated with the user's point of view is referred to herein as camera 340POV.

[0018] Video display 314 provides video / images captured by camera 340POV. Video display 314 also overlays additional data onto the video / images. This additional data includes words associated with objects detected on the video / images.

[0019] EEG device 350 is configured to be worn by a user on his / her head. EEG device 350 then captures EEG signals from the user's brain. In one arrangement, EEG device 350 includes 22 electrodes with a sampling rate of 250 Hz. EEG device 350 also includes a notch filter of 48 to 52 Hz and a bandpass filter of 0.2 to 60 Hz. Signals from EEG device 350 are processed by method 800 (described below).

[0020] like Figure 1A As shown, device 301 also includes an embedded controller 302. The controller 302 has a processing unit (or processor) 305, which is bidirectionally connected to an internal storage module 309. Figure 1B As shown, the storage module 309 may be composed of a non-volatile semiconductor read-only memory (ROM) 360 and a semiconductor random access memory (RAM) 370. The RAM 370 may be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory.

[0021] Device 301 includes a display controller 307 connected to a video display 314, such as a liquid crystal display (LCD) panel. The display controller 307 is configured to display graphic images on the video display 314 according to instructions received from an embedded controller 302, which is connected to the embedded controller 302.

[0022] Device 301 also includes user input devices 313, which typically consist of buttons, keypads, or similar controls. In some embodiments, user input device 313 may include a touch-sensitive panel physically associated with display 314, thereby collectively forming a touchscreen. Thus, such a touchscreen can operate as a form of graphical user interface (GUI) rather than a prompt- or menu-driven GUI typically used with a keyboard-display combination. Other forms of user input devices may also be used, such as a microphone (not shown) for voice commands or a joystick / dial (not shown) for easy navigation on menus.

[0023] like Figure 1A As shown, device 301 also includes a removable memory interface 306, which is connected to processor 305 via connection 319. Removable memory interface 306 allows complementary removable memory devices 325 to be connected to electronic device 301 as a data source or destination, or to supplement internal storage module 309. Examples of such interfaces allow connection to a variety of removable memory devices, such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, PCMIA cards, optical discs, and hard disks.

[0024] Device 301 also has a communication interface 308 to allow device 301 to be connected to a computer or communication network 320 via connection 321. Connection 321 can be wireless or wired. For example, connection 321 can be radio frequency or optical. Examples of wired connections include Ethernet. Furthermore, examples of wireless connections include Bluetooth™ local interconnect, Wi-Fi (including protocols based on the IEEE 802.11 family of standards), Infrared Data Association (IrDA), etc.

[0025] The method described below can be implemented using an embedded controller 302, wherein Figure 2 , Figure 3 , Figure 5 and Figure 8 The process can be implemented as one or more software applications 333 that can be executed within the embedded controller 302. Figure 1A The device 301 in the document implements the method. Specifically, refer to... Figure 1B The steps of the method are implemented by instructions in software 333 executed within controller 302. The software instructions may be formed as one or more code modules, each code module being used to perform one or more specific tasks. The software may also be divided into two separate parts, where a first part and its corresponding code modules execute the method, and a second part and its corresponding code modules manage the user interface between the first part and the user.

[0026] The software 333 of the embedded controller 302 is typically stored in a non-volatile ROM 360 of the internal storage module 309. The software 333 stored in ROM 360 can be updated as needed by a computer-readable medium. The software 333 can be loaded into and executed by the processor 305. In some cases, the processor 305 can execute software instructions located in RAM 370. Software instructions can be loaded into RAM 370 by initiating a copy of one or more code modules from ROM 360 to RAM 370 via the processor 305. Alternatively, the software instructions for one or more code modules can be pre-installed by the manufacturer in a non-volatile area of ​​RAM 370. After locating one or more code modules in RAM 370, the processor 305 can execute the software instructions of those one or more code modules.

[0027] Application 333 is typically pre-installed by the manufacturer and stored in ROM 360 before the distribution of device 301. However, in some cases, application 333 may be coded onto one or more CD-ROMs (not shown) and provided to the user, and may be stored via internal storage module 309 or removable memory 325 before being stored in the device. Figure 1AThe removable storage interface 306 reads the data. Alternatively, the software application 333 may be read by the processor 305 from the network 320 or loaded from other computer-readable media into the controller 302 or the removable storage medium 325. A computer-readable storage medium refers to any non-transitory tangible storage medium that participates in providing instructions and / or data to the controller 302 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical disks, flash memory, or computer-readable cards such as PCMCIA cards, whether such devices are internal or external to the device 301. Examples of temporary or non-tangible computer-readable transmission media that may also participate in providing software, applications, instructions, and / or data to the device 301 include radio or infrared transmission channels and network connections to another computer or networked device, as well as the Internet or intranets, including email transmissions and information recorded on websites. A computer-readable medium on which such software or computer programs are recorded is a computer program product.

[0028] The second part of application 333 and the corresponding code module mentioned above can be executed to achieve the desired result. Figure 1A One or more graphical user interfaces (GUIs) are presented or otherwise represented on the display 314. This is achieved by manipulating a user input device 313 (e.g., a keypad) or... Figure 2 , 3 With the communication method described in 5, users of device 301 and application 333 can manipulate the interface in a functionally adaptable manner to provide control commands and / or inputs to applications associated with the GUI. Other forms of functionally adaptable user interfaces can also be implemented, such as an audio interface utilizing voice prompts output via a speaker (not shown) and user voice commands input via a microphone (not shown).

[0029] Figure 1B An embedded controller 302 is shown in detail, having a processor 305 for executing an application program 333 and internal storage 309. Internal storage 309 includes read-only memory (ROM) 360 and random access memory (RAM) 370. The processor 305 is capable of executing the application program 333 stored in one or both of the connected memories 360 and 370. When the electronic device 301 is initially powered on, a system program residing in ROM 360 is executed. The application program 333, permanently stored in ROM 360, is sometimes referred to as "firmware." The execution of the firmware by the processor 305 can implement various functions, including processor management, memory management, device management, storage management, and user interface.

[0030] Processor 305 typically includes multiple functional modules, including a control unit (CU) 351, an arithmetic logic unit (ALU) 352, a digital signal processor (DSP) 353, and local or internal memory including a set of registers 354 that typically contain atomic data elements 356, 357, as well as internal buffers or caches 355. One or more internal buses 359 interconnect these functional modules. Processor 305 also typically has one or more interfaces 358 for communicating with external devices via system bus 381 using connection 361.

[0031] Application program 333 contains a sequence of instructions 362 to 363, which may include conditional branch instructions and loop instructions. Application program 333 may also contain data for executing application program 333. This data may be stored as part of the instructions or in a separate location 364 within ROM 360 or RAM 370.

[0032] Typically, processor 305 is provided with a set of instructions, in which these instructions are executed. This set of instructions can be organized into multiple blocks, each block performing a specific task or processing a specific event occurring in processing device 301. Typically, application 333 waits for an event and then executes the code block associated with that event. Events can be triggered in response to user input detected by processor 305, which can be transmitted via... Figure 1A User input device 313 or Figure 2 , 3 The communication methods described in sections 5 and 8 are used as inputs. Events can also be triggered in response to other sensors and interfaces in electronic device 301.

[0033] Executing a set of instructions may require reading and modifying numerical variables. Such numerical variables are stored in RAM 370. The disclosed method uses input variables 371 stored in known locations 372 and 373 of memory 370. Input variables 371 are processed to generate output variables 377, which are stored in known locations 378 and 379 of memory 370. Intermediate variables 374 may be stored in other memory locations 375 and 376 of memory 370. Alternatively, some intermediate variables may exist only in register 354 of processor 305.

[0034] In processor 305, instruction sequences are executed by repeatedly applying fetch-execute cycles. The control unit 351 of processor 305 maintains a register called the program counter, which contains the address of the next instruction to be executed in ROM 360 or RAM 370. At the beginning of the fetch-execute cycle, the contents of the memory address indexed by the program counter are loaded into control unit 351. This loaded instruction controls subsequent operations of processor 305, such as loading data from ROM memory 360 into processor register 354, arithmetically combining the contents of one register with the contents of another, or writing the contents of another register to a location stored in another register. At the end of the fetch-execute cycle, the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed, this may involve incrementing the address contained in the program counter or loading the program counter to a new address to implement a branch operation.

[0035] Each step or sub-process in the following method process is associated with one or more segments of application 333 and is executed by repeatedly executing an instruction fetch-execution cycle in processor 305 or similar program operations of other independent processor modules in electronic device 301.

[0036] Figure 2 A flowchart of method 100 for operating device 301 to provide an AAC system is shown. Method 100 is a software application 333 that can be executed in an embedded controller 302, as previously described. In an alternative arrangement, some steps of method 100 are performed by a server that is in electrical communication with device 301 via network 320. For example, if some steps require processing power, it may be faster to transfer the data to be processed to the server, since the server's processing power is higher than that of processor 305.

[0037] In one arrangement, method 100 begins when the user of device 301 presses the power-on button.

[0038] Method 100 begins with a word display for subprocess 200. Figure 3 A flowchart of subprocess 200 is shown, which will be described below. Subprocess 200 receives a first image from camera 340POV, detects objects in the received first image, and displays the detected objects as words on video display 314. The term "first image" is associated with the image captured by camera 340POV.

[0039] Figure 4 Examples of such displays are shown, where words 410, 412, and 420 are associated with the detected objects. Figure 4 This will be discussed in more detail when we talk about subprocess 200 below.

[0040] Method 100 proceeds from sub-procedure 200 to function selection sub-procedure 400. Figure 5 A flowchart of subprocess 400 is shown. Subprocess 400 receives a second image from camera 340EYES and determines the gaze point of one or more of the user's eyes based on the received second image and EEG device 350. The term "second image" is associated with the image captured by camera 340EYES.

[0041] Based on the determined gaze point, subprocess 400 determines the functions selected on video display 314 (e.g., words 410, 412, 414, functions 420, 430). Figure 4 The video display 314 shows words 410, 412, 414 and functions 420, 430. If subprocess 400 determines that the user's gaze is associated with any of words 410, 412, or 414, then the word is selected. If subprocess 400 determines that the user's gaze is associated with the speaking function 420, then the speaking function is selected. If subprocess 400 determines that the user's gaze is associated with the deletion function 430, then the deletion function is selected.

[0042] Subprocess 400 returns the selected function (i.e., word, speak, delete) to method 100. Subprocess 400 will be described in more detail below.

[0043] Once subprocess 400 completes, control returns to method 100. Method 100 then proceeds from subprocess 400 to step 120.

[0044] In step 120, method 100 determines whether the speak function 420 was selected in subprocess 400 (see [link]). Figure 4 If the speak function 420 is selected (Yes), then method 100 proceeds from step 120 to step 130. However, if the speak function 420 is not selected (No), then method 100 proceeds from step 120 to step 160.

[0045] Next, we will discuss step 160 and its subsequent steps, and then we will discuss step 130.

[0046] In step 160, method 100 determines whether a word is selected or deleted. If a word is selected, method 100 proceeds from step 160 to step 170. If a word is deleted, method 100 proceeds from step 160 to step 180.

[0047] In step 170, method 100 adds the selected word to the sentence. If there are no pending sentences, a new sentence is created, with the selected word as the first word of that sentence. Then, method 100 proceeds from step 170 to subprocess 400 to select the next function (i.e., add word, delete word, speak).

[0048] In step 180, method 100 deletes the last word of the pending sentence. If there is no pending sentence, method 100 displays a notification on video display 314, indicating that the selected function is invalid because there is no pending sentence.

[0049] In one arrangement, the pending sentence 450 is displayed on a video display 314 (see [link]). Figure 4 This allows users to review currently unresolved sentences.

[0050] Method 100 then proceeds from step 180 to sub-process 400.

[0051] Now let's discuss step 130.

[0052] In step 130, method 100 verifies the unresolved sentence by reviewing its grammatical issues. For example, a user might form the sentence "I want to eat apples." In step 130, method 100 can then modify this sentence to "I want..." to eat an apple.

[0053] In one arrangement, this verification is performed using a neural network model. There are generally available neural network models that can be adapted for use in method 100. The neural network model needs to be provided with correct language usage, and then the neural network model will be able to verify such sentences.

[0054] Method 100 then proceeds from step 130 to step 140.

[0055] In step 140, method 100 generates an audio track of the sentence verified in step 130. Method 100 then proceeds from step 140 to step 150.

[0056] In step 150, method 100 plays the generated audio track through the speaker of device 301. As previously described, in an alternative arrangement, steps 130 to 150 are performed by a server that is in electrical communication with device 301 via network 320. For example, device 301 communicates with the server via network 320 at the end of step 120. The server then performs steps 130 to 150. This alternative arrangement is advantageous because some computer processing is performed by a server external to device 301.

[0057] Method 100 ends at the end of step 150. Method 100 may provide the user with the option to generate another sentence at the end of step 150.

[0058] Subprocess 200 will now be discussed. As previously described, subprocess 200 is a software application 333 that can be executed within the embedded controller 302. In an alternative arrangement, certain steps of subprocess 200 may be executed by a server that is in electrical communication with device 301 via network 320.

[0059] Subprocess 200 begins in step 210 by receiving a first image from camera 340 POV. Then, subprocess 200 proceeds from step 210 to step 220.

[0060] In step 220, subprocess 200 detects objects within the received first image. In one arrangement, subprocess 200 uses YOLOv5 (an object detection model) to detect objects within the received first image.

[0061] Figure 4 This shows an example of subprocess 200 detecting apples and cups.

[0062] Subprocess 200 then proceeds from step 220 to step 230.

[0063] In step 230, subprocess 200 displays words related to the detected object on video display 314. Figure 4 The word 412 for the detected apple and the word 410 for the detected cup are displayed.

[0064] Figure 4 It also shows common words 414 that may be needed to generate the sentence (e.g., I, it, you, she, etc.) and functions for the AAC system (e.g., speaking function 420 and deletion function 430). Speaking function 420 provides instructions to device 301 to speak the user-created sentence (see steps 120 to 150 of method 100). Deletion function 430 provides instructions to device 301 to delete the last word in a pending sentence (see steps 160 and 180 of method 100).

[0065] Subprocess 200 ends when step 230 is completed.

[0066] Now let's discuss Figure 5 The sub-process 400 is shown. As previously described, sub-process 400 is a software application 333 that can be executed within the embedded controller 302. In an alternative arrangement, some steps of sub-process 400 may be executed by a server that is in electrical communication with device 301 via network 320.

[0067] Subprocess 400 begins at step 505 by detecting the user's eyes. This detection is performed on a second image received from camera 340EYES. In one arrangement, eye detection is performed by an algorithm that detects two circles of contrasting colors in a monochrome filtered image. These two circles consist of a larger circle outside the smaller circle, representing the iris and pupil of the eye, respectively. If an eye is detected (yes), subprocess 400 proceeds from step 505 to step 510. If no eye is detected (no), step 505 is repeated.

[0068] In step 510, subprocess 400 uses a second image received from camera 340EYES to determine the user's gaze point. The relative shape, position, and distortion of the iris and pupil in the second image represent the gaze direction. The intersection of the gaze directions of both eyes is the gaze point.

[0069] Before the user uses device 301, device 301 performs a calibration process for the user. The calibration process includes presenting a dot sequence on video display 314. Device 301 uses camera 340EYES to capture a sequence of images of the user's eyes as the user views the dot sequence. Each captured image contains the shape, position, and distortion (referred to as reference parameters) of the iris and pupil of each eye. Therefore, in step 510, the eyes captured in the second image are compared with the reference data to determine the user's gaze direction. The gaze point is then determined using the intersection of the gaze directions of the two eyes.

[0070] In one arrangement, three or more consecutive points are used to obtain reference data.

[0071] In one arrangement, the gaze point data is mathematically transformed to transform the determined gaze point onto a point on the video display 314. Figure 7 The image shows eye 710 (labeled point O) fixating on point B or point I on video display 314. However, camera 340 captures the eye's fixation at point A. Therefore, to determine the eye's fixation on video display 314, the eye's fixation is transformed using the following equation: in K This is the scaling factor from OA to OB. Parameter K This is determined during the calibration process of device 301.

[0072] Subprocess 400 then proceeds from step 510 to step 520.

[0073] In step 520, subprocess 400 verifies the determined gaze point. Verification is performed by randomly flashing words / functions associated with the determined gaze point along with the three to five closest words / functions. For example, if the gaze point is determined to be directed to the speaking function 420, flashing lights are generated at the speaking function 420, the deletion function 430, the word "No," and its adjacent words. The flashing lights are generated randomly. Simultaneously, the controller 302 of device 301 monitors EEG data received from EEG device 350. When the controller 302 detects a brainwave spike, the timing of the spike is associated with the word / function that the flashing light appears at that specific time. For example, if a brainwave spike is detected at 1 second, and the speaking function 420 is flashing at that time, the controller 302 determines based on the EEG data that the user intends to select the speaking function 420. In one arrangement, the controller 302 performs method 800 to determine the presence of a positive brainwave spike. The following will be combined with Figure 8 Method 800 is described.

[0074] Subprocess 400 then proceeds from step 520 to step 530.

[0075] In step 530, subprocess 400 determines whether the fixation point is accurate. This determination is performed by comparing the fixation point determined in step 510 with the fixation point determined in step 520 based on EEG data. If the two determined fixation points match (yes), subprocess 400 proceeds from step 530 to step 540. Otherwise (no), subprocess 400 proceeds from step 530 to step 510.

[0076] In step 540, subprocess 400 determines the word / function associated with the determined fixation point. Subprocess 400 then returns the selected word / function to method 100. Subprocess 400 terminates at the end of step 450.

[0077] The method 800 for processing signals from the EEG device 350 will now be discussed. As previously described, method 800 is a software application 333 that can be executed within an embedded controller 302. In an alternative arrangement, some steps of method 800 may be performed by a server that is in electrical communication with the device 301 via a network 320.

[0078] Method 800 begins at step 815, which is used to process the signal (i.e., input 810) of the EEG device 350. One or more signal processing methods are used in step 815. These signal processing methods include feature extraction, low-pass filtering, noise feature-based filtering (e.g., filtering based on non-target event-related potentials (ERPs)), and feature selection.

[0079] Feature extraction transforms the input 810 into a set of numerical features. Examples of feature extraction methods that can be used include statistical methods, principal component analysis, linear discriminant analysis (LDA), autoencoders, etc.

[0080] Low-pass filters are used to filter out additional noise while retaining the main features identified by the feature extraction method.

[0081] The non-target ERP-based filter function is used to mitigate noise that is highly similar to the ERP characteristics. The non-target ERP-based filter first derives the spectrum based on the average value of the non-target segment (epoch) in the raw signal from the EEG device 350. This derivation uses the following equation: (1) (2) (3) (4) in, It is the i-th segment in the time domain. It is the average value of the non-target segment in the time domain. Equation (2) is the Fourier transform. It is the average value of the non-target segment in the frequency domain, corresponding to , and by Export. It is the spectrum of each segment.

[0082] In equation (4), the data is transformed back to the time domain by inverse Fourier transform.

[0083] Feature selection is performed in the time domain. An example of a feature selection algorithm is Linear Support Vector Regression (Linear-SVR).

[0084] Once the signal from EEG device 350 has been processed in step 815, method 800 proceeds to step 820. In step 820, method 800 performs classification on the processed signal using a neural network (e.g., a convolutional neural network or a recurrent neural network). Examples of classification algorithms include support vector machines (SVM), latent Dirichlet assignment (LDA), etc.

[0085] Table 1 below shows an example of an RNN structure for a classification algorithm. Then, method 800 proceeds from step 820 to step 825.

[0086] In step 825, method 800 performs labeling. In this case, labeling determines the probability of a positive EEG spike (see step 520). The labeling algorithm uses the following equation: (5) (6) in, For the event j (For example, brainwave spikes) belong to blocks with a value of 1. k (e.g., brainwave spikes) i The probability of occurrence (for example, over a period of time). Block with a value of 1 k Running i The sum of all probabilities. Method 800 proceeds from step 825 to step 830.

[0087] In step 830, method 800 outputs a label (i.e., whether a brainwave spike occurs). Method 800 terminates at the end of step 825.

[0088] Industrial applicability The arrangement described is suitable for the computer and data processing industry.

[0089] The above description only describes some embodiments of the present invention. Modifications and / or changes can be made to it without departing from the scope and spirit of the present invention. These embodiments are merely illustrative and not restrictive.

[0090] In the context of this specification, the word "comprising" means "primarily includes but not necessarily only includes," "has," or "contains," rather than "consisting of only." Variations of the word "comprising," such as "comprise" and "comprises," have correspondingly different meanings.

Claims

1. A portable communication device, comprising: A controller configured to receive a first image and a second image from one or more cameras and transmit the images to a video display; An electroencephalogram (EEG) device configured to capture EEG signals from a user’s brain, wherein the controller is configured to receive the EEG signals from the EEG device; The controller is configured to: Detect objects in the received first image; The video display shows words associated with the detected object. The gaze points of one or more of the user's eyes are determined based on a second image from the camera, wherein the second image is associated with one or more of the user's eyes; The gaze point determined based on the EEG signal verification; The word or function selected on the video display is determined based on the gaze point.

2. The device as claimed in claim 1, wherein, The camera and the video display are components of an extended reality (XR) device.

3. The device as described in claim 2, wherein, The extended reality (XR) device communicates with the controller.

4. The device as claimed in any one of claims 1 to 3, wherein, The controller is also configured to: Sentences are generated by selecting words on the video display.

5. The device as claimed in claim 4, wherein, The controller is also configured to: Verify the sentence; Generate the audio track for the sentence; and Play the generated audio track.

6. The device as claimed in any one of claims 1 to 5, wherein, The controller is also configured to: Common words and phrases are displayed on the video display to generate the sentence.

7. The device as claimed in any one of claims 1 to 6, wherein, The EEG signal is processed to determine brainwave spikes.

8. The device as claimed in claim 7, wherein, The processing of the EEG signal is performed using a neural network.

9. A communication method, comprising: The controller detects objects in the first image captured by the first camera. The controller displays words associated with the detected object on a video display. The controller determines the gaze point of one or more of the user's eyes based on a second image captured by a second camera, wherein the second image is associated with one or more of the user's eyes; The controller verifies the determined gaze point based on EEG signals received from the electroencephalogram (EEG) device; and The controller determines the word or function selected on the video display based on the gaze point.

10. The method of claim 9, further comprising: Sentences are generated by selecting words on the video display.

11. The method of claim 10, further comprising: Verify the sentence; Generate the audio track for the sentence; as well as Play the generated audio track.

12. The method of any one of claims 9 to 10, further comprising: Common words and phrases are displayed on the video display to generate the sentence.

13. The method of any one of claims 9 to 12, comprising processing the EEG signal to determine brainwave spikes.

14. The method of claim 13, wherein, The processing of the EEG signal is performed using a neural network.

15. A computer-readable medium comprising a computer program executable by a controller, wherein, When the controller executes the computer program, the controller performs a communication method, the communication method including: The controller detects objects in the first image captured by the first camera. The controller displays words associated with the detected object on a video display. The controller determines the gaze point of one or more of the user's eyes based on a second image captured by a second camera, wherein the second image is associated with one or more of the user's eyes; The controller verifies the determined gaze point based on EEG signals received from the electroencephalogram (EEG) device; and The controller determines the word or function selected on the video display based on the gaze point.

16. The computer-readable medium of claim 15, wherein the method further comprises: Sentences are generated by selecting words on the video display.

17. The computer-readable medium of claim 16, further comprising: Verify the sentence; Generate the audio track for the sentence; as well as Play the generated audio track.

18. The computer-readable medium of any one of claims 15 to 17, wherein the method further comprises: Common words and phrases are displayed on the video display to generate the sentence.

19. The computer-readable medium as claimed in any one of claims 15 to 18, wherein, The method includes processing the EEG signal to determine brainwave spikes.

20. The computer-readable medium of claim 19, wherein, The processing of the EEG signal is performed using a neural network.

Images