Wearable device for state sensing

Abstract

Implementations of the present disclosure provide a wearable device for state sensing. The device consists of one or more electrodes to measure and analyze signals, or a mix of them. These signals are measured and analyzed to estimate the state of the user, including human's true electrophysiological signals, eye movements, hearing, interest, emotion, intention, instruction, and related feelings triggered by human senses. The wearable device can have various suitable forms, such as earphones, glasses, etc. In terms of hardware, it can be designed to include different types or locations of electrodes and printed circuit boards. In terms of software, machine learning models can be pre-trained using unlabelled electrophysiological signals, and fine-tuned using labelled data for specific tasks. The labels can include, for example, human's true electrophysiological signals, gaze position, emotion, interest, intention, user instruction, feedback to visual, auditory, gustatory, olfactory, and / or tactile stimuli.

Description

Background Technology

[0001] Wearable devices are small electronic devices that can be worn directly on the body or embedded in clothing or accessories. Wearable devices can achieve functions such as health monitoring, motion tracking, and environmental sensing through sensors, software, and network connectivity. Summary of the Invention

[0002] According to an implementation of this disclosure, a wearable device is proposed. The wearable device includes multiple electrodes configured to be placed on the head of a target object when the wearable device is worn to acquire at least one electrophysiological signal at at least one location on the target object's head. A machine learning model can generate an estimate of the target object's state over a given time period based on at least one electrophysiological signal. This wearable device is small and portable and can be worn for extended periods. Furthermore, it can process multiple electrophysiological signals and generate estimates of the target object's physiological state, intentions, emotions, etc.

[0003] This section is provided to simplify the presentation of the selection of objects, which will be further described in the detailed embodiments below. This section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0004] Figure 1 A schematic block diagram of some implementations of wearable devices according to this disclosure is shown;

[0005] Figure 2 A flowchart of a signal processing procedure according to some implementations of this disclosure is shown;

[0006] Figure 3 A schematic diagram of signal segmentation according to some implementations of this disclosure is shown;

[0007] Figure 4 Example architectures for signal encoding based on some implementations of this disclosure are shown;

[0008] Figure 5A A schematic diagram of an example architecture for a pre-trained encoder according to some implementations of this disclosure is shown;

[0009] Figure 5B A schematic diagram of an example encoder-decoder architecture for state estimation according to some implementations of this disclosure is shown;

[0010] Figure 5C A schematic diagram of an example architecture for knowledge distillation for state estimation according to some implementations of this disclosure is shown;

[0011] Figures 6A to 6CA schematic diagram of an example structure of a wearable device according to some implementations of this disclosure is shown;

[0012] Figure 7 A schematic block diagram of an electronic device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0013] Implementations of this disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement this disclosure, and not to imply any limitation on the scope of this disclosure.

[0014] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0015] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various implementations are described throughout this document, and any type of implementation may be included under any section / subsection. Furthermore, an implementation described in any section / subsection may be combined in any way with any other implementation described in the same section / subsection and / or different sections / subsections.

[0016] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0017] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning (DL) is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0018] As mentioned above, current wearable devices are limited in their ability to support multiple tasks because they cannot process a variety of electrophysiological signals and provide users with corresponding state estimates. Furthermore, traditional devices for measuring electrophysiological signals are bulky, expensive, and unsuitable for long-term wear, thus hindering their application in daily life.

[0019] Therefore, a wearable device for state sensing is proposed according to an implementation of this disclosure. This wearable device can process various electrophysiological signals. In this wearable device, at least one electrophysiological signal can be acquired at at least one location on the head of a target object within a time period. Using a machine learning model, a representation of the at least one electrophysiological signal is generated to produce an estimate of the target object's state within the time period. The state may include the target object's physiological state or psychological state (e.g., emotion, intention, etc.).

[0020] An example implementation of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Example Architecture

[0022] Figure 1 A schematic block diagram of a wearable device 100 according to some implementations of the present disclosure is shown. When worn by a target object, the wearable device 100 can be placed at any suitable location on the target object, such as the head. The wearable device 100 includes a plurality of electrodes 102 configured to be placed on the head of the target object when the wearable device is worn, for acquiring at least one electrophysiological (EXG) signal at at least one location on the head of the target object. In some implementations, the wearable device 100 may be configured to be at least partially placed on the ear of the target object when the wearable device 100 is worn by the target object. For example, when the wearable device 100 is worn, the plurality of electrodes are placed near the ear of the target object to acquire EXG signals in the ear appendage. In such an implementation, the plurality of electrodes 102 may be configured to be attached to the target object when the wearable device 100 is worn by the target object to acquire at least one EXG signal at at least one location adjacent to the ear of the target object.

[0023] In this implementation, the wearable device 100, when worn by a target object (e.g., a person), does not affect the target object's normal activities. That is, the target object can move freely and normally while wearing the wearable device 100. Therefore, the wearable device 100 can be independent of attachment to and connection to other external devices. For example, the wearable device 110 does not need to be connected to a fixed device (such as a signal display device or signal processing device) via a connecting component (e.g., a cable).

[0024] In some implementations, such as Figure 1 As shown, the wearable device 100 may include a support portion 101. The support portion 101 may be configured to be placed on the head of the target object when the wearable device 100 is worn by the target object, for example, it may be placed on the ear.

[0025] Figure 1 The configuration of the wearable device 100 shown is merely exemplary and not intended to be limiting. The wearable device 100 can have any suitable configuration. For example, the wearable device 100 can be headphones, in which case the support portion 101 can be the part of the headphone frame that provides support. As another example, the wearable device 100 can be glasses, in which case the support portion 101 is the part of the glasses frame that provides support. As yet another example, the wearable device 100 can be a patch, in which case the support portion 101 can be the substrate of the patch. Furthermore, the wearable device 110 can be a headband, in which case the support portion can be a strap, and so on, without further details.

[0026] In some implementations, multiple electrodes 102 may be at least partially disposed on the support 101 and configured to attach to the target object when the wearable device is worn by the target object.

[0027] For example, such as Figure 1 As shown, the 10 electrodes 102 can be respectively disposed at different positions on the support 101. When the target object wears the wearable device 100, the electrodes 102 can be attached to the target object so as to collect at least one electrophysiological signal at the corresponding position.

[0028] In some implementations, the wearable device can be an earbud-style headset. Conductive tape can be placed on the frame of the headset, electrodes can be mounted through the conductive tape, and electrophysiological signals from the corresponding skin contact areas can be effectively acquired through these electrodes.

[0029] In some implementations, wearable devices can be implemented in other forms, such as eyeglass frames, virtual reality (VR) / augmented reality (AR) / mixed reality (MR) glasses, etc.

[0030] In some implementations, the plurality of electrodes 102 may include at least one reference electrode providing a reference signal and at least one measuring electrode for signal measurement. The reference electrode provides a stable potential reference as a reference point for the potentials of other electrodes in the measurement system. The measuring electrode is used to measure electrophysiological signals. In some implementations, the plurality of electrodes 102 may also include at least one bias electrode providing a bias signal. The bias electrode modulates the system potential to ensure that the working electrodes are at a preset potential state.

[0031] In some implementations, such as Figure 1As shown, the wearable device can be an earbud-style headset with five electrodes 102 on each side. The top electrode of the right earbud is a reference electrode, and the bottom electrode of the left earbud is a bias electrode. Electrodes 1, 2, 3, 4, 5, 6, 7, and 8 can all be measurement electrodes, forming eight signal channels. This arrangement helps to maximize spacing and enhance signal discrimination.

[0032] In some implementations, the wearable device 100 may include a control device 103 for processing acquired electrophysiological signals. The control device 103 may include a signal amplifier (bio amplifier chip) for amplifying the electrophysiological signals. The control device 103 may include an analog-to-digital converter 104 for converting analog signals to digital signals, an inertial measurement unit 105 as an inertial sensor, a microcontroller 106 for processing electrophysiological signals, a storage device 107 (SD card) for storing electrophysiological signals, or a communication device 108 (Bluetooth, WiFi) for transmitting electrophysiological signals. The storage device 107 may be, for example, an SD card. The communication device 108 may include, for example, Bluetooth, WiFi, etc.

[0033] In some implementations, at least one of a signal amplifier, a microcontroller 106, a storage device 107, or a communication device 108 is arranged on a printed circuit board (PCB). Electrodes are connected to the printed circuit board.

[0034] It should be understood that the structure and function of the various elements in the wearable device 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0035] Signal processing methods

[0036] Figure 2 A flowchart of a signal processing procedure 200 according to some implementations of this disclosure is shown. Procedure 200 can be implemented at any suitable electronic device. For example, procedure 200 can be implemented at wearable device 100, or can be partially implemented at wearable device 100. For example, some operations of procedure 200 can be implemented at wearable device 110, and others can be implemented at a signal processing device communicating with wearable device 110. In some implementations, the signal processing device can be a terminal device, such as a user's mobile phone or tablet computer. In some implementations, the signal processing device can be a cloud server. For example, the EXG signal collected by wearable device 110 can be sent to a cloud server to provide cloud services.

[0037] In some implementations, within box 210, at least one electrophysiological signal X can be acquired by the wearable device at at least one location on the head of the target object over a period of time. c In some implementations, the wearable device can be placed on the ear. Accordingly, at least one electrophysiological signal X can be acquired over a time period at at least one location adjacent to the ear of the target subject. c .

[0038] Electrophysiological signals are physical signals originating from electrical activity within a living organism. They are generated by specific cells and reflect the body's functional state and activities. Electrophysiological signals can provide insights into the body's sensory information, such as visual, auditory, olfactory, tactile, gustatory, interest, emotional, and visual information.

[0039] In some implementations, at least one electrophysiological signal is acquired by a wearable device placed on the user's ear. For example, the wearable device can be in the form of headphones. Because headphones are small and easy to wear, they can acquire abundant electrophysiological signals with minimal interference to the user. Such wearable devices are not only compact and comfortable, allowing for prolonged wear in daily life, but also support local storage via SD cards and real-time streaming via WiFi or Bluetooth. For example, the acquired electrophysiological signals can be transmitted to signal processing equipment via WiFi or Bluetooth.

[0040] For example, refer to Figure 1 The wearable device 100 includes electrodes 1-8, each of which acts as a channel. When a user wears the wearable device, electrophysiological signals X can be collected at eight locations, namely electrodes 1-8. c ∈R T X c This represents the signal recorded from channel c, where T is the total number of time steps. The acquired electrophysiological signals can include multiple modalities, each reflecting a type of physiological activity. Different modalities of electrophysiological signals can correspond to different frequency ranges. Each electrode can acquire electrophysiological signals over a wide frequency range to cover the frequency range of multiple modalities.

[0041] For example, electrophysiological signals may include at least one of the following: electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG), electrocardiogram (ECG), or electrodermal activity / galvanic skin response (EDG).

[0042] In box 215, a machine learning model can be used to generate an estimate of the target object's state during the time period, based at least one electrophysiological signal acquired within that time period. The state can include physiological state, psychological state, etc. In some implementations, the state during a time period can be estimated based on EXG signals acquired within that time period. In some implementations, the state during a time period can be further estimated based on EXG signals acquired at other times. For example, the state during a time period can be estimated based on EXG signals acquired within that time period and EXG signals acquired at one or more times prior to that time period.

[0043] The estimated state can include psychological and physiological states. In some implementations, the state can include at least one of the following: visual state, auditory state, olfactory state, tactile state, gustatory state, interest state, emotion, intention, eye movement state, or instruction (i.e., the instruction issued or desired by the user). Specifically, the visual state can infer the user's gaze position or the type of object the user sees. The auditory state can infer whether the user hears a sound and the type of sound the user hears. The olfactory state can infer the type of odor the user smells. The tactile state can infer the type of object the user touches. The gustatory state can infer the type of taste the user experiences. The interest state can infer what the user is interested in. The emotion can infer the user's emotional state during the time period, such as happiness, sadness, or disappointment. The intention can infer what the user hopes to achieve or the action they wish to perform during the time period. The eye movement state can infer the user's saccades, gazes, and other actions. In some implementations, the estimated instruction can be provided to a subsequent executor to execute the instruction. For example, in virtual reality or mixed reality scenarios, the instruction can be provided to an actuator. For example, instructions can be provided to intelligent assistants based on artificial intelligence (e.g., large language models).

[0044] In some implementations, the estimated state may include the electrophysiological signals of the target modality of the target object. For example, the ECG, EOG, or EMG of the target object may be estimated.

[0045] In some implementations, electrophysiological signals can be used to infer whether a user has heard a sound. For example, participants can listen to two different frequencies (1000Hz and 1140Hz) of buzzing played randomly with equal probability. Each buzzing lasts 25 seconds, with a 5-second interval, for 10 cycles per session, each session lasting 5 minutes, for a total of 5 sessions and 25 minutes. The collected electrophysiological signals are divided into 4-second time windows. Without pre-training, the average accuracy of inferring whether a user has heard a sound from electrophysiological signals is 66%. After pre-training, the accuracy improves to 71%. Even without pre-training, a certain level of accuracy can be achieved, and pre-training can further improve performance. In other words, once the target subject wears the wearable device, the above-mentioned information related to estimating physiological states can be obtained.

[0046] State estimates can be represented in various appropriate ways. In some implementations, a state estimate can be represented by one of several predetermined levels associated with the state. For example, for the state of whether or not a sound was heard, the predetermined levels could include a first level corresponding to not hearing and a second level corresponding to hearing. The estimate obtained using a machine learning model could then include either the first or the second level. Similarly, for the state of interest or degree of interest, the predetermined levels could include level 1, level 2, and level 3, with higher levels representing greater interest. The estimate obtained using a machine learning model could then include one of level 1, level 2, or level 3.

[0047] In some implementations, the estimate of a state can be represented by one of several candidate options associated with that state. For example, for the state of emotion, the candidate options could include three options: feeling happy, feeling calm, and feeling unhappy. Therefore, the resulting state estimate could include one of these three candidate options.

[0048] In some implementations, the estimation of a state can be represented by text describing the state. For example, for the state of a user-issued instruction, the estimation obtained using a machine learning model could include the text corresponding to the instruction, such as "make a cup of coffee." Similarly, for the state of an emotion, the estimation obtained using a machine learning model could include text describing the emotion, such as "the user is currently calm."

[0049] In some implementations, the estimate of the state can be represented numerically. For example, for the state of eye attention position, the estimate obtained using a machine learning model can include a numerical value representing the specific spatial location.

[0050] In some implementations, the state estimate can be represented by a numerical sequence. For example, if the estimated state is an electrophysiological signal of a certain modality of the target object (e.g., ECG, EOG), the estimation result can include a numerical sequence to represent the electrophysiological signal of that modality.

[0051] It should be understood that the above forms of state estimation are merely exemplary and are not intended to be limiting. An estimation of a type of state can be represented in any suitable form or one.

[0052] Continue to refer to Figure 2 This section describes an example architecture for state estimation using a machine learning model. In some implementations, the machine learning model may first encode the EXG signal and then derive a state estimate based on the encoded features.

[0053] like Figure 2 As shown, in some implementations, in box 220, a machine learning model can be used to generate fused coded features representing at least one electrophysiological signal. For example, at least one acquired electrophysiological signal can be input into a machine learning model, and the encoder in the machine learning model can be used to generate fused coded features.

[0054] In some implementations, this can be achieved by analyzing at least one electrophysiological signal X in both the frequency and time dimensions. c Segmentation is performed to determine multiple electrophysiological signal components p f,c,l Multiple electrophysiological signal components p f,c,l The electrophysiological signal components in the data correspond to a frequency band, a time interval, and a location.

[0055] In some implementations, at least one electrophysiological signal X is acquired. c Subsequently, in order to extract information at a specific frequency, multiple filters can be used to filter at least one electrophysiological signal X. c Frequency decomposition was performed to obtain electrophysiological signal components X in multiple frequency bands. f,c The filter is used to filter one frequency band out of multiple frequency bands, where f represents the frequency band selected from the electrophysiological signal component X. f,c The index of the frequency band. Then, based on a preset time step, the electrophysiological signal components X of multiple frequency bands are analyzed. f,c The signal is segmented along the time dimension to obtain multiple electrophysiological signal components p. f,c,l , l represents the electrophysiological signal component p f,c,l The index of the time step to which it belongs.

[0056] For example, Figure 3 A schematic diagram of signal segmentation is shown. Figure 3 Each box 301 in the diagram represents an electrophysiological signal X from one channel. cThe channels described here correspond to the acquisition locations of electrophysiological signals. At least one electrophysiological signal X is acquired. c Then, multiple bandpass filters can be applied to the electrophysiological signal X. c Perform frequency decomposition to obtain N. F Electrophysiological signal components X in each frequency band f,c ∈R T f∈{1,…,N F Each electrophysiological signal component X f,c Only the components within the corresponding frequency band are retained. Figure 3 Each box 302 in the diagram represents an electrophysiological signal component X in a frequency band. f,c This is obtained by frequency decomposition using 35Hz-75Hz, 15Hz-35Hz, 5Hz-15Hz, and 0.5Hz-5Hz as frequency bands. This is achieved through analysis of the electrophysiological signal X. c Frequency decomposition yields multiple electrophysiological signal components X. f,c This allows machine learning models to perform frequency domain feature analysis at higher resolution, thus enabling them to be adapted to human perception-related applications operating in different frequency ranges.

[0057] After acquiring electrophysiological signal components X in multiple frequency bands f,c Then, the electrophysiological signal components X of multiple frequency bands can be analyzed according to a preset time step. f,c The signal is segmented along the time dimension to obtain multiple electrophysiological signal components p. f,c,l ∈Rw. Figure 3 Each box 303 in the text represents an electrophysiological signal component p. f,c,l w represents the preset time step, f∈{1,…,N} F} represents the frequency band, c∈{1,…,N C} represents the channel index, l∈{1,…,N L} represents the component index in the time dimension.

[0058] In some implementations, multiple electrophysiological signal components p are determined. f,c,l Subsequently, it is possible to base the analysis on multiple electrophysiological signal components p f,c,l Using a machine learning model, a fused encoded feature z is generated.

[0059] In some implementations, multiple electrophysiological signal components P are acquired. f,c,l Subsequently, the p-values ​​used to characterize multiple electrophysiological signal components can be determined. f,c,l The first embedding representation sequence Next, based on a preset order related to the time, frequency, and spatial dimensions, the first embedded representation sequence is adjusted. The order of the embedding representations in the sequence is used to obtain the second embedding representation sequence. Next, using a machine learning model, based on the second embedding representation sequence Generate at least one electrophysiological signal X c The fused coding feature z = E θ (p).

[0060] For example, Figure 4 An example architecture for signal encoding is shown. Figure 4 Each box 401 in the text represents an electrophysiological signal component p. f,c,l Each electrophysiological signal component p f,c,l Tokenized embeddings can be generated using learnable tokenizers and linear projections. f,c,l ∈R d d is the embedding dimension.

[0061] In order for machine learning models to be able to extract multiple electrophysiological signal components p f,c,l The time information, channel information, and frequency band information obtained can be used to introduce three different learnable embeddings: time embedding. Channel embedding and frequency embedding These embeddings are learned during the training of the machine learning model, enabling the model to integrate information from three dimensions. The first embedding represents a sequence. For tokenized embedding e f,c,l ∈R d and the sum of the corresponding embeddings, Figure 4 Middle frame 402 represents the first embedding representation sequence. The formula is expressed as follows:

[0062]

[0063] Where e f,c,l Represents tokenized embedding, Indicates time embedding, Indicates channel embedding. This indicates frequency embedding.

[0064] In some implementations, to effectively simulate the hierarchical dependencies of frequency, channels, and time dimensions, the first embedding representation sequence can be adjusted. The order of the various embedding representations in the sequence is used to obtain the second embedding representation sequence. For example, the first embedding can be used to represent the sequence in the order of frequency embedding, channel embedding, and time embedding. Adjusted to the second embedding representation sequence And the second embedding representation sequence Input into the machine learning model.

[0065] In other words, the encoder of the machine learning model first processes frequency embeddings of different frequency bands to obtain the spectral dependencies of electrophysiological signals. After encoding the frequency embeddings, the channel embeddings of different channels are processed to obtain the spatial relationships of electrophysiological signals. After encoding the channel embeddings, the temporal embeddings are processed in the time dimension to obtain the temporal dependencies of electrophysiological signals. This approach ensures that spectral, spatial, and temporal features can be learned hierarchically. The second embedding represents the sequence. It can be represented as:

[0066]

[0067] That is, the first embedding represents the sequence. Before being input into the encoder of the machine learning model, it can be flattened into a one-dimensional sequence according to the order of frequency, channel, and time, that is, the second embedding representation sequence.

[0068] The machine learning model in this disclosure can be implemented using any suitable neural network architecture. In some implementations, the machine learning model can be any suitable sequence model, such as, but not limited to, sequence models based on Transformer, Mamba, and other architectures.

[0069] As an example and not intended to be limiting, in some implementations, to improve signal processing efficiency, a bidirectional Mamba machine learning model can be applied to perform the encoder function of the machine learning model. This can be based on a second embedding representation sequence. Generation and second embedding representation sequence A complementary third embedding representation sequence. The second embedding representation sequence is then processed via the forward path in the encoder. The forward feature sequence is obtained. Next, the third embedding representation sequence is processed through the reverse path in the encoder to obtain the reverse feature sequence. Then, by fusing the forward and reverse feature sequences, at least one electrophysiological signal X is obtained. c The fused coding feature z.

[0070] The bidirectional Mamba machine learning model is a neural network architecture that combines bidirectional processing mechanisms with the characteristics of the Mamba model. Its core idea is to capture more comprehensive contextual information within the sequence by processing the input sequence in both forward and backward directions. In the processing of the bidirectional Mamba machine learning model, input data needs to be processed in both forward and backward directions. Therefore, it is necessary to obtain the second embedding representation sequence. Complementary third embeddings represent sequences. In this way, information from both the forward and reverse directions can complement each other to more comprehensively capture the contextual information of the sequence.

[0071] For example, such as Figure 4 As shown, after the first embedding representation sequence 402 passes through the normalization layer 405 and the projection layer 406, the second embedding representation sequence 407 and the third embedding representation sequence 408 are obtained. The second embedding representation sequence 407 is then input into the forward path of the bidirectional Mamba model. In the forward path, the second embedding representation sequence 407 is processed by the forward convolutional layer 409 and the forward state space model layer 411 to obtain the forward feature sequence. The third embedding representation sequence 408 is input into the reverse path of the bidirectional Mamba model. In the reverse path, the third embedding representation sequence 408 is processed by the reverse convolutional layer 410 and the reverse state space model layer 412 to obtain the reverse feature sequence. Next, through the connection layer 413, the projection layer 414, and the connection layer 415, feature fusion is performed based on the first embedding representation sequence 402, the forward feature sequence, and the reverse feature sequence to obtain at least one electrophysiological signal X. c The fused coding feature 403.

[0072] In some implementations, in box 230, an estimate of the state of the target object within a given time period can be generated based on the fused coded features. As an example, an encoder-decoder architecture can be used for state estimation based on the EXG signal. For instance, in box 230, a decoder can be used to generate the state estimate.

[0073] Based on the fused coded features z, a decoder can be used to generate an estimate of the user's state over a time period. For example, ... Figure 4 As shown, after obtaining the fused coded features 403, the fused coded features 403 can be rearranged to obtain the output sequence 404. The output sequence 404 can be provided to the decoder. The decoder can output an estimate of the state. See the reference below for details about the decoder. Figure 5B The description.

[0074] The following describes the pre-training of machine learning models. In some implementations, wearable devices facilitate the acquisition of users' electrophysiological signals, making it convenient to collect these signals in daily life or outdoor settings. To utilize these acquired unlabeled electrophysiological signals, a self-supervised pre-training strategy can be employed to improve the generalization ability of the machine learning model. For example, during pre-training, sample electrophysiological signals can be acquired at a location near the ear of a nearby reference subject. The machine learning model is then used to reconstruct these sample electrophysiological signals. Based on the reconstruction results, the machine learning model is then pre-trained.

[0075] In some implementations, to ensure robust feature extraction, multiple self-supervised objectives can be used to pre-train the machine learning model. Multiple reconstruction tasks can be defined, each associated with a corresponding decoder. For example, the decoder could be a multilayer perceptron (MLP).

[0076] In some implementations, during the pre-training phase, a time-domain reconstructed signal corresponding to the sample electrophysiological signal can be generated based on the encoding of the sample electrophysiological signal using a machine learning model. Similarly, a time-domain reconstructed signal corresponding to the masked sample electrophysiological signal can be generated based on the encoding of the sample electrophysiological signal using a machine learning model. Furthermore, a frequency-domain reconstructed signal corresponding to the masked sample electrophysiological signal can also be generated based on the encoding of the sample electrophysiological signal using a machine learning model.

[0077] The following is for reference. Figure 5A Here is an example. In some implementations, the specific steps for encoding the sample electrophysiological signal based on the machine learning model are the same as the inference stage mentioned above, and will not be repeated here. That is, sample electrophysiological signal component 501 is generated from the sample electrophysiological signal, and then the encoder 503 in the machine learning model is used to generate the encoded sample electrophysiological signal z.

[0078] The encoded z of the sample electrophysiological signal is then input into the decoder 504 so that the decoder 504 can be used to reconstruct the original electrophysiological signal and obtain the time-domain reconstructed signal corresponding to the sample electrophysiological signal. The time-domain reconstructed signal is reconstructed over the time period corresponding to the sample's electrophysiological signal. This method allows the encoder to reduce noise while capturing temporal features.

[0079] In some implementations, to enhance contextual learning over time, model pre-training can be performed by applying masks. At least one masked electrophysiological signal component 502 can be generated by applying a mask to the sample electrophysiological signal component 501. The masked electrophysiological signal component can be represented as p. mask This partially masks the sample electrophysiological signal component 501 in the time, spatial, and frequency dimensions. Then, based on at least one masked electrophysiological signal component 502, the encoder 503 generates the encoded z of the masked sample electrophysiological signal. mask =E θ (p mask ).

[0080] Then the encoded z of the masked sample electrophysiological signal can be... maskThe signal is input into decoder 507 so that decoder 507 can be used to reconstruct the original electrophysiological signal and obtain the time-domain reconstructed signal corresponding to the masked sample electrophysiological signal.

[0081] To obtain more comprehensive spectral information, the Fast Fourier Transform can also be applied to obtain the amplitude p corresponding to the sample electrophysiological signal components. A and phase p P Then, the encoded z of the sample electrophysiological signal is input into the decoder 505 so that the decoder 505 can be used to reconstruct the amplitude characteristics of the original electrophysiological signal, thereby obtaining the frequency domain reconstructed signal corresponding to the sample electrophysiological signal. Furthermore, the encoded z of the sample electrophysiological signal is input into the decoder 506 so that the decoder 506 can be used to reconstruct the phase features of the original electrophysiological signal, thereby obtaining the frequency domain reconstructed signal corresponding to the sample electrophysiological signal.

[0082] To enhance the ability of machine learning models to predict spectral features from incomplete inputs, the encoded z of the masked sample electrophysiological signal can be... mask The signal is input into decoder 508 so that decoder 508 can be used to reconstruct the amplitude characteristics of the original electrophysiological signal, thereby obtaining the frequency domain reconstructed signal corresponding to the sample physiological signal. And the encoded z of the masked sample electrophysiological signal mask The signal is input into decoder 509 so that decoder 509 can reconstruct the phase features of the original electrophysiological signal to obtain the frequency domain reconstructed signal corresponding to the masked sample electrophysiological signal. In this way, the original frequency domain signal can be recovered.

[0083] To jointly optimize all self-supervised tasks, an electrophysiological signal loss can be set during the pre-training of the machine learning model based on the reconstruction results. For example, the electrophysiological signal loss can be defined as a weighted sum of the mean absolute error (MAE) loss for a specific task. The mean absolute error measures the average absolute difference between predicted and true values. Therefore, the formula for the electrophysiological signal loss can be expressed as follows:

[0084]

[0085] L = L AE +L MR +L Freq +L MFreq

[0086] Where L AE L represents the loss of the time-domain reconstructed signal. MRL represents the loss of the time-domain reconstructed signal when a mask is applied. Freq L represents the loss of the frequency domain reconstructed signal. MFreq λ represents the loss of the frequency domain reconstructed signal under masking conditions, and L represents the electrophysiological signal loss. Each λ controls the relative contribution of the corresponding loss.

[0087] In some implementations, the collected electrophysiological signals may include multiple modalities, each reflecting a type of physiological activity. For example, electroencephalogram (EEG) signals correspond to the electroencephalogram (EEG) modality, reflecting brain physiological activity; electrooculogram (EOG) signals correspond to the electrooculogram (EOG) modality, reflecting eye physiological activity; electromyography (EMG) signals correspond to the EMG modality, reflecting muscle physiological activity; electrocardiogram (ECG) signals correspond to the ECG modality, reflecting cardiac physiological activity; and electrodermal conductance (EDC) signals correspond to the ESC modality, reflecting skin physiological activity, and so on.

[0088] In this implementation, reconstructed signals corresponding to one or more modalities of electrophysiological signals can be generated based on the encoding of sample electrophysiological signals by a machine learning model. Each modality's electrophysiological signal reflects a type of physiological activity. These reconstructed signals are then used to calculate the electrical signal loss for each modality, thereby improving the model performance for each modality within the machine learning model. This type of reconstruction task can be used in conjunction with the reconstruction tasks described above, or it can be used independently. This reconstruction task can be used to update the machine learning model in various training methods, such as pre-training, fine-tuning, and regular training.

[0089] In this type of reconstruction task, the acquired raw EXG signal can be used to reconstruct EOG, EMG, EEG, ECG, or EDG, etc. This reconstruction task can extract individual electrophysiological signals, making it useful in itself. Simultaneously, the reconstruction error between the raw signal and the reconstructed signal can be used as a term in the loss function. For example, for eye tracking, a typical loss function includes the difference between the estimated gaze position and the true gaze position. The reconstruction error from reconstructing the EOG signal from the raw EXG signal can be added to the loss function. By adding such an additional term to the loss function, the performance of eye tracking can be improved. Similar approaches can be applied to EMG, EEG, ECG, etc.

[0090] In some implementations, dry electrodes are a type of electrode that collects bioelectrophysiological signals through direct contact with the skin without requiring an additional conductive medium. Wet electrodes, typically made of metal, require the use of conductive gel to fill the tiny gaps between the electrode and the skin. Dry electrodes are very convenient to use and can therefore be applied to wearable devices for daily monitoring, while the bioelectrophysiological signals collected using wet electrodes are more accurate. Therefore, to improve the accuracy of machine learning model outputs, the difference between the electrophysiological signals collected by dry electrodes and those collected by wet electrodes can be used to train the machine learning model.

[0091] In some implementations, the electrophysiological signals of the target modality can be reconstructed from the sample electrophysiological signals. These target modality signals reflect the physiological activity of the target type. Then, the target modality's electrophysiological signals are acquired from predetermined sites on a reference subject. These predetermined sites are related to the target modality. Specifically, the predetermined site is a location suitable for acquiring signals from the target modality. For example, for EEG signals or EEG modalities, the predetermined site could be the user's head, particularly the portion of the head close to the brain. Similarly, for ECG signals or ECG modalities, the predetermined site could be the user's chest or other areas close to the heart. Next, the machine learning model is updated based on the differences between the reconstructed and acquired electrophysiological signals. This reconstruction task can be used to update the machine learning model in various training methods, such as pre-training, fine-tuning, and regular training.

[0092] For example, dry electrodes in wearable devices can be used to collect sample electrophysiological signals, and the electrophysiological signals of EEG modalities can be reconstructed based on the collected sample electrophysiological signals. Then, EEG modal electrophysiological signals can be collected from the user's head. The differences between the reconstructed EEG modal electrophysiological signals and the collected EEG modal electrophysiological signals are used to train a machine learning model. This task of reconstructing wet electrode signals from dry electrode signals can be used in conjunction with the above-mentioned tasks, or it can be used alone.

[0093] In some implementations, to further optimize the performance of the machine learning model and support applications in various scenarios, the model can be fine-tuned. This involves acquiring labeled information indicating the true state of the target object. Then, the machine learning model is updated based on the difference between the estimate and the labeled information. For example, the user's gaze position can be inferred from collected electrophysiological signals, and the user's actual gaze position can be obtained. The machine learning model can then be trained using the difference between the two gaze positions.

[0094] In some implementations, the annotation information indicates at least one of the following: the gaze position of the target object, the degree of interest of the target object, the emotion of the target object, whether the target object hears a sound, the visual perception of the target object, the auditory perception of the target object, the olfactory perception of the target object, the tactile perception of the target object, the type of sound heard by the target object, the type of object seen by the target object, the electrophysiological signal of the target modality of the target object, or the instruction issued by the target object.

[0095] The above describes an example form of state estimation. Similarly, in some implementations, annotation information can include a selection of multiple predetermined levels related to the state. For example, for the state of whether or not a sound was heard, multiple predetermined levels could include a first level corresponding to not hearing and a second level corresponding to hearing a sound. The annotation information could then include either the first or the second level. As another example, for the state of interest or degree of interest, multiple predetermined levels could include level 1, level 2, and level 3, with higher levels representing greater interest. The annotation information could then include one of level 1, level 2, or level 3.

[0096] In some implementations, the annotation information may include a selection of multiple candidate options related to the state. For example, for the state of emotion, the multiple candidate options may include three options: feeling happy, feeling calm, and feeling unhappy. Then, the annotation information may include one of these three candidate options.

[0097] In some implementations, the annotation information may include text describing the state. For example, for the state of a user-issued instruction, the annotation information may include the text corresponding to the instruction, such as "make a cup of coffee." Similarly, for the state of an emotion, the annotation information may include text describing the emotion, such as "the user is currently calm."

[0098] In some implementations, the annotation information may include numerical values ​​representing the state. For example, for the state of eye attention position, the annotation information may include numerical values ​​representing the specific spatial location.

[0099] In some implementations, the annotation information may include a numerical sequence representing the state. For example, if the state is an electrophysiological signal of a certain modality of the target object (e.g., ECG, EOG), the annotation information may include a numerical sequence to represent the electrophysiological signal of that modality.

[0100] In some implementations, an encoder-decoder architecture can be used for state estimation. (See reference) Figure 5B Describe an example. For example... Figure 5BAs shown, the output of encoder 503 can be provided to pooling layer 510. Pooling layer 510 can apply average pooling to obtain a fixed-length vector as output. The output of pooling layer 510 can be provided to decoder 511. Decoder 511 can have any suitable network structure. In some implementations, decoder 511 can be a lightweight decoder, for example, consisting of two linear layers. However, this is merely exemplary and not intended to be any limitation.

[0101] Decoder 511 outputs an estimate of the state. In some implementations, pooling layer 510 may be omitted. For example, for eye-tracking tasks, the final output is position coordinates. Pooling layer 510 may be omitted. It is understood that whether and what type of data processing exists between encoder 503 and decoder 520 depends on the type or representation of the estimated state. Embodiments of this disclosure are not limited in this regard.

[0102] In the fine-tuning or training of this encoder-decoder architecture, the parameters of encoder 503 can be initialized using pre-trained parameter values. The parameters of decoder 511 can be initialized at any time. During fine-tuning, decoder 511 can output an estimate of the state. The difference between this estimate and the labeled information can be used to construct the loss function for fine-tuning. The parameters of encoder 511 and decoder 503 can be updated using the loss function, thereby achieving fine-tuning.

[0103] In some implementations, at least one electrophysiological signal includes signals of multiple modalities, so the parameter values ​​of the decoder can be adjusted according to the electrophysiological signals of different modalities in order to improve the processing effect of electrophysiological signals of different modalities.

[0104] The encoder-decoder architecture described above is an example. In some implementations, state estimation can be based on a single machine learning model. That is, such a machine learning model can be used to generate an estimate of the state based on the acquired EXG signal.

[0105] In some implementations, such machine learning models can be obtained through knowledge distillation. (Reference) Figure 5C Describe an example. In this example, encoder 503, decoder 511, and optional pooling layer 510 serve as the teacher model. State estimation model 520 serves as the student model. State estimation model 520 is a smaller, more efficient neural network. Exemplarily, the distillation process typically involves minimizing a weighted sum of two loss functions. One loss function is used to match the soft objective from the teacher model (e.g., using Kolb-Leibler divergence or a similar method). The other loss function is used to match the true label (e.g., it could be cross-entropy). This approach allows state estimation model 520, as the student model, to achieve high accuracy while significantly reducing the number of parameters and inference cost.

[0106] The above describes an example implementation of signal processing and model training. In this disclosure, the wearable device can have any suitable architecture. The following describes an example architecture and fabrication of the wearable device.

[0107] In some implementations, the wearable device may include one or more left electrodes configured to be adjacent to the left ear of the target object when the wearable device is worn by the target object. Alternatively or additionally, in some implementations, the wearable device may include one or more right electrodes configured to be adjacent to the right ear of the target object when the wearable device is worn by the target object.

[0108] In some implementations, the wearable device may include a left support portion and / or a right support portion. One or more left electrodes may be disposed on the left support portion. One or more right electrodes may be disposed on the right support portion. The left support portion may be configured to be positioned on the left ear of the target object when the wearable device is worn by the target object. The right support portion may be configured to be positioned on the right ear of the target object when the wearable device is worn by the target object. For example, if the wearable device is implemented as headphones, the left support portion may be the left earphone and the right support portion may be the right earphone. Figure 6A and Figure 6B Such headphones 600 are shown as an example of a wearable device. Figure 6A and Figure 6B As shown, the earphone 600 includes a left earphone 610 and a right earphone 620.

[0109] In some implementations, the electrodes can be implemented using a protruding structure (e.g., protruding from a support or body). For example, a left electrode disposed on a left support can be implemented using a protruding structure. Similarly, a right electrode disposed on a right support can be implemented using a protruding structure. For example, the left and / or right support may each include conductive protrusions. In some implementations, the left support may include one or more left conductive protrusions to serve as the one or more left electrodes. Alternatively or additionally, in some implementations, the right support may include one or more right conductive protrusions to serve as one or more right electrodes.

[0110] In some implementations, the wearable device may also include a central connector configured to connect one or more left electrodes and one or more right electrodes. For example, the central connector may be configured to connect a left support portion and a right support portion. In some implementations, the central connector comprises a resilient material formed by three-dimensional (3D) printing. (Continue to reference...) Figure 6A For example, the headphones 600 may include a strap 630, which is an example of an intermediate connecting part.

[0111] exist Figure 6Aand Figure 6B In the example, the earphone 600 comprises three parts: a left earphone 610, a right earphone 620, and a rear strap 630. The left and right earphones are mirror images of each other, featuring a frame design similar to traditional ear-hook headphones, and protruding structures on the frame. The ear-hook structure conforms to the shape of the user's ear, while the protruding structures contact the user's skin through an electroplated surface, allowing for the collection of corresponding physiological signals. The strap design not only serves to connect the left and right earphones but also possesses a degree of elasticity through 3D printing. This allows it to adapt to different user head sizes when worn. The wearing method is similar to existing bone conduction headphones. The user puts the device on from behind the head, first opening the left and right earphones, placing the strap behind the head, and then securing the earphones around the ears to complete the wearing process.

[0112] In the specific manufacturing process, the wearable device can be designed by dividing it into three parts: a left support part, a right support part, and a middle connecting part. Taking headphones as an example, the headphones can be designed by dividing them into three parts: a strap, a left earpiece, and a right earpiece.

[0113] For intermediate connecting parts (e.g., straps), the intermediate connecting parts can be designed and fabricated as individual units. This can be done using 3D printing technology. For example, thermoplastic polyurethane elastomer (TPU) can be chosen for printing because of its good elasticity and stretchability. This allows for greater deformation of the intermediate connecting parts to adapt to different user head sizes. Furthermore, an inward force can be applied to the left and right support parts, ensuring closer contact between the left and right support parts (especially the conductive protrusions) and the skin, thereby collecting signals with a higher signal-to-noise ratio. The two ends of the intermediate connecting parts can be assembled by having protruding ends that mate with the corresponding concave ends of the left and right support parts.

[0114] In some implementations, the wearable device (e.g., the left support portion) may also include one or more left non-conductive portions arranged alternately with one or more left conductive protrusions. The wearable device (e.g., the right support portion) may also include one or more right non-conductive portions arranged alternately with one or more right conductive protrusions. For example, each support portion includes a predetermined number of non-conductive portions (also referred to as non-electrode parts) and a predetermined number of conductive protrusions (also referred to as electrode parts). These non-electrode parts and electrode parts are arranged alternately.

[0115] Figure 6C An example is shown. Figure 6CIn the example, each earphone has four electrodes. Therefore, each earphone is divided into eight parts: non-electrode parts 611, 613, 615, and 617, and electrode parts 612, 614, 616, and 618. Each electrode part, as a whole, collects signals from its corresponding area by contacting the skin through a protrusion, while the non-electrode parts isolate the electrode area. Two parts can be fitted together as a whole using a combination of protrusions and grooves.

[0116] For non-electrode parts, conventional 3D printing methods can be used. For example, standard ABS material can be used. Each electrode part can be fabricated using the following process: First, it can be 3D printed from stainless steel. Then, each electrode part is electroplated, for example, using silver chloride. Finally, DuPont wires can be soldered to the back of each electrode part for measurement signal processing.

[0117] After preparing the non-electrode parts and the electrode parts, it can be done through Figure 6C The diagram shows the assembly of a single-sided earphone, comprising four electrode areas and four non-electrode areas. Each earphone has four electrode channels, resulting in a total of eight measurement channels across both sides.

[0118] Besides the one-piece molding method described above, multiple electrodes can be formed in other ways. For example, in some implementations, electrode sheets can be attached to a support portion to form electrodes. For instance, multiple electrodes may include an electrode sheet attached to the left support portion and an electrode sheet attached to the right support portion.

[0119] Overall, this design provides an easy-to-wear dry electrode array for collecting signals from the area around the user's ear without the need for conductive gel or medical tape.

[0120] The wearable device disclosed herein is small and easy to wear, allowing for long-term monitoring of physiological states in daily life. Because it can collect electrophysiological signals of multiple modalities and generate estimates of the user's physiological state, intentions, emotions, and other states, it greatly enhances the versatility of wearable device applications.

[0121] Example device

[0122] Figure 7 A schematic block diagram of an electronic device 700 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described in this disclosure.

[0123] like Figure 7As shown, the electronic device 700 includes an electronic device 700 in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing devices 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760.

[0124] In some implementations, the electronic device 700 can be implemented as a computing device, computing system, server, mainframe, or other device with computing capabilities.

[0125] Processing device 710 can be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700. Processing device 710 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller, etc.

[0126] Electronic device 700 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 may include volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 may include removable or non-removable media and may include computer-readable media, such as memory, flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 700.

[0127] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks and optical disc drives for reading from or writing to removable, non-volatile optical discs can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces.

[0128] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or other general network nodes.

[0129] Input device 750 can be one or more various input devices, such as a mouse, keyboard, data import device, etc. Output device 760 can be one or more output devices, such as a monitor, data export device, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. External devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0130] In some implementations, in addition to being integrated into a single device, some or all of the components of electronic device 700 may be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely deployed and can work together to achieve the functions described herein. In some implementations, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, cloud computing providers offer applications over a wide area network, and these applications can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated at remote data center locations or they may be distributed. Cloud computing infrastructure can provide services through shared data centers, even if they appear as a single access point for users. Therefore, the components and functions described herein can be provided from service providers at remote locations using a cloud computing architecture. Alternatively, they may be provided from conventional servers, or they may be installed directly or otherwise on client devices.

[0131] The electronic device 700 can be used to implement the signal acquisition and processing methods of this disclosure. The memory 720 may include one or more modules having one or more program instructions, which can be accessed and executed by the processing unit 710 to implement the functions of the various implementations described herein. For example, the memory 720 may include a signal processing module 722 for performing various signal processing and / or model training described herein. Figure 7 As shown, electronic device 700 can acquire the input required for signal processing through input device 750, and can provide an estimated output of the state of the target object within a time period through output device 760. In some implementations, electronic device 700 can also receive input from other devices (not shown) via communication unit 740.

[0132] Example implementation

[0133] The following are some example implementations of this disclosure.

[0134] In one aspect, this disclosure provides a signal acquisition and processing method. The method includes: acquiring at least one electrophysiological signal acquired by a wearable device at at least one location on the head of a target object over a time period; and generating an estimate of the target object's state over the time period using a machine learning model based on the at least one electrophysiological signal.

[0135] In some implementations, the machine learning model is pre-trained by: acquiring sample electrophysiological signals collected at at least one location on the head of a reference object over a period of time; reconstructing the sample electrophysiological signals using the machine learning model; and pre-training the machine learning model based on the reconstruction results.

[0136] In some implementations, reconstructing the sample electrophysiological signal includes at least one of the following: generating a time-domain reconstructed signal corresponding to the sample electrophysiological signal based on encoding the sample electrophysiological signal using a machine learning model; generating a time-domain reconstructed signal corresponding to the masked sample electrophysiological signal based on encoding the masked sample electrophysiological signal using a machine learning model; generating a frequency-domain reconstructed signal corresponding to the sample electrophysiological signal based on encoding the sample electrophysiological signal using a machine learning model; or generating a frequency-domain reconstructed signal corresponding to the masked sample electrophysiological signal based on encoding the masked sample electrophysiological signal using a machine learning model.

[0137] In some implementations, the reconstruction of sample electrophysiological signals also includes at least one of the following: encoding the sample electrophysiological signals based on a machine learning model to generate one or more reconstructed signals corresponding to one or more modal electrophysiological signals, each modal electrophysiological signal being used to reflect a type of physiological activity.

[0138] In some implementations, the method further includes: acquiring sample electrophysiological signals collected at a location near the ear of a neighboring reference object over a period of time; encoding the sample electrophysiological signals based on a machine learning model to generate one or more reconstructed signals corresponding to one or more modalities of electrophysiological signals, each modality of electrophysiological signal being used to reflect a type of physiological activity; and pre-training a machine learning model based on one or more reconstructed signals.

[0139] In some implementations, pre-training of the machine learning model also includes: reconstructing the electrophysiological signal of the target modality from the sample electrophysiological signal, the electrophysiological signal of the target modality being used to reflect the physiological activity of the target type; acquiring the electrophysiological signal of the target modality collected from a predetermined part of a reference object; and pre-training the machine learning model based on the difference between the reconstructed electrophysiological signal and the collected electrophysiological signal.

[0140] In some implementations, the method further includes: acquiring sample electrophysiological signals collected at at least one location on the head of a reference object over a period of time; reconstructing target modality electrophysiological signals from the sample electrophysiological signals using a machine learning model, the target modality electrophysiological signals being used to reflect the physiological activity of the target type; acquiring target modality electrophysiological signals collected from predetermined sites of the reference object; and pre-training the machine learning model based on the difference between the reconstructed electrophysiological signals and the collected electrophysiological signals.

[0141] In some implementations, the machine learning model is fine-tuned by: obtaining labeled information about the state of the target object, which indicates the truth value of the state; and updating the machine learning model based on the difference between the estimate and the labeled information.

[0142] In some implementations, the annotation information includes at least one of the following: selection of multiple predetermined levels related to the state, selection of multiple candidate options related to the state, text describing the state, numerical values ​​representing the state, or numerical sequences representing the state.

[0143] In some implementations, the annotation information indicates at least one of the following: the target object's gaze position, the target object's level of interest, the target object's emotion, the target object's intention, whether the target object heard a sound, the target object's visual perception, the target object's auditory perception, the target object's olfactory perception, the target object's tactile perception, the target object's gustatory perception, the type of sound heard by the target object, the type of object seen by the target object, the electrophysiological signal of the target object's target modality, or the instruction issued by the target object.

[0144] In some implementations, the electrophysiological signal includes at least one of the following: electroencephalogram (EEG) signal, electrooculogram (EOG) signal, electromyogram (EMG) signal, electrocardiogram (ECG) signal, electroskin activity and electroskin response signal, or a mixture of two or more of the following: EEG signal, EEG signal, ECG signal, or electroskin activity and electroskin response signal.

[0145] In some implementations, a state includes at least one of the following: visual state, auditory state, olfactory state, tactile state, gustatory state, interest state, emotion, intention, eye-tracking state, electrophysiological signal of the target modality of the target object, or instruction.

[0146] In some implementations, using machine learning models to generate estimates of the state of a target object over a time period includes: using machine learning models to generate fused coded features representing at least one electrophysiological signal; and generating estimates of the state of the target object over a time period based on the fused coded features.

[0147] In some implementations, generating fused coded features representing at least one electrophysiological signal includes: segmenting at least one electrophysiological signal in the frequency and time dimensions to determine multiple electrophysiological signal components, wherein the electrophysiological signal components correspond to a frequency band, a time interval, and a location; and generating fused coded features based on the multiple electrophysiological signal components using a machine learning model.

[0148] In some implementations, determining multiple electrophysiological signal components includes: performing frequency decomposition on at least one electrophysiological signal using multiple filters to obtain electrophysiological signal components in multiple frequency bands, wherein the filters are used to filter one of the multiple frequency bands; and segmenting the electrophysiological signal components in multiple frequency bands in the time dimension based on a preset time step to obtain multiple electrophysiological signal components.

[0149] In some implementations, generating fused encoded features includes: determining an embedding representation sequence for characterizing multiple electrophysiological signal components; adjusting the order of each embedding representation in the embedding representation sequence according to a preset order related to the time, frequency, and spatial dimensions, based on the time, frequency, and position corresponding to each embedding representation in the embedding representation sequence, to obtain an adjusted embedding representation sequence; and using a machine learning model to generate fused encoded features from the adjusted embedding representation sequence.

[0150] In some implementations, the wearable device includes multiple electrodes configured to be placed on the head of a target when the wearable device is worn, for the purpose of acquiring at least one electrophysiological signal.

[0151] In some implementations, the multiple electrodes include: at least one reference electrode for providing a reference signal, and at least one measuring electrode for signal measurement.

[0152] In some implementations, the multiple electrodes also include at least one bias electrode that provides a bias signal.

[0153] In some implementations, wearable devices include at least one of the following: headphones, glasses, patches, or headbands.

[0154] In some implementations, machine learning models include sequence models.

[0155] In another aspect, this disclosure provides a wearable device with a plurality of electrodes configured to be placed on the head of a target object when the wearable device is worn, for acquiring at least one electrophysiological signal at at least one location on the head of the target object, wherein the at least one electrophysiological signal is used to provide a machine learning model to generate an estimate of the state of the target object.

[0156] In some implementations, the multiple electrodes include: at least one reference electrode for providing a reference signal, and at least one measuring electrode for signal measurement.

[0157] In some implementations, the multiple electrodes also include at least one bias electrode that provides a bias signal.

[0158] In some implementations, wearable devices include at least one of the following: headphones, glasses, patches, or headbands.

[0159] In some implementations, wearable devices may also include one of the following: a signal amplifier for amplifying electrophysiological signals, a microcontroller for processing electrophysiological signals, a storage device for storing electrophysiological signals, or a communication device for transmitting electrophysiological signals.

[0160] In some implementations, at least one of a signal amplifier, a microcontroller, a storage device, or a communication device is arranged on a printed circuit board.

[0161] In some implementations, the multiple electrodes include at least one of the following: one or more left electrodes configured to be adjacent to the left ear of the target object when the wearable device is worn by the target object, or one or more right electrodes configured to be adjacent to the right ear of the target object when the wearable device is worn by the target object.

[0162] In some implementations, the wearable device includes at least one of the following: one or more left conductive protrusions to serve as one or more left electrodes, or one or more right conductive protrusions to serve as one or more right electrodes.

[0163] In some implementations, the wearable device also includes at least one of the following: one or more left non-conductive portions arranged alternately with one or more left conductive protrusions, or one or more right non-conductive portions arranged alternately with one or more right conductive protrusions.

[0164] In some implementations, the wearable device also includes a central connection configured to connect one or more left electrodes and one or more right electrodes.

[0165] In some implementations, multiple electrodes comprise multiple electrode sheets.

[0166] In another aspect, this disclosure provides a signal acquisition and processing system, comprising: a wearable device including a plurality of electrodes configured to be placed on the head of a target object when the wearable device is worn, for acquiring at least one electrophysiological signal at at least one location on the head of the target object; and a signal processing device coupled to the wearable device to receive at least one electrophysiological signal, and the signal processing device being configured to: acquire at least one electrophysiological signal acquired by the wearable device at at least one location within a time period; and generate an estimate of the state of the target object within the time period using a machine learning model, based at least on the at least one electrophysiological signal.

[0167] In some implementations, the signal processing device is further configured to: utilize a machine learning model to generate fused coded features representing at least one electrophysiological signal; and based on the fused coded features, generate an estimate of the state of the target object over a time period.

[0168] In some implementations, the signal processing device is further configured to: segment at least one electrophysiological signal in the frequency and time dimensions to determine multiple electrophysiological signal components, wherein the electrophysiological signal components correspond to a frequency band, a time interval and a location; and generate fused encoded features based on the multiple electrophysiological signal components using a machine learning model.

[0169] In some implementations, the electrophysiological signal includes at least one of the following: electroencephalogram (EEG) signal, electrooculogram (EOG) signal, electromyogram (EMG) signal, electrocardiogram (ECG) signal, or skin electrical activity and skin electrical response signal.

[0170] In some implementations, wearable devices include or are adapted to be attached to at least one of the following: headphones, glasses, patches, or headbands.

[0171] In another aspect, this disclosure provides a computer program product tangibly stored in a computer storage medium and including computer-executable instructions that, when executed by a device, cause the device to perform the following actions: acquiring at least one electrophysiological signal collected by a wearable device at at least one location on the head of a target object over a time period; and generating an estimate of the state of the target object over the time period using a machine learning model based on the at least one electrophysiological signal.

[0172] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, example types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Load Programmable Logic Devices (CPLDs), and so on.

[0173] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of a single implementation may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0176] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A signal acquisition and processing method, comprising: Acquire at least one electrophysiological signal from at least one location on the head of a target subject by a wearable device over a period of time; as well as Based on the at least one electrophysiological signal, a machine learning model is used to generate an estimate of the state of the target object during the time period.

2. The method according to claim 1, wherein the machine learning model is pre-trained in the following manner: Acquire electrophysiological signals from samples taken at at least one location on the head of a reference subject over a period of time; The electrophysiological signals of the sample are reconstructed using the machine learning model; and Based on the reconstruction results, the machine learning model is pre-trained.

3. The method according to claim 2, wherein reconstructing the electrophysiological signal of the sample comprises at least one of the following: Based on the encoding of the sample electrophysiological signal by the machine learning model, a time-domain reconstructed signal corresponding to the sample electrophysiological signal is generated. Based on the encoding of the masked sample electrophysiological signal by the machine learning model, a time-domain reconstructed signal corresponding to the masked sample electrophysiological signal is generated. Based on the encoding of the sample electrophysiological signal by the machine learning model, a frequency domain reconstructed signal corresponding to the sample electrophysiological signal is generated, or Based on the encoding of the masked sample electrophysiological signal by the machine learning model, a frequency domain reconstructed signal corresponding to the masked sample electrophysiological signal is generated.

4. The method according to claim 3, wherein reconstructing the electrophysiological signal of the sample further comprises at least one of the following: Based on the encoding of the sample electrophysiological signals by the machine learning model, one or more reconstructed signals are generated, each corresponding to one or more modal electrophysiological signals, with each modal electrophysiological signal used to reflect a type of physiological activity.

5. The method according to claim 1, further comprising: Acquire electrophysiological signals from samples taken at locations near the ear of a nearby reference subject over a period of time; Based on the encoding of the sample electrophysiological signals by the machine learning model, one or more reconstructed signals are generated, each corresponding to one or more modalities of electrophysiological signals, wherein each modality of electrophysiological signal reflects a type of physiological activity; and The machine learning model is updated based on the one or more reconstructed signals.

6. The method according to claim 1, further comprising: Acquire sample electrophysiological signals collected at at least one location on the head of a reference subject over a period of time; Using the machine learning model, the electrophysiological signals of the target modality are reconstructed from the sample electrophysiological signals, and the electrophysiological signals of the target modality are used to reflect the physiological activities of the target type; Acquire the electrophysiological signals of the target modality from a predetermined location on the reference object; as well as The machine learning model is updated based on the difference between the reconstructed electrophysiological signal and the acquired electrophysiological signal.

7. The method of claim 1, wherein the machine learning model is fine-tuned in the following manner: Obtain annotation information for the state of the target object, wherein the annotation information indicates the truth value of the state; and The machine learning model is updated based on the difference between the estimate and the labeled information.

8. The method according to claim 7, wherein the annotation information includes at least one of the following: The selection of multiple predetermined levels related to the state, The selection of multiple candidate options related to the state, Text describing the state. The numerical value representing the state, or A numerical sequence representing the state.

9. The method of claim 7, wherein the annotation information indicates at least one of the following: The gaze position of the target object The degree of interest of the target object The emotions of the target object. The intent of the target object Did the target object hear the sound? The visual perception of the target object. The auditory perception of the target object The olfactory perception of the target object. The tactile sensation of the target object The taste sensation of the target object The type of sound heard by the target object. The type of object seen by the target object. The electrophysiological signal of the target modality of the target object, or The instructions issued by the target object.

10. The method of claim 1, wherein the electrophysiological signal comprises at least one of the following: EEG signals, Electrooculogram (EOG) signal, Electromyography signals, Electrocardiogram (ECG) signals, Electrodermal activity and electrodermal response signals, or The electroencephalogram (EEG) signal, the electrooculogram (EOG) signal, the electrocardiogram (ECG) signal, or a mixture of two or more of the electrodermal activity and electrodermal response signals.

11. The method of claim 1, wherein the state comprises at least one of the following: Visual state Auditory state olfactory state, tactile state, Taste state state of interest mood, intention, Eye movement state, The electrophysiological signal of the target modality of the target object, or instruction.

12. The method of claim 1, wherein generating an estimate of the state of the target object during the time period using a machine learning model comprises: Using the machine learning model, fused coded features representing the at least one electrophysiological signal are generated; as well as Based on the fused encoded features, an estimate of the state of the target object during the time period is generated.

13. The method of claim 1, wherein the machine learning model comprises a sequence model.

14. A wearable device comprising a plurality of electrodes, the plurality of electrodes being configured to: When the wearable device is worn, the plurality of electrodes are placed on the head of the target object to collect at least one electrophysiological signal at at least one location on the head of the target object, wherein the at least one electrophysiological signal is used to provide a machine learning model to generate an estimate of the state of the target object.

15. The wearable device of claim 14, wherein the plurality of electrodes comprises: At least one reference electrode providing a reference signal, and At least one measuring electrode for signal measurement.

16. The wearable device of claim 15, wherein the plurality of electrodes further comprises: At least one bias electrode that provides a bias signal.

17. The wearable device of claim 14, wherein the wearable device comprises at least one of the following: headphones, glasses, patches, or headbands.

18. The wearable device of claim 14, further comprising one of the following: A signal amplifier used to amplify the electrophysiological signals. A microcontroller for processing the electrophysiological signals. Storage device for storing the electrophysiological signals, or A communication device for transmitting the electrophysiological signals.

19. The wearable device of claim 14, wherein the plurality of electrodes comprises at least one of the following: One or more left electrodes are configured to be adjacent to the left ear of the target object when the wearable device is worn by the target object, or One or more right electrodes are configured to be adjacent to the right ear of the target object when the wearable device is worn by the target object.

20. A signal acquisition and processing system, comprising: A wearable device includes a plurality of electrodes configured to be placed on the head of a target object when the wearable device is worn, for acquiring at least one electrophysiological signal at at least one location on the head of the target object. as well as A signal processing device, coupled to the wearable device to receive the at least one electrophysiological signal, and the signal processing device is configured to: Acquire at least one electrophysiological signal collected by the wearable device at at least one location within a time period; Based on the at least one electrophysiological signal, a machine learning model is used to generate an estimate of the state of the target object during the time period.

Images