Health detection method, health detection device, program product, and electronic device

Abstract

The embodiment of the invention relates to a health detection method, a health detection device, a program product and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring electroencephalogram data and video data of a target object; the video data comprises local video data and global video data, and the local video data comprises a preset body part of the target object; processing the electroencephalogram data through an electroencephalogram feature extraction network to obtain an electroencephalogram feature sequence; processing the video data through a video feature extraction network to obtain a video feature sequence; fusing the electroencephalogram feature sequence with the video feature sequence to obtain a multi-modal fusion feature sequence; and inputting the multi-modal fusion feature sequence into a multi-modal fusion processing network for processing, and outputting a health detection result of the target object. The efficiency and reliability of object health detection can be improved.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of artificial intelligence technology, and more specifically, the embodiments of this disclosure relate to a health detection method, a health detection device, a computer program product, and an electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure set forth in the claims, and the description herein is not acknowledged as prior art simply because it is included in this section.

[0003] With the deep integration of brain science and artificial intelligence, brain health has become a research hotspot in related technological fields. Current health detection methods typically rely on the analysis of single bioelectrical signals, which are susceptible to unreliability due to uneven sample distribution. Therefore, improving the efficiency and reliability of health detection is a pressing issue that needs to be addressed by existing technologies. Summary of the Invention

[0004] However, the reliability of current health testing needs to be improved.

[0005] Therefore, there is a great need for a health testing method that can improve the efficiency and reliability of health testing for individuals.

[0006] In this context, embodiments of the present disclosure are intended to provide a health detection method, a health detection device, a computer program product, and an electronic device.

[0007] According to a first aspect of this disclosure, a health detection method is provided, comprising: acquiring electroencephalogram (EEG) data and video data of a target object; the video data including local video data and global video data, wherein the local video data includes preset body parts of the target object; processing the EEG data through an EEG feature extraction network to obtain an EEG feature sequence; processing the video data through a video feature extraction network to obtain a video feature sequence; fusing the EEG feature sequence and the video feature sequence to obtain a multimodal fusion feature sequence; inputting the multimodal fusion feature sequence into a multimodal fusion processing network for processing, and outputting a health detection result of the target object; wherein the EEG feature extraction network, the video feature extraction network, and the multimodal fusion processing network are integrated in the same joint network model.

[0008] In one embodiment, acquiring the EEG data and video data of the target object includes: acquiring continuous EEG signals of the target object and acquiring continuous video streams of the target object; dividing the continuous EEG signals into multiple EEG signal segments according to a preset duration window, and using each EEG signal segment as the EEG data of the target object; and extracting video frames that are time-aligned with each EEG signal segment from the continuous video stream as the video data of the target object.

[0009] In one embodiment, acquiring the continuous video stream of the target object includes: acquiring a continuous global video stream of the target object; extracting video frames that are temporally aligned with each EEG signal segment from the continuous video stream as video data of the target object includes: extracting global video frames that are temporally aligned with each EEG signal segment from the continuous global video stream; detecting and cropping image regions containing preset body parts of the target object from the global video frames to form local video frames corresponding to the global video frames; and using the global video frames and local video frames as video data of the target object.

[0010] In one embodiment, processing the video data through a video feature extraction network to obtain a video feature sequence includes: inputting the video data into the video feature extraction network for processing, so as to extract features of the video data in the spatial and temporal dimensions through the video feature extraction network, and obtaining the video feature sequence.

[0011] In one embodiment, the step of inputting the video data into a video feature extraction network for processing, so as to extract features of the video data in the spatial and temporal dimensions through the video feature extraction network and obtain the video feature sequence, includes: inputting the global video data and the local video data into the video feature extraction network respectively; extracting global features of the global video data in the spatial and temporal dimensions and extracting local features of the local video data in the spatial and temporal dimensions through three-dimensional convolution operations in the video feature extraction network; and fusing the extracted global features and local features to generate the video feature sequence.

[0012] In one embodiment, fusing the EEG feature sequence with the video feature sequence to obtain a multimodal fusion feature sequence includes: splicing the EEG feature sequence and the video feature sequence to obtain an intermediate feature sequence; adding a learnable classification label at the beginning position of the intermediate feature sequence; and using the intermediate feature sequence with the added classification label as the multimodal fusion feature sequence.

[0013] In one embodiment, the step of inputting the multimodal fusion feature sequence into a multimodal fusion processing network for processing to output the health detection result of the target object includes: inputting the multimodal fusion feature sequence containing the classification label into the multimodal fusion processing network; calculating the attention weight between the features in the EEG feature sequence and the video feature sequence through a multi-layer self-attention mechanism in the multimodal fusion processing network, and performing weighted fusion of the features based on the attention weight to update the feature representation of the classification label; performing classification and recognition based on the updated feature representation of the classification label, and obtaining the health detection result of the target object.

[0014] In one embodiment, the method further includes: acquiring training sample data; wherein the training sample data includes time-aligned sample EEG data and sample video data, as well as sample health status labels; inputting the training sample data into a joint network model to be trained for forward propagation to obtain prediction results; the joint network model includes an EEG feature extraction network, a video feature extraction network, and a multimodal fusion processing network to be trained; calculating a training loss based on the prediction results and sample health status labels; and, based on the training loss, simultaneously updating the network parameters in the EEG feature extraction network, the video feature extraction network, and the multimodal fusion processing network through a backpropagation algorithm to obtain a trained joint network model.

[0015] According to a second aspect of this disclosure, a health detection device is provided, comprising: a target data acquisition module for acquiring electroencephalogram (EEG) data and video data of a target object; the video data includes local video data and global video data, the local video data including preset body parts of the target object; an EEG sequence extraction module for processing the EEG data through an EEG feature extraction network to obtain an EEG feature sequence; a video sequence extraction module for processing the video data through a video feature extraction network to obtain a video feature sequence; a feature sequence fusion module for fusing the EEG feature sequence and the video feature sequence to obtain a multimodal fusion feature sequence; and a detection result output module for inputting the multimodal fusion feature sequence into a multimodal fusion processing network for processing and outputting a health detection result of the target object; wherein the EEG feature extraction network, the video feature extraction network, and the multimodal fusion processing network are integrated in the same joint network model.

[0016] According to a third aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the health detection method of the first aspect and its possible implementations.

[0017] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the health detection method of the first aspect and possible implementations thereof by executing the executable instructions.

[0018] In this disclosed solution, on the one hand, this exemplary embodiment integrates physiological signals and visual behavioral features by combining EEG data, local video data, and global video data. This overcomes the information limitations of a single data dimension, effectively alleviates the problem of insufficient reliability of results that easily occurs in the analysis of single data types, and significantly improves the credibility of health detection results. On the other hand, this exemplary embodiment adopts a unified feature extraction and fusion process, avoiding cumbersome processing of multiple independent steps, reducing redundancy between different steps, and reducing the error impact that may be caused by multi-step processing, thus balancing the efficiency and stability of health detection. Attached Figure Description

[0019] Figure 1 A schematic diagram illustrating the system architecture of a health detection method in this exemplary embodiment is shown; Figure 2 This diagram illustrates a flowchart of a health detection method according to this exemplary embodiment; Figure 3 A schematic diagram of the network architecture illustrating the training and inference process of a joint network model in this exemplary embodiment is shown. Figure 4 This diagram illustrates a flowchart of a joint network model training method in this exemplary embodiment. Figure 5 This diagram illustrates the structure of a health monitoring device according to this exemplary embodiment. Figure 6 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown.

[0020] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0021] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0022] The embodiments of this disclosure can be implemented as a health detection method, a health detection device, a computer program product, or an electronic device. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0023] The principles and spirit of this disclosure are explained in detail below with reference to several representative embodiments. Invention Overview

[0025] The inventors have discovered that the efficiency and reliability of current health detection methods need improvement. Specifically, related technologies for brain health detection typically suffer from the following limitations: First, analysis methods based on single EEG signals are affected by factors such as low EEG signal-to-noise ratio, susceptibility to signal interference, or uneven sample distribution, resulting in insufficient accuracy and reliability of the detection results. Second, although existing technologies have attempted multimodal fusion, they mostly employ non-end-to-end processing methods, i.e., first independently extracting features from EEG and other modalities (such as video) and then performing subsequent fusion. This approach is cumbersome, inefficient, and makes it difficult to achieve sufficient interaction and deep integration of cross-modal information, thus hindering the overall performance improvement of health detection.

[0026] In view of the foregoing, this disclosure provides a health detection method, a health detection device, a computer program product, and an electronic device. On one hand, this exemplary embodiment integrates physiological signals and visual behavioral features by combining EEG data, local video data, and global video data, overcoming the information limitations of a single data dimension and effectively mitigating the reliability issues that easily arise in single-data-type analysis, thus significantly improving the credibility of health detection results. On the other hand, this exemplary embodiment employs a unified feature extraction and fusion process, avoiding cumbersome processing of multiple independent steps, reducing redundancy between different steps, and mitigating the error impact that may arise from multi-step processing, thus balancing the efficiency and stability of health detection.

[0027] The various non-limiting embodiments of this disclosure are described in detail below.

[0028] Application Scenarios Overview

[0029] It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the implementation of this disclosure is not limited in any way. On the contrary, the implementation of this disclosure can be applied to any applicable scenario.

[0030] The embodiments disclosed herein can be applied to scenarios involving the detection of whether a user's health status is abnormal. The application scenario is described in detail below in conjunction with the system architecture.

[0031] Figure 1 A schematic diagram of a system architecture for a health detection method is shown. The system 100 may include a user 110, a terminal device 120, and a server 130. The terminal device 120 may be an electronic device configured with external detection devices, such as a computer or wearable device configured with a camera and an EEG acquisition device. The server 130 may be the main body for data processing, and may be a server or a cloud-based system. The terminal device 120 can collect EEG data and video data from the user 110 through the configured external detection devices, and forward the raw data, or preprocessed data, to the server 130. The server 130 can process the received data using a configured EEG feature extraction network, a video feature extraction network, and a multimodal fusion processing network to generate health detection results, which are then returned to the terminal device 120.

[0032] Exemplary methods

[0033] Exemplary embodiments of this disclosure provide a health detection method. (See reference...) Figure 2 As shown, the method may include steps S210 to S250. The following describes... Figure 2 Each step in the process will be explained in detail.

[0034] refer to Figure 2 In step S210, the EEG data and video data of the target object are acquired; the video data includes local video data and global video data, and the local video data includes preset body parts of the target object.

[0035] The target audience refers to individuals requiring brain health status testing, such as suspected epilepsy patients, the elderly, and athletes undergoing high-intensity training. Electroencephalogram (EEG) data refers to signal data reflecting the electrical activity of neurons in the target audience's brain, such as 32-channel EEG signals acquired by an electroencephalogram (EEG) machine, 16-channel dynamic EEG waveforms recorded by an EEG device, or electrical signals from local intracranial regions acquired by an EEG sensor. Video data can be dynamic image sequences reflecting the target audience's physical state, acquired through image acquisition devices or obtained from data sources. Examples include continuous video streams captured by high-definition cameras, which may include behavioral characteristics such as limb movements and facial expressions; nighttime video data captured by infrared cameras, which may include behavioral characteristics of the target audience in specific states (such as sleep); and panoramic videos recorded by multi-view cameras.

[0036] In this exemplary embodiment, the video data may include local video data and global video data. Local video data may include videos of preset body parts of the target object, used to accurately capture behavioral characteristics of key areas, such as videos of the face or upper limbs, the head, or the neck and shoulder areas. Global video data may be complete video data containing the entire body of the target object and its surrounding environment, reflecting the overall behavioral state and environmental context of the target object. Examples include videos of the target object's entire body and its surrounding environment captured by a panoramic camera, panoramic videos of elderly people's daily activities recorded by a wide-angle camera in a living room, or full-process videos of athletes training or competing simultaneously from multiple cameras in a sports field. Preset body parts may be key monitoring areas pre-defined according to the detection target, and can be determined by the detection scenario and health detection needs. For example, preset body parts may be one or more of the face, upper limbs, head, neck, and shoulders.

[0037] In this exemplary embodiment, local video data can be extracted from global video data, for example, by identifying preset body parts in global video data and cropping the global video data to obtain local video data. Local video data can also be directly acquired, for example, by configuring a panoramic camera for acquiring global video data and a dedicated camera for capturing facial areas, acquiring global video data through the global camera and acquiring local video data through the dedicated camera, etc.

[0038] In one embodiment, for epilepsy detection, the target subject can be a suspected epilepsy patient. Continuous EEG signals can be collected using an electroencephalogram (EEG) machine, and a global video stream of the patient's entire body and ward environment can be collected using a high-definition panoramic camera synchronized with the EEG machine. Simultaneously, a target detection algorithm is used to crop local video streams of the patient's face and upper limbs in real time. In another embodiment, for brain health detection in the elderly, the target subject is an elderly person. Continuous EEG data and a global video stream can be received in real time from wearable devices at home, such as portable EEG devices or smartwatch cameras. Local video data of the head is extracted using predefined region cropping rules.

[0039] In step S220, the EEG data is processed by an EEG feature extraction network to obtain an EEG feature sequence.

[0040] Among them, EEG feature extraction networks refer to neural networks used to extract effective features from raw EEG data. They can capture the temporal and channel correlation information of EEG signals. EEG feature extraction networks can be ConvNeXtV2 networks, RNNs (Recurrent Neural Networks), or GRUs (Gate Recurrent Units), etc. EEG feature sequences refer to the ordered high-dimensional feature sets obtained after EEG data is processed by feature extraction networks. They are abstract representations of EEG signals, such as the feature sequences output by ConvNeXtV2; the EEG amplitude variation and time-frequency feature sequences extracted by RNNs; or the EEG rhythm correlation feature sequences output by GRUs.

[0041] In this exemplary embodiment, EEG data is processed by an EEG feature extraction network to obtain an EEG feature sequence. Specifically, for example, the collected 32-channel EEG data is preprocessed (e.g., filtered using a bandpass filter to remove power line interference and baseline drift), and then reshaped (which can be achieved through the reshape function or method, and then filled in by padding) into a 256×160 two-dimensional rectangular EEG signal. This signal is then input into the pre-trained ConvNeXtV2 EEG feature extraction network, and after multi-layer convolution, activation and pooling operations, the EEG feature sequence EEG-Tokens is output.

[0042] In step S230, the video data is processed by a video feature extraction network to obtain a video feature sequence.

[0043] Video feature extraction networks are network models used to extract spatial (morphological, structural) and temporal (motion, change) features from video data. These networks can be X3D-M (a lightweight 3D convolutional neural network model for video action classification), Two-Stream CNN, or ViT (VisionTransformer), etc. Video feature sequences are ordered sets of features that fuse spatial and temporal information, obtained after processing video data by the feature extraction network. Examples include the fused feature sequence output by X3D-M; the spliced ​​feature sequence of static texture and dynamic optical flow output by Two-Stream CNN; or the temporal correlation feature sequence of key video regions extracted by ViT.

[0044] In this exemplary embodiment, video data is processed by a video feature extraction network to obtain a video feature sequence. Specifically, for example, global video data and local video data can be input into a pre-trained X3D-M video feature extraction network, and spatial dimension body posture / facial expression features and temporal dimension action frequency / change trend features can be extracted simultaneously through three-dimensional convolutional layers, and the video feature sequence video-Tokens can be fused and output.

[0045] In step S240, the EEG feature sequence and the video feature sequence are fused to obtain a multimodal fused feature sequence.

[0046] The multimodal fusion feature sequence refers to a unified feature set formed by integrating EEG feature sequences and video feature sequences, achieving complementarity between physiological and visual signals. In this exemplary embodiment, the multimodal fusion feature sequence can be a fusion feature sequence obtained by splicing feature dimensions, or a fusion feature sequence with a classification label added after splicing feature dimensions. For example, splicing EEG feature sequences and video feature sequences and adding a classification label at the beginning of the sequence yields a fusion feature sequence. Alternatively, it can be a weighted summation fusion feature sequence based on attention weights. For example, first determining the importance weights of EEG feature sequences and video feature sequences (weight values ​​can be learned or customized as needed), and then adding the values ​​of these two types of feature sequences according to their importance weights to finally obtain a fusion feature sequence. It can also be a fusion feature sequence in which features are added element by element. For example, directly adding the values ​​of EEG feature sequences and video feature sequences that correspond to each other in position is equivalent to merging feature values ​​at the same position to obtain a fusion feature sequence.

[0047] In this exemplary embodiment, the EEG feature sequence and the video feature sequence are fused to obtain a multimodal fusion feature sequence. Specifically, for example, the EEG feature sequence [B, T-eeg, C] and the video feature sequence [B, T-video, C] can be concatenated along the time dimension to obtain the concatenated feature sequence [B, T-eeg+T-video, C]. A learnable classification identifier cls-token (dimension [B, 1, C]) is added at the beginning of the sequence, finally forming the multimodal fusion feature sequence [B, 1+T-eeg+T-video, C]. Here, in [B, 1+T-eeg+T-video, C], B can represent the batch size, T-eeg can represent the time dimension of the EEG data, T-video can represent the time dimension of the video data, and C can represent the feature dimension. T-eeg and T-video represent the time steps of the corresponding modality at the feature level, which may differ from the number of frames / sample points at the original sampling rate. Before concatenation, EEG feature sequences and video feature sequences can be mapped to the same feature dimension C through linear layers or 1×1 convolutions, so that they can be concatenated in the time dimension.

[0048] In step S250, the multimodal fusion feature sequence is input into the multimodal fusion processing network for processing, and the health detection result of the target object is output; wherein, the EEG feature extraction network, the video feature extraction network, and the multimodal fusion processing network are integrated into the same joint network model.

[0049] Multimodal fusion processing networks refer to the core networks used to process multimodal fusion features, mine cross-modal associations, and output detection results. They can be a 6-layer ROPE (Rotary Position Embedding)-Transformer model, which achieves deep interaction of cross-modal features through a self-attention mechanism; an MLP (Multilayer Perceptron) fusion model, which achieves nonlinear transformation and classification of features through fully connected layers; or a GNN (Graph Neural Network) fusion model, which constructs a cross-modal association graph to extract global features.

[0050] Health test results refer to the assessment of the target individual's brain health status, including whether there are any abnormalities, or if abnormalities are present, specific information such as the type or severity of the abnormality. For example, whether epileptic seizures occur, the specific type of seizure (e.g., temporal lobe-initialized seizures, frontal lobe-initialized seizures, centroparietal lobe-initialized seizures, occipital lobe-initialized seizures, typical absence seizures, atypical absence seizures, myoclonic epilepsy), and the confidence level of the seizure; the level of brain health status (e.g., normal, mildly abnormal, moderately abnormal, severely abnormal); and the probability of risk for specific brain diseases (e.g., risk of epileptic seizures, risk of cognitive impairment).

[0051] In this exemplary embodiment, the multimodal fusion feature sequence is input into a multimodal fusion processing network for processing, and the health detection result of the target object is output. Specifically, for example, the multimodal fusion feature sequence with dimensions [B, T, C] can be input into a 6-layer ROPE-Transformer model. Through a multi-layer self-attention mechanism, deep interaction and weighted fusion of EEG features and video features are achieved. Based on the feature representation of the updated classification identifier cls-token, the epileptic seizure type (such as temporal lobe seizure, typical absence seizure) and confidence level are output through the MLP classification head. Here, B represents the batch size, T represents the time dimension, and C represents the feature dimension.

[0052] In this exemplary embodiment, the aforementioned EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are integrated into the same joint network model. That is, the three networks can be integrated into a unified network architecture, sharing training objectives and parameter optimization processes to achieve a closed-loop processing flow from data input to result output. For example, during the model building phase, the ConvNeXtV2 EEG feature extraction network, the X3D-M video feature extraction network, and the 6-layer ROPE-Transformer multimodal fusion processing network can be encapsulated into a unified joint network model. The EEG extraction network processes EEG data, the video extraction network processes video data, and the multimodal fusion processing network processes the fused feature sequences. The parameters of all three networks are registered under the same model instance. During training, the Adam (Adaptive Moment Estimation) optimizer can synchronously update the convolutional kernel weights, attention weights, and other parameters of all networks, achieving end-to-end optimization. During inference, the user can input EEG data and video data into the joint network model, and the model automatically completes feature extraction, fusion, and result output without requiring additional independent modules.

[0053] In one exemplary embodiment, the acquisition of the target object's EEG data and video data may include: Acquire continuous EEG signals of the target object, and acquire continuous video streams of the target object; The continuous EEG signal is divided into multiple EEG signal segments according to a preset duration window, and each EEG signal segment is used as the EEG data of the target object. From a continuous video stream, video frames that are temporally aligned with each EEG signal segment are extracted as video data for the target object.

[0054] Continuous EEG signals refer to uninterruptedly acquired raw signals that reflect the continuous changes in the electrical activity of the target's brain over time. They form the foundation of EEG data. Examples include continuous 32-channel EEG signals acquired by an EEG machine at a sampling rate of 250 Hz for at least 24 hours; or 16-channel continuous signals acquired by a portable EEG device at a sampling rate of 128 Hz for at least 8 hours. Continuous video streams refer to dynamic image sequences recorded uninterruptedly by image acquisition devices that reflect the continuous changes in the target's behavioral state over time. Examples include continuous panoramic video streams recorded by high-definition cameras at 30 frames per second, which can be acquired synchronously with EEG signals; continuous nighttime video streams recorded by infrared cameras at 25 frames per second, suitable for monitoring in low-light environments; and first-person perspective continuous video streams recorded by wearable cameras at 15 frames per second, suitable for monitoring mobile scenes.

[0055] In this exemplary embodiment, the acquisition of continuous EEG signals and continuous video streams of the target object can be specifically achieved, for example, in a hospital ward setting, by using a clinical EEG machine (32 channels, 250Hz) and a high-definition panoramic camera (30 frames / second) to acquire continuous EEG signals and continuous global video streams of suspected epilepsy patients, respectively, and by using a device hardware synchronization interface to ensure that the start time of acquisition is consistent and the timestamp error is ≤10 milliseconds; or in a home setting, elderly users can wear a portable EEG device (16 channels, 128Hz) and a smart camera (25 frames / second) to acquire continuous EEG signals and continuous video streams, respectively, and by using a Network Time Protocol (NTP) to calibrate the timestamps to ensure time synchronization, etc.

[0056] A preset duration window refers to a pre-defined time segment length used to divide continuous EEG signals. The window size can be determined according to the detection requirements and data characteristics; for example, the preset duration window can be set to 4 seconds, 8 seconds, etc. An EEG signal segment refers to an independent time segment obtained after dividing continuous EEG signals through the preset duration window. It is the basic processing unit of EEG data, such as a 4-second 32-channel EEG signal segment or an 8-second 16-channel EEG signal segment. In this exemplary embodiment, the continuous EEG signal is divided according to a preset duration window to obtain multiple EEG signal segments, and each EEG signal segment is used as the EEG data of the target object. Specifically, for example, a continuous 32-channel EEG signal sampled at 250Hz is divided non-overlappingly using a fixed duration window of 4 seconds, with each window containing 1000 sampling points (250Hz × 4 seconds), to obtain multiple independent EEG signal segments, which are directly used as EEG data for subsequent processing; or a continuous 16-channel EEG signal sampled at 128Hz is divided using a sliding partition with a fixed duration window of 8 seconds and a step size of 4 seconds, with each window containing 1024 sampling points (128Hz × 8 seconds), and adjacent windows overlapping by 4 seconds, and the resulting EEG signal segments are used as EEG data, etc.

[0057] Temporally aligned video frames refer to a set of video frames that perfectly match the time interval of an EEG signal segment, ensuring the temporal synchronization of physiological signals and behavioral characteristics. For example, 120 frames of video data (30 frames / second × 4 seconds) corresponding to a 4-second EEG segment; or 200 frames of video data (25 frames / second × 8 seconds) corresponding to an 8-second EEG segment, etc. In this exemplary embodiment, the above-mentioned extraction of video frames that are temporally aligned with each EEG signal segment from a continuous video stream as video data of the target object can be, for example, for a 4-second EEG signal segment, extracting 120 frames of video data (30 frames / second × 4 seconds) from a continuous video stream of 30 frames / second that perfectly match the time interval of the segment, ensuring that the timestamps of the video frames correspond one-to-one with the timestamps of the EEG segments, and these 120 frames of video data can be used as video data of the target object; or for an 8-second EEG signal segment, extracting 200 frames of video data (25 frames / second × 8 seconds) from a continuous video stream of 25 frames / second, calibrating the start and end times of the video frames and the EEG segments using timestamp matching software to ensure that the alignment error is ≤20 milliseconds, and these 200 frames of video data can be used as video data, etc.

[0058] In one exemplary embodiment, the acquisition of the continuous video stream of the target object may include: Acquire a continuous global video stream of the target object; The above-mentioned extraction of video frames that are temporally aligned with each EEG signal segment from a continuous video stream, as the video data of the target object, may include: Extract global video frames that are temporally aligned with each EEG signal segment from a continuous global video stream; From the global video frame, detect and crop out the image region containing the preset body parts of the target object to form a local video frame corresponding to the global video frame; Use global video frames and local video frames as video data for the target object.

[0059] Continuous global video streams refer to a complete sequence of dynamic images, including the entire body of the target object and its surrounding environment, recorded uninterruptedly from a global perspective by an image acquisition device. Examples include a panoramic camera in a hospital ward recording a video stream of the patient's entire body and the surrounding environment at 30 frames per second; or a wide-angle camera in a home living room recording a video stream of an elderly person's entire body and the living room environment at 25 frames per second. Global video frames refer to a complete set of video frames extracted from the continuous global video stream and time-aligned with the EEG signal segments, preserving global perspective information. Examples include 120 panoramic video frames (30 frames per second × 4 seconds) aligned with a 4-second EEG segment; or 200 wide-angle video frames (25 frames per second × 8 seconds) aligned with an 8-second EEG segment.

[0060] Detection and cropping refers to the process of identifying the location of a preset body part of a target object using algorithms or manual methods, and then extracting the corresponding image region from the global video frame. For example, object detection algorithms (such as YOLOv8) can automatically identify and crop the face and upper limbs; or the coordinate range of the preset body parts can be manually marked for batch cropping of video frames; or human keypoint detection algorithms can be used to locate head joints and adaptively crop the head region. Local video frames refer to the set of video frames obtained after detection and cropping that only contain the preset body parts of the target object, focusing on key monitoring areas, such as cropped video frames of the face and upper limbs, or cropped video frames of the head region. In this exemplary embodiment, an image region containing a preset body part of the target object is detected and cropped from the global video frame to form local video frame data corresponding to the global video frame data. Specifically, for example, the preset body part can be the face and upper limbs. The YOLOv8 target detection algorithm is used to identify the bounding box coordinates of the face and upper limbs frame by frame in 120 global video frames. A 128×128 image region is cropped according to the bounding box. The cropped image of 120 frames forms local video frame data, which corresponds one-to-one with the global video frame data. Alternatively, the preset body part can be the head. The coordinate range of the head in the first 10 global video frames is manually marked to establish a head position template. Subsequent frames automatically locate the head based on template matching and crop a 256×256 local video frame, which corresponds to the global video frame. Or, the preset body part can be the neck and shoulders. Human key point detection algorithms (such as OpenPose) are used to locate the neck and shoulder joints. The cropping region is adaptively generated according to the joint coordinates, and the global video frames are cropped frame by frame to obtain local video frames.

[0061] Finally, global and local video frames can be used as video data for the target object. For example, 120 global and 120 local video frames can be stored sequentially to form the target object's video data, which can then be synchronously input into a video feature extraction network for processing. Alternatively, 200 global and 200 local video frames can be converted to RGB color mode and aligned by frame index for use as video data for subsequent feature extraction. Or, 100 global and 100 local video frames can be compressed, stored in association, and used as video data to reduce the computational cost of subsequent processing.

[0062] In an exemplary embodiment, the aforementioned EEG data is EEG data in the form of a two-dimensional matrix; the aforementioned processing of the EEG data through an EEG feature extraction network to obtain an EEG feature sequence may include: The EEG data in two-dimensional matrix form is input into the EEG feature extraction network for processing to obtain the EEG feature sequence.

[0063] In this context, two-dimensional matrix EEG data refers to EEG data with width and height structures, preserving the channel correlation and temporal information of the EEG signals. Two-dimensional matrix EEG data can adapt to the input requirements of convolutional neural networks. In this exemplary embodiment, the original EEG data can be in two-dimensional matrix form, or it can be EEG data in two-dimensional matrix form obtained after processing the original EEG data. For example, for an EEG signal segment with 32 channels × 1000 sampling points (4 seconds, 250Hz), EEG Packing operation can be used to first normalize the EEG data, such as mapping the amplitude to [0, 1024] µV, and then reconstruct it into a 256 × 160 two-dimensional matrix EEG data through shape adjustment operations (such as reshape operation). The width can correspond to the time dimension, and the height can correspond to the channel dimension.

[0064] Furthermore, EEG data in two-dimensional matrix form can be input into an EEG feature extraction network for processing to obtain an EEG feature sequence. For example, 256×160 two-dimensional matrix EEG data can be input into a pre-trained ConvNeXtV2 EEG feature extraction network. In this exemplary embodiment, the ConvNeXtV2 EEG feature extraction network can adopt a staged convolutional backbone structure. Specifically, spatial resolution can be gradually reduced and channel dimension increased through convolution and downsampling. Each stage consists of several stacked ConvNeXtV2 modules. Within each module, depthwise separable convolution, normalization, and activation functions are combined to extract multi-level features. The ConvNeXtV2 EEG feature extraction network can capture key discriminative features of EEG signals while compressing data dimensionality through hierarchical feature extraction and aggregation, ultimately outputting an EEG feature sequence.

[0065] In an exemplary embodiment, the above-described process of processing video data through a video feature extraction network to obtain a video feature sequence may include: Video data is input into a video feature extraction network for processing, in order to extract features of the video data in the spatial and temporal dimensions and obtain a video feature sequence.

[0066] Among these features, spatial features can be static attribute features extracted from single-frame video images. These features do not rely on time-series information and only reflect the instantaneous state within a single frame, such as the facial contours of the target object, limb joint positions, body posture, the relative position of the target object with surrounding objects, and the layout of environmental objects. Temporal features can be dynamic change attribute features extracted from continuous multi-frame video sequences. Relying on the temporal correlation between frames, these features can reflect the evolution of the target object or environment over time, such as the target object's limb movement speed, facial twitching frequency, duration of action, body displacement trajectory, amplitude of posture changes, action interval period, frame-by-frame change trend of ambient light, and direction of movement of the target object in consecutive frames.

[0067] Video data is input into a video feature extraction network for processing. This network extracts features from the video data in both spatial and temporal dimensions to obtain a video feature sequence. For example, global and local video frames can be input into the X3D-M video feature extraction network. Spatial convolutional kernels extract spatial features such as facial contours and limb joint positions, while temporal convolutional kernels extract temporal features such as limb movement speed and facial twitching frequency. After multiple convolutions, activation, and pooling operations, the spatial and temporal features are fused to output a video feature sequence. Alternatively, video data can be input into a dual-stream CNN network. The spatial-stream CNN extracts spatial features such as clothing color and body posture of the target object through 2D convolutions, while the temporal-stream CNN extracts temporal features such as motion displacement trajectory and posture change amplitude through optical flow. The two streams of features are then concatenated to obtain a video feature sequence.

[0068] In an exemplary embodiment, the above-described input of video data into a video feature extraction network for processing, to extract features of the video data in the spatial and temporal dimensions through the video feature extraction network and obtain a video feature sequence, may include: The global video data and local video data are respectively input into the video feature extraction network; By using the 3D convolution operation in the video feature extraction network, global features of global video data in spatial and temporal dimensions are extracted, as well as local features of local video data in spatial and temporal dimensions. The extracted global and local features are fused to generate a video feature sequence.

[0069] Among them, 3D convolution operations can be performed on the width, height, and time dimensions of video data, and can simultaneously capture the correlation information of spatial and temporal features. Global features refer to features extracted from global video data that reflect the overall behavioral state of the target object and its relationship with the environment, such as the trend of whole-body movement, changes in overall body posture, and the symmetry of whole-body movements; local features refer to features extracted from local video data that reflect the detailed state of preset body parts of the target object, such as the amplitude of facial muscle twitching, the degree of upper limb rigidity, the frequency of head shaking, and the amplitude of shoulder movement.

[0070] This exemplary embodiment can input global video data and local video data into a video feature extraction network, and extract global features of the global video data in both spatial and temporal dimensions, as well as local features of the local video data in both spatial and temporal dimensions, through three-dimensional convolution operations in the video feature extraction network. For example, the three-dimensional convolutional layer in the X3D-M network can use a 3×3×3 kernel size to perform convolution calculations on the global video data. In the spatial dimension, it captures global spatial features such as whole-body posture and environmental layout, and in the temporal dimension, it captures global temporal features such as whole-body movement trends and displacement speed, and outputs global features. Similarly, for local video data, in the spatial dimension, it captures local spatial features such as facial contours and upper limb joint positions, and in the temporal dimension, it captures local temporal features such as facial twitching frequency and upper limb movement amplitude, and outputs local features.

[0071] Furthermore, the extracted global and local features can be fused to generate a video feature sequence. For example, the global and local features can be directly concatenated along the feature dimension (the last dimension) to obtain fused features. Then, a 1×1 convolution can be used to reduce the dimensionality and generate the final video feature sequence. Alternatively, the association weights of the global and local features (weight range [0, 1]) can be calculated through an attention mechanism, and the two features can be weighted and summed according to the weights (e.g., global feature × 0.6 + local feature × 0.4) to obtain the video feature sequence.

[0072] In an exemplary embodiment, the above-described fusion of EEG feature sequences and video feature sequences to obtain a multimodal fusion feature sequence may include: By splicing EEG feature sequences and video feature sequences, an intermediate feature sequence is obtained. At the beginning of the intermediate feature sequence, add a learnable classification label; The intermediate feature sequence with added classification labels is used as the multimodal fusion feature sequence.

[0073] In this exemplary embodiment, splicing the EEG feature sequence and the video feature sequence can be achieved by connecting two or more feature sequences in the time dimension, integrating feature information from different modalities, and forming a feature sequence with a higher dimension. The intermediate feature sequence is the feature sequence formed after splicing. For example, the EEG feature sequence [B, T-eeg, C] and the video feature sequence [B, T-video, C] can be spliced ​​in the time dimension, and the spliced ​​feature sequence is [B, T-eeg+T-video, C], where T-eeg represents the time dimension of the EEG data and T-video represents the time dimension of the video data.

[0074] Learnable classification labels refer to trainable vectors added to the intermediate feature sequence to carry classification information. Their parameters are continuously updated as the model is trained. They can represent the core classification information of multimodal fusion features. For example, learnable classification labels can be cls-tokens, classification vectors, or feature anchors.

[0075] At the beginning of the intermediate feature sequence, a learnable classification label is added. For example, for the intermediate feature sequence [B, T-eeg+T-video, C], a learnable cls-token is added at the beginning of the sequence. The initial value of the cls-token can be randomly generated, such as having a dimension of [B, 1, C], which is consistent with the feature dimension of the concatenated feature sequence. After adding it, the feature sequence dimension becomes [B, 1+T-eeg+T-video, C].

[0076] Finally, the feature sequence with added classification labels can be used as the multimodal fusion feature sequence. For example, the feature sequence [B, 1+T-eeg+T-video, C] after adding cls-token can be directly used as the multimodal fusion feature sequence. This sequence contains one classification label and multimodal concatenation features with T-eeg+T-video time steps. It is then input into the multimodal fusion processing network for feature interaction and classification.

[0077] In one exemplary embodiment, a multimodal fusion feature sequence is input into a multimodal fusion processing network for processing to output a health detection result of a target object, including: The multimodal fusion feature sequence containing classification labels is input into the multimodal fusion processing network; By using the multi-layer self-attention mechanism in the multimodal fusion processing network, the attention weights between features in the EEG feature sequence and the video feature sequence are calculated, and the features are weighted and fused based on the attention weights to update the feature representation of the classification label. Based on the feature representation of the updated classification label, classification and recognition are performed, and the health detection results of the target object are obtained.

[0078] Among them, the multi-layer self-attention mechanism refers to a feature interaction mechanism composed of multiple stacked self-attention layers. Each self-attention layer can realize information interaction and weighted aggregation of features of different modalities and different time steps by calculating the correlation weights between features. For example, the 6-layer ROPE-Transformer self-attention mechanism can contain 8 attention heads per layer; the 4-layer standard Transformer self-attention mechanism can contain 12 attention heads per layer, etc.

[0079] This exemplary embodiment calculates the attention weights between features in the EEG feature sequence and the video feature sequence, and performs weighted fusion of features based on the attention weights. This allows EEG features and video features to pay attention to each other's key information, and aggregates them according to the attention weights to form more discriminative fused features, strengthening cross-modal association. For example, there is bidirectional attention interaction between EEG features and video features, which are assigned weights to each other and fused with weights; or unidirectional attention fusion with EEG features as the core, and video features assisting in correcting EEG features; or dynamic weight fusion based on feature importance, which adaptively adjusts modal weights at different time steps, etc.

[0080] The feature representation of a classification label refers to the core classification information of the multimodal fusion features carried by the classification label after undergoing multiple layers of self-attention interaction. It is the specific label value of the classification label, such as the updated cls-token feature data, the updated classification vector, the updated feature anchor point feature data, etc.

[0081] In this exemplary embodiment, by using a multi-layer self-attention mechanism in the fusion model to interact and weightedly fuse features in the EEG feature sequence and the video feature sequence, the feature representation of the classification label can be updated. For example, a 6-layer ROPE-Transformer model can be used, in which each layer of the self-attention mechanism contains 8 attention heads. By calculating the attention weights of EEG features and video features at different time steps, bidirectional interaction of cross-modal features can be achieved (e.g., EEG abnormality features pay attention to the corresponding video behavior features, and video twitching features pay attention to the corresponding EEG discharge features). The interactive features are formed by weighted fusion. At the same time, the classification label (cls-token) continuously updates its own feature representation by interacting with the fused features at all time steps, and finally carries global multimodal fusion information. The updated cls-token dimension is still [B, 1, C].

[0082] Classification and recognition refers to the process by which, based on the updated classification label feature representation, a classifier can determine the health status of a target object and output the detection result. For example, the updated classification label feature representation can be input into an MLP classification head (which may contain two fully connected layers and a Softmax activation function). The classification head outputs 10 probability distributions (corresponding to 10 states such as normal, temporal lobe seizures, and frontal lobe seizures). The category with the highest probability is taken as the epilepsy detection result, and the confidence level of that category is output (e.g., temporal lobe seizures, confidence level 0.92). Alternatively, it can be input into a Softmax classifier for classification and recognition, outputting a probability distribution of 4 health status levels (normal, mild abnormality, moderate abnormality, and severe abnormality), and the level with the highest probability is taken as the detection result (e.g., moderate abnormality, confidence level 0.85). Or, it can be input into an SVM (Support Vector Machine) classifier for classification and recognition. During training, a specific brain disease risk label is used as the target, and the classifier outputs three detection results: "high risk," "medium risk," and "low risk" (e.g., high risk of epileptic seizures, confidence level 0.88), etc.

[0083] In one exemplary embodiment, the method may further include: Acquire training sample data; the training sample data includes time-aligned sample EEG data and sample video data, as well as sample health status labels; Training sample data is input into the joint network model to be trained for forward propagation to obtain prediction results; the joint network model includes the EEG feature extraction network, video feature extraction network, and multimodal fusion processing network to be trained. Calculate the training loss based on the prediction results and the health status labels of the samples; Based on the training loss, the network parameters in the EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are updated simultaneously through the backpropagation algorithm to obtain the trained joint network model.

[0084] In this exemplary embodiment, the entire process from training sample data to detection results can be completed by a joint network model, and the network parameters of all networks in the joint network model can be collaboratively optimized through a unified loss function.

[0085] Training sample data refers to the labeled data set used for model training, which may include input data and corresponding ground truth labels, such as time-aligned sample EEG data and sample video data, as well as sample health status labels. Time-aligned sample EEG data and sample video data mean that the time intervals of the sample EEG data and sample video data are completely matched, ensuring that the multimodal samples input into the model are consistent in the time dimension.

[0086] Sample health status labels refer to the real health status information corresponding to the sample data, which is manually or automatically labeled and used to calculate the model prediction error. Examples include epileptic seizure type labels (normal, temporal lobe seizure, etc.); health status level labels (normal, mild abnormality, etc.); or disease risk labels (high risk, medium risk, low risk), etc.

[0087] The joint network model to be trained refers to an overall model containing multiple networks that has not yet completed parameter optimization. This model may include an EEG feature extraction network, a video feature extraction network, and a multimodal fusion processing network to be trained.

[0088] Forward propagation refers to the process where sample training data flows into the model's input layer, is processed layer by layer by each network, and finally outputs the prediction result from the output layer. For example, sample data → EEG feature extraction network → video feature extraction network → multimodal fusion processing network → prediction result. The prediction result refers to the predicted health status output by the model after forward propagation, which is compared with the true label to calculate the training loss. Backpropagation is an algorithm that uses the error calculated based on the loss function to propagate backward from the model's output layer to the input layer, calculating the gradients of each network parameter. Network parameters refer to the trainable weights, biases, and other parameters in the model. Through backpropagation and optimizer updates, the model gradually converges. Examples include the convolutional kernel weights in ConvNeXtV2, the 3D convolutional weights in X3D-M, and the attention weights in Transformer.

[0089] In this exemplary embodiment, the joint network model to be trained may include a ConvNeXtV2 EEG feature extraction network to be trained, an X3D-M video feature extraction network to be trained, and a 6-layer ROPE-Transformer multimodal fusion processing network to be trained. Sample EEG data is input into the ConvNeXtV2 network to extract EEG feature sequences, and sample video data is input into the X3D-M network to extract video feature sequences. The two feature sequences are fused and then input into the ROPE-Transformer multimodal fusion processing network. After forward propagation, the network outputs a predicted probability distribution of epileptic seizure types. Then, a cross-entropy loss function is applied. The model outputs the predicted probability distribution of epileptic seizure types (preset results) and compares it with the labeled real labels (sample health status labels). The classification loss of each training sample is calculated, and the average of the losses of all samples is taken as the global loss function value. Finally, the Adam optimizer can be used to calculate the gradient of the global loss function with respect to all parameters of the ConvNeXtV2, X3D-M, and ROPE-Transformer networks based on the backpropagation algorithm. The convolutional kernel weights, biases, attention weights, and other parameters of each network are updated according to the gradient descent direction. The training is iterated until the loss function value converges to obtain the trained joint network model.

[0090] Figure 3 The diagram illustrates the network architecture for the training and inference process of the joint network model in this exemplary embodiment, which may specifically include the following stages: In the pre-training stage of the feature extraction network, the EEG feature extraction network 311 and the video feature extraction network 312 can first undergo transfer learning. For example, the EEG feature extraction network 311 can be pre-trained 330 using a large-scale image dataset 320 to learn general visual feature extraction capabilities; the video feature extraction network 312 can be pre-trained 350 using a large-scale video action dataset 340 to learn the ability to extract spatiotemporal features from videos.

[0091] In the end-to-end joint training phase, the pre-trained EEG feature extraction network 311 and video feature extraction network 312 are integrated with the multimodal fusion processing network 313 to construct a joint network model 310 to be trained. A training sample set is obtained, and the joint network model is trained end-to-end. During this process, the parameters of the EEG feature extraction network and the video feature extraction network are further adjusted to better adapt to the health detection task.

[0092] In the inference application phase, EEG data 360 and video data 370 of the target object are acquired and processed by the trained EEG feature extraction network 311 and video feature extraction network 312, respectively, to obtain EEG feature sequence 314 and video feature sequence 315. The EEG feature sequence 314 and video feature sequence 315 are then spliced ​​and fused to form a multimodal fusion feature sequence 316, which is input into the trained multimodal fusion processing network 313 for inference. The data after deep fusion and processing by the multimodal fusion processing network 313 is passed to the classifier 380 (such as a multilayer perceptron classification head). Finally, the classifier outputs the health detection result 390 of the target object, which may include the detection of epileptic seizures and the specific seizure type classification.

[0093] In this exemplary embodiment, the loss function used to calculate the training loss is configured as follows: For each of the multiple health status categories, calculate the corresponding category loss component; When the target category label is missing in the health status label of a sample, the category loss component corresponding to that target category is ignored when calculating the training loss.

[0094] In this exemplary embodiment, when calculating the loss function used for training loss, the corresponding category loss component can be calculated for each of the multiple health state categories, and the total training loss can be determined based on all the category loss components.

[0095] When calculating the categorical loss component, it may occur that some samples in the training data are not labeled with specific health status labels, including single-label missing, multi-label missing, etc., such as missing labels for specific seizure types (e.g., some samples are not labeled "typical absence seizure"). The target category refers to the category in which the label is missing.

[0096] When a target category label is missing from the health status labels of a sample, the category loss component corresponding to that target category can be ignored when calculating the training loss. That is, for samples with missing labels, only the loss calculation corresponding to the missing label is skipped, while the loss contribution from other labeled labels is retained, without discarding the entire sample. This approach avoids training bias caused by missing labels while fully utilizing the effective annotation information of the samples.

[0097] Figure 4 The flowchart illustrating a joint network model training method in this exemplary embodiment is shown. Specifically, it may include: acquiring EEG data 360 and video data 370 of the target object as input data and inputting them into a joint network model 310; processing the EEG data 360 through an EEG feature extraction network 311 to obtain an EEG feature sequence; processing the video feature extraction network 312 to obtain a video feature sequence; fusing the EEG feature sequence and the video feature sequence to form a multimodal fusion feature sequence, and inputting it into a multimodal fusion processing network 313 based on rotational position coding (such as ROPE-Tr). In the EEG feature extraction network 311, the multimodal fusion feature sequence is processed by the multimodal fusion processing network 313 to output the prediction result 317 of health status. The prediction result 317 is compared with the externally provided sample health status label 318, and the training loss 319 is calculated based on the difference between the prediction result 317 and the sample health status label 318. The training loss 319 is used to update the network parameters in the EEG feature extraction network 311, video feature extraction network 312 and multimodal fusion processing network 313 through the backpropagation algorithm, thereby optimizing the joint network model 310.

[0098] Exemplary device

[0099] Exemplary embodiments of this disclosure also provide a health monitoring device. (See reference...) Figure 5As shown, the health detection device 500 may include the following program modules: a target data acquisition module 510, used to acquire EEG data and video data of the target object; the video data includes local video data and global video data, and the local video data includes preset body parts of the target object; an EEG sequence extraction module 520, used to process the EEG data through an EEG feature extraction network to obtain an EEG feature sequence; a video sequence extraction module 530, used to process the video data through a video feature extraction network to obtain a video feature sequence; a feature sequence fusion module 540, used to fuse the EEG feature sequence and the video feature sequence to obtain a multimodal fusion feature sequence; and a detection result output module 550, used to input the multimodal fusion feature sequence into a multimodal fusion processing network for processing and output the health detection result of the target object; wherein, the EEG feature extraction network, the video feature extraction network, and the multimodal fusion processing network are integrated into the same joint network model.

[0100] In one embodiment, the object data acquisition module includes: a continuous signal acquisition unit, used to acquire continuous EEG signals of the target object and acquire a continuous video stream of the target object; an EEG signal segmentation unit, used to divide the continuous EEG signals according to a preset duration window to obtain multiple EEG signal segments, and use each EEG signal segment as the EEG data of the target object; and a video data alignment unit, used to extract video frames that are temporally aligned with each EEG signal segment from the continuous video stream as the video data of the target object.

[0101] In one embodiment, the continuous signal acquisition unit includes: a global video acquisition subunit for acquiring a continuous global video stream of the target object; and a video data alignment unit including: a global video alignment subunit for extracting global video frames that are temporally aligned with each EEG signal segment from the continuous global video stream; a local video cropping subunit for detecting and cropping image regions containing preset body parts of the target object from the global video frames to form local video frames corresponding to the global video frames; and a video data determination subunit for using the global video frames and local video frames as video data of the target object.

[0102] In one embodiment, the video sequence extraction module includes: a feature extraction unit, used to input video data into a video feature extraction network for processing, so as to extract features of the video data in the spatial and temporal dimensions through the video feature extraction network and obtain a video feature sequence.

[0103] In one embodiment, the feature extraction unit includes: a data input subunit for inputting global video data and local video data into a video feature extraction network respectively; a global and local feature extraction subunit for extracting global features of the global video data in the spatial and temporal dimensions and extracting local features of the local video data in the spatial and temporal dimensions through three-dimensional convolution operations in the video feature extraction network; and a feature fusion subunit for fusing the extracted global and local features to generate a video feature sequence.

[0104] In one embodiment, the feature sequence fusion module includes: an intermediate sequence acquisition unit for splicing EEG feature sequences and video feature sequences to obtain an intermediate feature sequence; a classification label addition unit for adding learnable classification labels at the beginning position of the intermediate feature sequence; and a fusion sequence determination unit for using the intermediate feature sequence with added classification labels as a multimodal fusion feature sequence.

[0105] In one embodiment, the feature sequence fusion module includes: a fusion sequence input unit, used to input a multimodal fusion feature sequence containing classification labels into a multimodal fusion processing network; a feature update unit, used to calculate the attention weights between features in the EEG feature sequence and the video feature sequence through a multi-layer self-attention mechanism in the multimodal fusion processing network, and to perform weighted fusion of features based on the attention weights to update the feature representation of the classification labels; and a classification and recognition unit, used to perform classification and recognition based on the updated feature representation of the classification labels, and to obtain the health detection result of the target object.

[0106] In one embodiment, the health detection device further includes: a training data acquisition module for acquiring training sample data; wherein the training sample data includes time-aligned sample EEG data and sample video data, as well as sample health status labels; a prediction result acquisition module for inputting the training sample data into a joint network model to be trained for forward propagation to obtain prediction results; the joint network model includes an EEG feature extraction network, a video feature extraction network, and a multimodal fusion processing network to be trained; a loss calculation module for calculating training loss based on the prediction results and sample health status labels; and a network parameter update module for simultaneously updating the network parameters in the EEG feature extraction network, the video feature extraction network, and the multimodal fusion processing network based on the training loss and through a backpropagation algorithm to obtain a trained joint network model.

[0107] Furthermore, other specific details of the embodiments of this disclosure have been described in detail in the embodiments of the above methods, and will not be repeated here.

[0108] Exemplary program product

[0109] An exemplary embodiment of this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the above-described health detection method.

[0110] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0111] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0112] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0113] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to be executed by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure.

[0114] Exemplary electronic devices

[0115] Exemplary embodiments of this disclosure also provide an electronic device, which may be Figure 1 Any device in the present disclosure. The electronic device includes a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the methods described above by executing the executable instructions.

[0116] refer to Figure 6 An electronic device according to an exemplary embodiment of the present disclosure will be described. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0117] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including storage unit 620 and processing unit 610).

[0118] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 610 can perform actions such as... Figure 2 The methods and steps shown are as follows.

[0119] Storage unit 620 may include volatile storage units, such as random access memory (RAM) 621 and / or cache storage unit 622, and may further include read-only memory (ROM) 623.

[0120] Storage unit 620 may also include a program / utility having a set (at least one) program module 624, such program module 624 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0121] Bus 630 may include a data bus, an address bus, and a control bus.

[0122] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via input / output (I / O) interface 640. Electronic device 600 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 650. As shown, network adapter 650 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] It should be noted that although several modules or sub-modules of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0124] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0125] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A health detection method, characterized in that, include: Acquire EEG data and video data of the target object; the video data includes local video data and global video data, and the local video data includes preset body parts of the target object; The EEG data is processed by an EEG feature extraction network to obtain an EEG feature sequence; The video data is processed by a video feature extraction network to obtain a video feature sequence; The EEG feature sequence and the video feature sequence are fused to obtain a multimodal fusion feature sequence; the multimodal fusion feature sequence is a fusion feature sequence obtained by concatenating the EEG feature sequence and the video feature sequence in the feature dimension and adding classification labels. The multimodal fusion feature sequence is input into a multimodal fusion processing network for processing, and the health detection result of the target object is output. The EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are integrated into the same joint network model. The joint network model is obtained through end-to-end training of training sample data including sample health status labels. During training, the network parameters of the EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are updated simultaneously based on the training loss.

2. The method according to claim 1, characterized in that, The acquisition of the target object's EEG data and video data includes: Acquire continuous electroencephalogram (EEG) signals of the target object and acquire continuous video streams of the target object; The continuous EEG signal is divided into multiple EEG signal segments according to a preset duration window, and each EEG signal segment is used as the EEG data of the target object. From the continuous video stream, video frames that are temporally aligned with each EEG signal segment are extracted as video data of the target object.

3. The method according to claim 2, characterized in that, The acquisition of the continuous video stream of the target object includes: Obtain the continuous global video stream of the target object; Extracting video frames that are temporally aligned with each EEG signal segment from the continuous video stream as video data for the target object includes: From the continuous global video stream, extract global video frames that are temporally aligned with each EEG signal segment; From the global video frame, detect and crop out the image region containing the preset body part of the target object to form a local video frame corresponding to the global video frame; The global video frames and local video frames are used as the video data of the target object.

4. The method according to claim 1, characterized in that, The step of processing the video data through a video feature extraction network to obtain a video feature sequence includes: The video data is input into a video feature extraction network for processing, so as to extract the features of the video data in the spatial and temporal dimensions through the video feature extraction network and obtain the video feature sequence.

5. The method according to claim 4, characterized in that, The step of inputting the video data into a video feature extraction network for processing, so as to extract the features of the video data in the spatial and temporal dimensions through the video feature extraction network, and obtain the video feature sequence, includes: The global video data and the local video data are respectively input into the video feature extraction network; The video feature extraction network uses 3D convolution operations to extract global features of global video data in both spatial and temporal dimensions, as well as local features of local video data in both spatial and temporal dimensions. The extracted global and local features are fused to generate the video feature sequence.

6. The method according to claim 1, characterized in that, The step of fusing the EEG feature sequence with the video feature sequence to obtain a multimodal fusion feature sequence includes: The EEG feature sequence and the video feature sequence are spliced ​​together to obtain an intermediate feature sequence; At the beginning position of the intermediate feature sequence, a learnable classification label is added; The intermediate feature sequence with the added classification label is used as the multimodal fusion feature sequence.

7. The method according to claim 6, characterized in that, The step of inputting the multimodal fusion feature sequence into a multimodal fusion processing network for processing, and outputting the health detection result of the target object, includes: The multimodal fusion feature sequence containing the classification label is input into the multimodal fusion processing network; The attention weights between the features in the EEG feature sequence and the video feature sequence are calculated using the multi-layer self-attention mechanism in the multimodal fusion processing network, and the features are weighted and fused based on the attention weights to update the feature representation of the classification label. Based on the updated feature representation of the classification label, classification and identification are performed, and the health detection results of the target object are obtained.

8. The method according to claim 1, characterized in that, The method further includes: Acquire training sample data; wherein, the training sample data includes time-aligned sample EEG data and sample video data, as well as sample health status labels; The training sample data is input into the joint network model to be trained for forward propagation to obtain the prediction result; the joint network model includes an EEG feature extraction network, a video feature extraction network, and a multimodal fusion processing network to be trained. Calculate the training loss based on the prediction results and the sample health status labels; Based on the training loss, the network parameters in the EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are updated simultaneously through the backpropagation algorithm to obtain the trained joint network model.

9. A health monitoring device, characterized in that, include: The object data acquisition module is used to acquire the electroencephalogram (EEG) data and video data of the target object; the video data includes local video data and global video data, and the local video data includes preset body parts of the target object. The EEG sequence extraction module is used to process the EEG data through an EEG feature extraction network to obtain an EEG feature sequence. The video sequence extraction module is used to process the video data through a video feature extraction network to obtain a video feature sequence; The feature sequence fusion module is used to fuse the EEG feature sequence with the video feature sequence to obtain a multimodal fusion feature sequence; the multimodal fusion feature sequence is a fusion feature sequence obtained by concatenating the EEG feature sequence and the video feature sequence in the feature dimension and adding classification labels. The detection result output module is used to input the multimodal fusion feature sequence into the multimodal fusion processing network for processing, and output the health detection result of the target object; The EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are integrated into the same joint network model. The joint network model is obtained through end-to-end training of training sample data including sample health status labels. During training, the network parameters of the EEG feature extraction network, video feature extraction network, and multimodal fusion processing network are updated simultaneously based on the training loss.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 8 by executing the executable instructions.

Images