A multimodal video stream object detection method based on brain-eye fusion

Abstract

The present invention discloses a multimodal video stream target detection method based on brain-eye fusion. The method collects EEG data and eye movement data of subjects when they watch complex video content, and produces EEG positive and negative sample pairs, eye movement positive and negative sample pairs, and EEG-eye movement spliced ​​samples spliced ​​in the channel dimension. The positive and negative sample pairs are then input into the feature extraction model for comparative learning and training of model parameters. The EEG samples, eye movement samples and spliced ​​samples are then input into the classifier for learning. Finally, the unknown samples are input into the trained classifier, and the corresponding sample eye movement data predicted to be the target is taken out, the gaze area is drawn according to the gaze point and gaze time, a heat map is generated, the target's motion trajectory is obtained, and target detection is achieved. The method uses multimodal information to search for similar information of the same category, realizes information complementarity, reveals the target's movement trajectory at a higher recognition rate, and improves the reliability of recognition.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer collaboration technology and relates to a target detection method, specifically to a multimodal video stream target detection method based on brain-eye fusion. Background Art

[0002] Classifying and labeling large-scale video datasets to understand and organize video content involves identifying and distinguishing different scenes, actions, objects, etc. from video data. How to locate targets in massive and complex videos has become an important research direction.

[0003] The human brain possesses powerful parallel processing capabilities, leveraging contextual information to understand and identify objects, and rapidly adapting to diverse environments and situations. This enables humans to more accurately understand the meaning and characteristics of objects by linking them to their surroundings and context, and successfully identify them in a variety of complex situations, such as low lighting, cluttered backgrounds, or partial obstructions. These advantages make the human brain highly efficient, accurate, and robust in identifying objects from vast amounts of video information, far exceeding the capabilities of current computer vision systems.

[0004] When the human brain receives a stimulus indicating a low-probability event, a positive peak with a latency of approximately 300 milliseconds is generated in the EEG signal. This signal is named the P300. The P300 is an event-related potential (ERP) that is part of the brain's cognitive process and can be extracted from EEG data. As one of the key ERP features, the P300 has received extensive attention and application in EEG research and applications. Compared to images and videos, processing EEG data requires relatively low computational effort, making it a viable approach to transform complex video processing into extracting effective features from EEG. Brain-computer interfaces (BCIs), technologies that directly connect the human brain to external devices, can record EEG signals when a person detects an object, providing a technical foundation for this purpose. However, in complex videos, the precise timing of object appearance is difficult to pinpoint, making it difficult to capture the ERP waveform generated by the brain during video viewing. Furthermore, the low probability of object appearance in videos results in extremely unbalanced target data, which in turn affects the accuracy of object recognition.

[0005] Using eye movement data to identify objects in complex videos combines the characteristics of the human visual system and cognitive processes. Eye movement data provides the location and movement trajectory of an observer's eyes while watching a video. By analyzing this data, the position of the observer's object of interest in the video can be quickly and accurately determined, achieving efficient target localization. However, since subjects are prone to shifting their gaze due to distractions, this can lead to false positives. Furthermore, eye movement data can also suffer from extreme imbalance in target data.

[0006] Therefore, improving the accuracy of object recognition in complex videos remains a significant challenge. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention proposes a multimodal video stream target detection method based on brain-eye fusion, which uses the pairing of EEG data and eye movement data to achieve multimodal information complementarity, and combines the spatiotemporal multi-head self-attention mechanism to extract cross-membrane features of the attention information of EEG and eye movements, decode and classify them, and use contrastive representation learning to solve the class imbalance problem, thereby improving the target recognition rate in complex videos.

[0008] A multimodal video stream target detection method based on brain-eye fusion, the specific steps are as follows:

[0009] Step 1: Create a brain-computer experiment paradigm

[0010] Videos of target pedestrians were filmed at multiple locations and angles, then edited into streams of equal duration to create a rapid visual presentation paradigm. In videos without the target, distracting objects appeared randomly; in videos with the target, the target's appearance duration was equal, with randomized timing, location, and camera angles, and the distracting objects were dressed differently.

[0011] Step 2: Collect EEG and eye movement data

[0012] The complex video content created in step 1 was played to participants using the Rapid Serial Visual Presentation (RSVP) paradigm within a brain-computer interface (BCI). EEG and eye movement data were simultaneously collected while the participants watched the video content. Before each round of the experiment, the participants' gaze was calibrated, requiring them to focus on the target when it appeared and to follow it. The rest of the time, they remained focused on the center of the screen.

[0013] Step 3: Data preprocessing and sample pair construction

[0014] Read the EEG data collected in step 2, interpolate and reconstruct the bad tracks, then filter and downsample, mark the ERP features when there is no target, when the target appears, and when the target disappears, divide the segments and perform baseline calibration to obtain EEG samples.

[0015] Read the eye movement data collected in step 2, downsample it to the same frequency as the EEG data, mark the gaze point coordinates and the pupil sizes of the left and right eyes when there is no target, when the target appears, and when the target disappears, divide it into segments, and obtain eye movement samples.

[0016] The EEG samples and eye movement samples are matched one by one in sequence, and the corresponding samples are spliced ​​in the channel dimension to obtain spliced ​​samples.

[0017] The obtained EEG samples and eye movement samples are used to construct positive and negative sample pairs according to the labels of no target, target appearance, and target disappearance. Samples with the same label are combined in pairs to form three types of positive sample pairs, and samples with different labels are combined in pairs to form three types of negative sample pairs.

[0018] Step 4: Extract EEG features and eye movement features

[0019] The two samples of the sample pair constructed in step 3 are input into the feature extraction model in sequence. First, the multi-head self-attention mechanism is used to extract the time-space attention weights to obtain the dependency between different dimensions. Then, feature extraction of the time dimension and space dimension is performed to obtain the feature vectors of the two samples.

[0020] Step 5: Contrastive Learning

[0021] The two feature vectors of a sample pair obtained in step 4 are mapped into a vector space. In this vector space, the spatial distance between the two sample vector features is calculated and back-propagated through the loss function to update the parameters of the feature extraction model in step 4. This makes the distance between samples of the same category as small as possible, while the distance between samples of different categories as large as possible, to achieve better representation learning.

[0022] Step 6: Multimodal feature fusion

[0023] The corresponding EEG and eye movement samples from step 3 are fed into the feature extraction model after contrastive learning, while the concatenated samples are fed into the feature extraction model without contrastive learning. The multi-head self-attention mechanism in the feature extraction model explicitly models the interdependencies between different multimodal channels, adapts to the importance of each channel, and extracts transmembrane features from the concatenated samples.

[0024] The features of the three samples are fed into a classifier, which is trained using the downsampled, class-balanced data to obtain independent predictions for the three samples. Then, a set of adaptive learnable weights is trained to fuse the three independent predictions together to produce the final prediction.

[0025] Step 7: Online video target detection

[0026] The EEG data and eye movement data generated by the subjects when watching the video are collected, and after processing, they are input into the feature extraction model and classifier trained in step 6 to obtain predictions of whether the unknown sample has a target, and the appearance and disappearance of the target. Then, the eye movement data of the sample where the prediction result is the appearance of the target is taken out, and the gaze area is drawn according to the gaze point and gaze time. A heat map is generated to obtain the movement trajectory of the target.

[0027] The present invention has the following beneficial effects:

[0028] This method leverages multimodal information to search for similar information within the same category and distinguish features across different categories to achieve information complementarity. Leveraging a convolutional neural network with a multi-head self-attention mechanism, it effectively captures the channel and temporal characteristics of EEG and eye movements. By employing decoupled contrastive representation learning and classifier learning, it enhances the ability to find subtle features of both intra- and inter-class information and addresses the problem of class imbalance. By leveraging the characteristics of EEG and eye movements, it reveals the target's trajectory at a high recognition rate, improving recognition reliability and significantly boosting the accuracy of target recognition in complex videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the multimodal video stream target detection method based on brain-eye fusion;

[0030] Figure 2 This is a flowchart of collecting EEG data and eye movement data in the embodiment;

[0031] Figure 3 Schematic diagram of a model architecture for comparative representation learning in an embodiment;

[0032] Figure 4 Schematic diagram of a module for feature extraction in an embodiment;

[0033] Figure 5 Schematic diagram of the self-attention module structure;

[0034] Figure 6 Schematic diagram of a model architecture for classification training in an embodiment;

[0035] Figure 7 Schematic diagram of motion trajectory of video target detection in an embodiment. DETAILED DESCRIPTION

[0036] The method of the present invention is further explained below with reference to the accompanying drawings. Figure 1 As shown in FIG, a multimodal video stream target detection method based on brain-eye fusion is described. The specific steps are as follows:

[0037] Step 1: Create a brain-computer experiment paradigm

[0038] Surveillance cameras were used to simulate video streams captured at various locations and angles, targeting a specific individual. The video streams were then edited into 10-second segments, with some segments containing the individual and others not.

[0039] In videos without a target, objects and pedestrians unrelated to the target may appear, disrupting the subject's judgment. In videos with a target, the target's appearance time and location are random, and they may appear using different means of transportation and wearing different clothing. In some videos, the target may be partially obscured. Objects and pedestrians unrelated to the target may also appear, disrupting the subject's judgment.

[0040] After editing, in the video clips where the target appears, the target will appear from the 1st to the 4th second of the video and disappear from the 5th to the 8th second of the corresponding video. The duration of the target appearance is 4 seconds.

[0041] Step 2: Collect EEG and eye movement data

[0042] The rapid serial visual presentation (RSVP) paradigm in brain-computer interface (BCI) was adopted. The video edited in step 1 was used as the stimulus video for the experiment. The ratio of target video to non-target video was 1:1. E-Prime 3 was used to produce the experimental paradigm.

[0043] like Figure 2 As shown, the experiment consisted of 16 blocks. At the beginning of each block, the subject clicked the mouse to start the experiment. After a 10-second prompt, fixation calibration was performed. The subject moved their gaze sequentially following the flashing dots on the screen. When the fixation error fell below a set threshold, the experiment began. Each block began with 2 seconds of central fixation, followed by a 10-second video. Each video was separated by a 2-second central fixation, for a total of 25 videos. After all videos were played, a prompt appeared, and the subject clicked the mouse to end the block. The EEG and eye movement data for that session were saved.

[0044] During the experiment, the subjects were asked to sit 70 cm away from the screen and use a 64-channel Neuroscan device for non-invasive EEG acquisition. The 10-20 system electrode placement method was adopted. The impedance of all electrodes was kept below 15KΩ before acquisition, and the sampling rate was 1000 Hz. An SMI eye tracker was used to collect eye movement data. The subjects were asked to look at the center of the screen when the target was not recognized. When the target was observed, the gaze moved with the target and continued to look at the center of the screen after the target disappeared.

[0045] Step 3: Data preprocessing and EEG eye movement data pairing

[0046] Read the EEG data collected in step 2. First, interpolate and reconstruct the marked bad tracks, and then perform Butterworth bandpass filtering to filter the signal to 0.1-40Hz. Next, downsample the data to 250Hz, and mark the ERP features when there is no target, when the target appears, and when the target disappears. Divide the data into multiple segments, and use the EEG data 300ms before the stimulus starts to perform baseline calibration on the EEG data 1500ms after the stimulus starts. For EEG data without a target, slice it at a certain interval as a target-free sample; for data with a target, take the data from 300ms before the target appears to 1200ms after it appears as the target appearance sample, and take the data from 300ms before the target disappears to 1200ms after the target disappears as the target disappearance sample.

[0047] Read the eye movement data collected in step 2 and downsample it to 250Hz. Mark the gaze point coordinates and pupil sizes of the left and right eyes when there is no target, when the target appears, and when the target disappears. Then divide the segments into segments according to the timestamp to obtain eye movement samples.

[0048] Due to the different scales of eye movement and EEG, the number of sample pairs required to compare subtle features is also different. Given an initial value NUM, the processed EEG samples are used to construct positive and negative sample pairs according to the labels of no target, target appearing, and target disappearing. The same category is combined into three positive sample pairs, and different categories are combined into three negative sample pairs. Positive sample pairs and Given the initial value NUM′, the processed eye movement data is constructed in the same way Positive sample pairs and negative sample pairs. NUM′>NUM, in this embodiment, NUM′=2*NUM.

[0049] Use N corresponding EEG samples and eye movement samples Splicing is performed in the channel dimension to obtain N splicing samples Where i = 1, 2, …, N, C represents the number of electrodes in the EEG data, T represents the time length of a sample, and C′ represents the number of channels in the eye movement data.

[0050] Step 4: Extract EEG features and eye movement features

[0051] like Figure 3 As shown in FIG, the constructed EEG positive and negative sample pairs and eye movement positive and negative sample pairs are respectively input into the feature extraction model to extract EEG features and eye movement features.

[0052] like Figure 4As shown in the figure, for each input sample, the multi-head self-attention mechanism is first used to extract the spatiotemporal attention weights, and the dependencies between different dimensions of the input samples are obtained through H self-attention modules, so that the network can adaptively explore the spatiotemporal discriminative features.

[0053] like Figure 5 As shown, the input of each self-attention module is [m,c h ,t], where m represents the number of samples in a batch, h represents the index of the self-attention module, and c h The number of channels of the sample is divided into H parts and then input into the h-th self-attention module. t represents the time node. The input of the self-attention module is respectively compared with the weight matrix W q 、W k 、W v Multiply to get Query(Q h )、Key(K h )、Value(V h ). h and K h Perform a scaled dot product to calculate the degree of similarity, then perform a softmax operation to normalize it, highlight the maximum value of the similarity feature, suppress other components far below the maximum value, and then compare the normalized value with V h Multiplying the weight distribution of the input, we get the output Head (Q h ,K h ,V h ) h :

[0054]

[0055] Among them, d k Represents the dimension of the input vector, used to stabilize the gradient, K h The self-attention module can effectively extract the spatiotemporal attention weights in the sample, showing the interdependence between different multimodal channels and adaptively adjusting the importance of each channel, allowing the network to focus more on discriminative features.

[0056] The outputs of the h self-attention modules are re-joined in the channel dimension to obtain the output of the multi-head self-attention mechanism:

[0057] MultiHead(Q [1,2,…,H] ,K [1,2,…,H] ,V [1,2,…,H] )=Concat(Head1, Head2,…, Head H )

[0058] Concat represents a concatenation operation. The same module explores the importance of different features in space and time. If the same attention mechanism initially focuses on incorrect features, they may be amplified during subsequent learning. The purpose of using multi-head attention is to enable the network to simultaneously learn various attentions, discarding erroneous features and correcting them in a timely manner, thereby extracting the correct features.

[0059] Then, the output of the multi-head self-attention mechanism is passed to the spatiotemporal feature extraction network EEGNet to further extract feature vectors in the temporal and spatial dimensions:

[0060]

[0061]

[0062] in, Represents two sample data in a sample pair, F m () represents the feature extraction operation, Represents the feature vectors obtained after two samples pass through the feature extraction model.

[0063] Step 5: Contrastive Learning

[0064] For the feature vector of a sample pair obtained in step 4, two fully connected layers and one activation layer are used to expand, activate, and reduce the dimension of the feature vector to obtain the corresponding estimated values ​​xz1 and xz2:

[0065]

[0066]

[0067] Among them, δ'() represents the ELU function, W∈R is the parameter of the one-dimensional convolution layer, g represents the activation layer and the one-dimensional adaptive convolution function, are the parameters of the two fully connected layers, r is a hyperparameter, indicating the scaling factor. By projecting, we get a mapping of a sample pair on the same low-dimensional projection space and calculate the cosine similarity z between the two. sim To return the loss function Loss:

[0068]

[0069]

[0070] Where M is the set of negative sample pairs, N is the set of positive sample pairs, and bb represents the bias value, which is a hyperparameter. After calculation, the distance between samples of the same category is as small as possible, while the distance between samples of different categories is as large as possible to achieve better representation learning.

[0071] Step 6: EEG and eye movement fusion

[0072] like Figure 6 As shown, after the representation learning in step 5, the parameters of the previous contrast representation learning stage are fixed, the EEG samples and eye movement samples are input into the feature extraction model that has undergone contrast representation learning, and the spliced ​​samples are input into the feature extraction model that has not undergone contrast representation learning. Then, the features of the three samples are input into the classifier, and the classifier is trained specifically, while the feature extraction module of the spliced ​​samples is trained at the same time. In this embodiment, a fully connected layer and a softmax activation function are used for classification learning. In order to prevent the classifier training from being affected by class imbalance, the non-target samples are downsampled to the same number as the target samples.

[0073] After classifier training is complete, the classifier parameters are fixed and EEG samples, eye movement samples, and EEG-eye movement concatenation samples are fed into the model to obtain the model's three outputs. A set of learnable weight parameters is set and trained to adaptively prioritize the importance of each modality's output. After weight training is complete, the three outputs are weighted and fused as the final output.

[0074] Step 7: Online video target detection

[0075] The EEG data and eye movement data generated by the subjects when watching the video are collected, and after processing, they are input into the model trained in step 6 to obtain predictions of whether the unknown sample has a target, and whether the target appears and disappears. This method detects the appearance and disappearance of the target at the same time. After detecting the appearance or disappearance of the target, it is regarded as a target discovery, and the corresponding sample eye movement data predicted to be the target is taken out. The gaze area is drawn according to the gaze point and gaze time, and a heat map is generated to obtain the target's motion trajectory. The results are as follows Figure 7 As shown, it can be seen that there is a big difference in the motion trajectories of targets and non-targets.

[0076] This embodiment also conducts comparative experiments with other commonly used target detection methods in the prior art, and selects two indicators, F1score and BA, to measure the experimental results:

[0077] F1 score=(2*Precision*Recall) / (Precision+Recall)

[0078] BA=(TP / (TP+FP)+TN / (TN+FN)) / 2

[0079] Among them, Precision is the proportion of true positive samples among samples predicted to be positive, Precision = TP / (TP+FP). Recall is the proportion of correctly predicted positive samples, Recall = TP / (TP+FN). F1score can take into account both precision and recall. TP represents the number of positive classes predicted as positive, FN represents the number of positive classes predicted as negative, FP represents the number of negative classes predicted as positive, and TN represents the number of negative classes predicted as negative. BA is used to characterize the average value of the precision between each class. In the case of sample imbalance, the BA score can be used to judge the classification performance of the algorithm and effectively obtain the local accuracy in each class. The higher the value, the higher the accuracy between each class.

[0080] Therefore, these two evaluation indicators can reflect the performance of the detection model more objectively and comprehensively. The experimental results are shown in the following table:

[0081]

[0082] According to the data in the table, this method has better performance under both evaluation scales and can effectively improve the recognition accuracy.

Claims

1. A multimodal video stream target detection method based on brain-eye fusion, characterized by: The specific steps include: Step 1: Create a brain-computer experiment paradigm Shoot target pedestrian videos at multiple locations and angles, and create a fast visual presentation paradigm; Step 2: Collect EEG and eye movement data Play the video content produced in step 1 to the subjects, and simultaneously collect the subjects' EEG data and eye movement data while watching the video content; Step 3: Data preprocessing and sample pair construction Preprocess the data collected in step 2 to obtain EEG samples and eye movement samples, and construct positive and negative sample pairs based on the labels; then correspond the EEG samples and eye movement samples one by one in sequence, and splice them in the channel dimension to obtain spliced ​​samples; Step 4: Extract EEG features and eye movement features The two samples in a sample pair are sequentially input into the feature extraction model. First, the multi-head self-attention mechanism is used to extract the temporal and spatial attention weights to obtain the dependencies between different dimensions. Then, the features of the temporal and spatial dimensions are extracted to obtain the feature vectors of the two samples. Step 5: Contrastive Learning Map the two feature vectors of a sample pair into the vector space, calculate the spatial distance between the two feature vectors, and backpropagate the spatial distance through the loss function to update the parameters of the feature extraction model; Step 6: Multimodal feature fusion The EEG samples and eye movement samples are sequentially fed into a feature extraction model that has undergone contrastive learning. The corresponding concatenated samples are then fed into a feature extraction model that has not undergone contrastive learning. The feature vectors corresponding to the three samples are obtained. These feature vectors are then fed into a classifier and trained using the downsampled, class-balanced data to obtain independent prediction results for the three samples. A set of adaptive learnable weights is then trained to fuse the three independent prediction results together, thereby outputting the final prediction result. Step 7: Online video target detection The EEG data and eye movement data generated by the subjects when watching the video are collected, and after processing, they are input into the feature extraction model and classifier trained in step 6 to obtain predictions of whether the unknown sample has a target, and the appearance and disappearance of the target. Then, the eye movement data of the sample where the prediction result is the appearance of the target is taken out, and the gaze area is drawn according to the gaze point and gaze time. A heat map is generated to obtain the movement trajectory of the target.

2. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: When creating a brain-computer experiment paradigm, video streams were captured from multiple locations and angles, simulating surveillance cameras. The target was a specific person. The video streams were edited into multiple 10-second video clips, some of which contained the target, while others did not. In the target-free videos, objects and pedestrians unrelated to the target appeared randomly to interfere with the subjects' judgment; In the videos where the target appears, the time and location of the target appear at random, the means of transportation used are random, and the clothing is different. In some videos, the target is partially occluded, and objects and pedestrians unrelated to the target appear randomly. After editing, in the video clips where the target appears, the target will appear from the 1st to the 4th second of the video and disappear from the 5th to the 8th second of the corresponding video. The duration of the target appearance is always 4 seconds.

3. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: Non-invasive EEG data were collected using a 64-channel Neuroscan device with a 10-20 system electrode placement method. The impedance of all electrodes was kept below 15 kΩ before data collection, and the sampling rate was 1000 Hz. Eye movement data were collected using an SMI eye tracker. Participants were instructed to gaze at the center of the screen when the target was not recognized, follow the target when the target was observed, and continue to gaze at the center of the screen after the target disappeared.

4. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: The collected EEG data were preprocessed. First, the marked bad tracks were interpolated and reconstructed, and then Butterworth bandpass filtering was performed to filter the signal to 0.1-40 Hz. Next, the data were downsampled to 250 Hz, and the ERP features when there was no target, target onset, and target disappearance were marked. The data were then divided into multiple segments, and the EEG data 300 ms before the onset of stimulation was used to perform baseline calibration on the EEG data 1500 ms after the onset of stimulation. For the EEG data without a target, slices were taken at certain intervals as no-target samples. For the data with a target, the data from 300 ms before the target onset to 1200 ms after the target onset was taken as the target onset sample, and the data from 300 ms before the target disappearance to 1200 ms after the target disappearance was taken as the target disappearance sample.

5. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: The collected eye movement data was first downsampled to 250Hz; then the gaze point coordinates and left and right eye pupil sizes when there was no target, when the target appeared, and when the target disappeared were marked, and the segments were divided according to the timestamps to obtain eye movement samples.

6. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: The processed EEG samples were used to construct positive and negative sample pairs according to the labels of no target, target appearance, and target disappearance. The same category was combined into three positive sample pairs, and different categories were combined into three negative sample pairs. The number of positive and negative sample pairs was equal. The processed eye movement samples were paired into positive and negative samples in the same way. The number of eye movement sample pairs was greater than the number of EEG sample pairs.

7. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: Use EEGNet to extract features from samples that have passed through the multi-head self-attention mechanism.

8. The method for multimodal video stream target detection based on brain-eye fusion according to claim 1, characterized in that: Use two fully connected layers and one activation layer to expand, activate, and reduce the dimensionality of the feature vectors of the sample pairs to obtain the corresponding estimated values ​​xz1 and xz2: Among them, δ'() represents the ELU function, W∈R is the parameter of the one-dimensional convolution layer, g represents the activation layer and the one-dimensional adaptive convolution function, are the parameters of the two fully connected layers, r is a hyperparameter, indicating the scaling factor; Calculate the cosine similarity z between the estimated values ​​xz1 and xz2 sim To return the loss function Loss: Where M is the set of negative sample pairs, N is the set of positive sample pairs, and bb represents the bias value.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

Images