A method for identifying complex targets in massive videos based on human-computer collaboration

Abstract

The application discloses a kind of based on man-machine cooperation's mass video complex target retrieval method.Currently, the generalization ability of intelligent system based on machine vision is weak, when environment changes, target exists shielding or presents camouflage state, still need a lot of manpower intervention in mass video retrieval target, time-consuming and laborious.The application first according to the coarse-grained feature of target uses retrieval model in video library and carries out pre-screening, and the target candidate set obtained by screening is made into brain-eye cooperation RSVP paradigm and presented to subject.Subject determines target, and its specific features are input into retrieval model, so as to realize the fast positioning and tracking of target in mass video.The application combines the generalization reasoning ability of human and the rapid retrieval ability of machine, can effectively improve the efficiency and accuracy of video investigation, has strong scientific significance and social significance.

Description

Technical Field

[0001] This invention belongs to the field of video target retrieval technology, and relates to a method for identifying massive video targets based on human-computer collaboration. Background Technology

[0002] With the development of modern society, surveillance cameras have become indispensable equipment in combating crime and terrorism, and therefore their numbers have been growing rapidly. A city-wide surveillance network can quickly generate a large amount of video data, and how to quickly locate and track a given target in a massive video database has become a major challenge.

[0003] Thanks to advancements in artificial intelligence, research on machine learning-based video object detection technology has garnered widespread attention and yielded a series of excellent results. Examples include two-level object detection frameworks such as R-CNN, Fast R-CNN, and Faster R-CNN, and faster single-level frameworks like YOLO and SSD. While computers can retrieve data quickly and operate stably for extended periods, most machine vision-based intelligent methods are data-driven, resulting in weak generalization capabilities. Changes in the environment, occlusion of targets, or camouflage often render machine vision models unusable, preventing them from replacing humans in object retrieval tasks. Although the human brain possesses powerful generalization and reasoning abilities and can respond quickly to visually perceived images, relying solely on manual review of massive amounts of video information is often time-consuming, labor-intensive, and inefficient. Furthermore, prolonged viewing of videos can easily cause visual fatigue and even vision damage for reviewers, impacting work quality and harming their health.

[0004] If the advantages of machine intelligence and human intelligence can be combined and complemented to build a hybrid intelligent retrieval system, then this system will achieve better results than any single intelligent mode. Compared with the efficiency of machines, the retrieval efficiency of humans has become the key factor limiting the performance of hybrid intelligent systems. Thanks to the development of brain-computer interface technology, people can interact with the external environment by directly decoding brain activity, which makes efficient target retrieval possible. Therefore, this paper proposes a human-machine collaborative method for complex target recognition in massive video scenarios to improve the target retrieval efficiency in such scenarios. Summary of the Invention

[0005] One objective of this invention is to address the shortcomings of existing technologies by proposing a method for complex target recognition in massive video datasets based on human-machine collaboration. This method integrates human and machine intelligence to accomplish complex target retrieval tasks in massive video scenarios.

[0006] A method for recognizing complex targets in massive video datasets based on human-machine collaboration includes the following steps:

[0007] Step 1: Build and train the object detection model and the re-identification model.

[0008] Step 2: Video Preprocessing. Extract video frame images from the retrieved surveillance video data; use a target detection model to detect and analyze the video frame images, framing and cropping the targets in the video frame images to obtain cropped images; use a re-identification model to extract the features of the targets in the cropped images.

[0009] Step 3: Build and calibrate the EEG classification model. The EEG classification model determines whether the user whose EEG signals were collected observed the target.

[0010] Step 4: Human-machine collaborative video target retrieval

[0011] Step 4-1: Using a re-identification model, pre-screen the cropped images identified by the target detection model based on the target's coarse-grained features. The re-identification model selects n cropped images based on confidence levels. Extract the video frame images corresponding to each cropped image.

[0012] Step 4-2: Provide the subjects with the video frames selected in Step 4-1 for viewing. While the subjects are viewing the video frames, simultaneously record EEG signals and eye movement information. The obtained EEG signals are preprocessed and then input into the EEG classification model. When the EEG classification model determines that the subject has observed a target, the eye movement data of the subject while viewing that video frame is processed to extract the region of interest (ROI) for that video frame. Candidate target sets are then extracted from each ROI.

[0013] Step 4-3: Select several target images from the candidate target set obtained in Step 4-2 that need to be retrieved.

[0014] Step 4-4: Use the re-identification model to extract features from each target image obtained in Step 4-3. Then, perform similarity matching between the extracted features of each target image and the features of video frame images in the retrieved surveillance video data. Select video frame images corresponding to features with similarity exceeding a threshold as the images containing the target (i.e., the retrieval results).

[0015] Preferably, the object detection model described in step 1 uses a Faster R-CNN model retrained on the CoCo dataset. The model is trained for 12 epochs and optimized using a stochastic gradient descent optimizer with a learning rate of 0.005.

[0016] Preferably, the re-identification model described in step 1 uses the ResNet50 model. The average pooling layer in the ResNet50 model is replaced with an adaptive average pooling layer.

[0017] Preferably, the feature vector of the re-identification model has a dimension of 512.

[0018] As a preferred option, the specific process of step 3 is as follows:

[0019] Step 3-1: Prepare stimulation images for several test blocks in advance. Each test block contains multiple trials, and each trial contains multiple images.

[0020] Step 3-2: Present the stimulus images to the subjects and collect EEG data.

[0021] Step 3-3: Preprocess the collected EEG data.

[0022] Steps 3-4: Construct a deep learning-based EEG classification model to perform binary classification of EEG signals. The EEG classification model is trained using decoupled representation learning, dividing the training into two stages: the first stage trains the feature extractor, and the second stage trains the classifier.

[0023] Preferably, in step 3-1, the ratio of the number of stimulus images containing the target to the number of stimulus images not containing the target in each trial is 1:10.

[0024] Preferably, in step 3-2, there is a 5-second fixation time before each trial begins. After the trial begins, each stimulus image is presented for 500ms. After each trial, the subject rests for 10 seconds. The subject can rest for any length of time between two trials. When presenting stimulus images to the subject, there must be at least one stimulus image that does not contain the target between any two adjacent stimulus images containing the target.

[0025] Preferably, the preprocessing in step 3-3 is as follows: a 2-40Hz bandpass filter is used to remove voltage drift and high-frequency noise from the EEG signal, and the EEG signal is downsampled to 100Hz. The downsampled EEG signal is then segmented, with each stimulus image presentation as the starting point, and a 1s segment of the EEG signal is taken as a sample.

[0026] As a preferred embodiment, the process of training the EEG classification model in steps 3-4 is as follows:

[0027] Step 3-4-1: Construct triplet samples. To train the feature extractor using the triplet loss function, triplet samples need to be constructed. x anchor Sample and x positive The samples belong to the same category, x negative The sample belongs to another class.

[0028] Step 3-4-2: Train the feature extractor to extract spatiotemporal information from the EEG data. The feature extractor consists of a multilayer perceptron and a recurrent network layer. The multilayer perceptron is constructed from four fully connected layers, an ELU activation function, and residual connections. The recurrent network layer uses a long short-term memory network.

[0029] The feature h output by the feature extractor is:

[0030]

[0031]

[0032]

[0033] in, Here, x represents the weights of the fully connected layer, σ is the ELU activation function, and x... i Let i be an element in the triplet sample, where i = anchor, positive, negative. These are the intermediate layer output and the final output of the multilayer perceptron, respectively.

[0034] Flatten the feature h and use a projection layer to map the feature h to a low-dimensional sample space to obtain the feature projection h. i 'as follows:

[0035] h i =W6ELU(W5h i +b5)+b6

[0036] in, b5 and b6 are the weights of the projection layer. Calculate the projection h of different sample features. i The similarity between ''; the expression for the similarity dis(a,b) between the feature projections a and b of any two samples in a triplet is as follows:

[0037] dis(a,b)=||a-b+∈e||2

[0038] Where ∈ is a preset parameter. e is a one-dimensional vector.

[0039] Next, the feature extractor is trained using the triplet loss function, L, which is defined as follows:

[0040] L = max{dis(h) anchor ′,h positive ′)-dis(h anchor ′,h negative ′)+margin,0}

[0041] Here, margin represents the minimum distance between positive and negative samples.

[0042] Step 3-4-3: Resample the training samples. The number of samples in the resampled EEG data sample space X is M.

[0043] Step 3-4-4: Train the classifier. The classifier consists of a fully connected layer, defined as follows:

[0044]

[0045] in, is the predicted label of the classifier; W0 is the weight matrix; b0 is the bias term.

[0046] The classifier is trained by minimizing the cross-entropy loss function, as shown in the following equation:

[0047]

[0048] in, Let y be the predicted label of the classifier for the i-th sample, and y be the true label of the i-th sample. M is the number of samples in the resampled EEG data sample space X. The classifier is trained using the Adam optimizer.

[0049] As a preferred embodiment, the specific process of eye-tracking data processing described in step 4-2 is as follows:

[0050] Step 4-2-1: Extract fixation points from the raw eye-tracking data and identify them as a set of continuous points within a specific dispersion.

[0051] The process of extracting fixation points is as follows: First, the eye movement points are sorted chronologically. Then, a sliding window is used to scan consecutive eye movement points. The initial size of the sliding window is determined by a duration threshold. The dispersion D of the eye movement points within the sliding window is calculated using the following formula:

[0052] D=[max(x)-min(x)]+[max(y)-min(y)]

[0053] Where max(x) and min(x) represent the x-coordinates of the leftmost and rightmost eye-moving points in the plane, and max(y) and min(y) represent the y-coordinates of the topmost and bottommost eye-moving points in the plane.

[0054] If the dispersion is less than the threshold, the next eye-tracking point is added to the sliding window until the dispersion is greater than the threshold. At this point, the average coordinates of the eye-tracking points in the sliding window are calculated as the coordinates of a fixation point. When the dispersion is greater than the threshold, the window is moved to the right, and the separation is recalculated. This process is repeated until all eye-tracking points have been scanned.

[0055] Step 4-2-2: Based on the gaze points extracted in Step 4-2-1, calculate the Euclidean distance between each pair of gaze points, and use a density clustering algorithm to generate multiple regions of interest, as shown in the following formula:

[0056]

[0057] Among them, F i ,F j For two different fixation points, (x i ,y i ), (x j ,y j (F) represents the fixation point F. i F j The coordinates. If the distance d between two fixation points is less than the threshold d. ∈ The coordinates of the two fixation points are averaged and then merged to generate a new fixation point.

[0058] Step 4-2-3: For each stimulus image, select the region of interest generated at the end as the target region determined by human vision.

[0059] Step 4-2-4: Compare the bounding box region identified by the object detection model with the target region determined by human vision, select the bounding box region that is closest to the target region determined by human vision as the final target region, and add it to the candidate target set.

[0060] The beneficial effects of this invention are:

[0061] This invention combines the advantages of both human and machine intelligence, making up for the inherent shortcomings of low efficiency in human recognition and weak generalization ability of machine intelligence. It can effectively improve the efficiency and accuracy of video investigation in massive video scenarios, and has significant social implications. Attached Figure Description

[0062] Figure 1 This is a flowchart of a method for recognizing complex targets in massive videos based on human-machine collaboration, according to the present invention.

[0063] Figure 2 This is a design diagram of the brain-eye RSVP experimental paradigm for a human-machine collaborative method for recognizing complex targets in massive video data according to the present invention. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings.

[0065] like Figure 1 As shown, a method for recognizing complex targets in massive videos based on human-machine collaboration specifically includes the following steps:

[0066] Step 1: Training the deep learning-based object detection and re-identification models

[0067] Step 1-1: Training the deep learning-based object detection model. Based on the pre-trained Faster R-CNN model provided by PyTorch, the model is retrained using the CoCo dataset to fine-tune it. The model is trained for 12 epochs, optimized using a stochastic gradient descent optimizer with a learning rate of 0.005.

[0068] Steps 1-2: Training the Deep Learning-Based Re-identification Model. A pre-trained ResNet50 model provided by PyTorch is used as the backbone network of the re-identification model to achieve better image feature extraction. Market-1501 is used as the training dataset for the model. Since the Market-1501 dataset only contains 751 classes of data, the classifier structure of the model needs to be modified for training. Simultaneously, the average pooling layer in ResNet50 is replaced with an adaptive average pooling layer. The model is trained for 60 epochs, optimized using a stochastic gradient descent optimizer with a learning rate of 0.05.

[0069] Step 2: Preprocessing of surveillance video

[0070] The existing surveillance video data is extracted at a rate of 1 frame per second; the target detection model obtained in step 1-1 is used to detect and analyze the video frame images, and objects in the video frames are bounded and cropped according to the region suggestions provided by the model; the re-identification model described in step 1-2 is used to extract the features of the objects in the cropped image, specifically:

[0071] Step 2-1: For high frame rate surveillance videos, since there is almost no difference between adjacent video frames, in order to improve retrieval efficiency, the video frames are sampled at a frequency of 1 frame per second and saved locally. The naming method of the video frames is: video name_current frame number.jpg.

[0072] Step 2-2: Use the object detection model trained in Step 1-1 to detect and analyze the video frame images. The object detection model outputs corresponding region suggestions based on the category of the target to be retrieved. Based on the region suggestions, the objects at the corresponding positions in the video frame are cropped and saved locally. The naming format of the cropped image is: video name_current frame number_current object number.jpg.

[0073] Step 2-3: Use the re-identification model trained in Step 1-2 to extract features from all the object cropping images described in Step 2-2. The feature vector has a dimension of 512. The features are saved locally as a dictionary: the key is the cropping image name, and the value is the cropping image feature vector. The image feature file is named pic_feature.mat.

[0074] Step 3: Calibrating the EEG classification model

[0075] Create using pre-prepared stimulating images, such as Figure 2 The RSVP target detection paradigm shown is used for EEG classification model calibration. This step only requires collecting EEG signals and does not require recording eye movement data.

[0076] Step 3-1: Prepare stimulus images for two trial blocks in advance. Each trial block contains 10 trials, and each trial contains 110 images, including 10 target stimulus images. When presenting the stimulus, ensure that there is at least one non-target image between the two target images.

[0077] Step 3-2: The subject sits upright in front of the screen. Once the equipment is ready, the experiment begins and EEG data is collected. There is a 5-second fixation period before each trial to allow the subject to focus their attention. After the trial begins, each stimulus image is presented for 500ms. After each trial (i.e., after all 110 images in the trial have been displayed), the subject has a 10-second rest period. The subject can rest for any length of time between trials until they feel ready to proceed to the next trial.

[0078] Step 3-3: Preprocess the EEG data from the two acquired test blocks. A 2-40Hz bandpass filter was used to remove voltage drift and high-frequency noise from the EEG signals, and the EEG signals were downsampled to 100Hz. The downsampled EEG signals were then segmented, with each stimulus image presentation as the starting point, and a 1-second segment of the EEG signal was extracted as one sample. For the two test blocks, a total of 2200 samples were obtained, with the EEG data dimensions being (2200, 62, 100). Here, 62 represents the number of channels, and 100 represents the number of sampling points.

[0079] Steps 3-4: Construct a deep learning-based EEG classification model to perform binary classification of EEG signals. Because the ratio of target to non-target stimulus images is 1:10, the EEG data exhibits a long-tailed distribution. Class imbalance significantly impacts model performance, causing the classifier to perform poorly on the tail classes. Since high-quality representation learning is unaffected by class imbalance, a decoupled representation learning approach is used to train the model, dividing the training into two stages: the first stage trains the feature extractor, and the second stage trains the classifier.

[0080] The training process of the EEG classification model described in steps 3-4 is as follows:

[0081] Step 3-4-1: Constructing Triple Samples. To train the feature extractor using the triple loss function, triples need to be constructed as training samples, in the following form:

[0082] Triplet(anchor,positive,negative)

[0083] In this dataset, anchor samples and positive samples belong to the same category, while negative samples belong to other categories. The EEG data sample space is defined as follows: N is the number of samples, T is the number of sampling points, and C is the number of channels for EEG data. Three samples are randomly selected from X. x1 and x2 are used as anchor samples and positive samples, respectively, and x3 is used as a negative sample. A total of 10,000 triplet samples are constructed, of which the anchor samples in 5,000 triplet samples are target samples, and the anchor samples in the remaining 5,000 triplet samples are non-target samples.

[0084] Step 3-4-2: Training the feature extractor to extract spatiotemporal information from EEG data. The feature extractor consists of a multilayer perceptron and recurrent network layers, used to extract spatiotemporal information from EEG data. The multilayer perceptron is constructed from four fully connected layers, an ELU activation function, and residual connections, used to aggregate channel information from EEG data. The recurrent network layer uses a Long Short-Term Memory (LSTM) network; the recurrent network layer is used to extract temporal activity features of each spatial source in the EEG data. Anchor samples, positive samples, and negative samples are defined as samples x1, x2, and x3, respectively; for each triplet sample... The feature h output by the feature extractor is defined as:

[0085]

[0086]

[0087]

[0088] in, Here, x represents the weights of the fully connected layer, σ is the ELU activation function, and x... i Let i be an element in the triplet sample, where i = anchor, positive, negative. These are the intermediate layer output and the final output of the multilayer perceptron, respectively.

[0089] The obtained features h are flattened and mapped to a low-dimensional sample space using a projection layer. The projection layer is only used to train the feature extractor and does not participate in model classification. Feature projection h i ′ is represented as:

[0090] h i =W6ELU(W5h i+b5)+b6

[0091] in, b5 and b6 are the weights of the projection layer. Pairwise distance is used to calculate the feature projection h of different samples. i The similarity between ''; the expression for the similarity dis(a,b) between the feature projections a and b of any two samples in a triplet is as follows:

[0092] dis(a,b)=||a-b+∈e||2

[0093] Where ∈ is a preset small value close to 0, taken as 1×10. -6 e is a vector of all ones (i.e., a vector whose elements are all 1s).

[0094] Next, the feature extractor is trained using the triplet loss function, L, which is defined as follows:

[0095] L = max{dis(h) anchor ′,h positive ′)-dis(h anchor ′,h negative ′)+margin,0}

[0096] Here, margin represents the minimum distance between positive and negative samples, which is set to 2.

[0097] Step 3-4-3: Resample the training samples to train the classifier. Since the ratio of non-target samples to target samples is 10:1, a severe class imbalance problem exists. Therefore, a downsampling method is used to randomly select samples from the non-target samples, making the number of non-target samples equal to the number of target samples. The number of samples in the resampled EEG data sample space X is M.

[0098] Step 3-4-4: Train the classifier. The classifier consists of a fully connected layer, defined as follows:

[0099]

[0100] in, is the predicted label of the classifier; W0 is the weight matrix; b0 is the bias term.

[0101] For the feature extractor obtained in step 3-4-2, its parameters are frozen so that it does not participate in the training of the classifier. The classifier is trained by minimizing the cross-entropy loss function, as shown in the following equation:

[0102]

[0103] in, Let y be the predicted label of the classifier for the i-th sample, and y be the true label of the i-th sample. M is the number of samples in the resampled EEG data sample space X. The classifier is trained using the Adam optimizer with a learning rate of 0.0001.

[0104] Step 4: Human-machine collaborative video target retrieval

[0105] Step 4-1: Using the re-identification model described in Step 1-2, pre-screen the cropped images identified by the target detection model in Step 2-2 based on the user-provided coarse-grained target features to reduce the workload of manual screening. The re-identification model will output n cropped images that it considers most likely to be the target, in descending order of confidence. Find the corresponding video frame image according to the naming convention used in Step 2-2.

[0106] Step 4-2: Shuffle the order of the video frames obtained in Step 4-1, and then present them to the subjects in an RSVP paradigm. The RSVP paradigm settings are the same as in Step 3-2. While the subjects are watching the video frames, EEG signals and eye movement information need to be recorded simultaneously. Preprocess the collected EEG signals according to the processing method in Step 3-3. Input the processed EEG signals into the EEG classification model obtained in Step 3-4. If the output of the EEG classification model is 1, then the video frame contains a target, and the eye movement data for the corresponding time period is processed.

[0107] The specific process of eye-tracking data processing described in step 4-2 is as follows:

[0108] Step 4-2-1: Human visual attention guides eye movements. The eye movement data acquired by the eye tracker can be analyzed in real time as fixation or saccades. Fixation indicates focusing attention on a target of interest, while saccades indicate changes or shifts in attention. Therefore, the saccade portion of the acquired eye movement data is discarded. The region around the fixation point is the region of interest, i.e., the area where the target appears. The Dispersion-Threshold Identification (I-DT) algorithm is used to extract the fixation point from the raw eye movement data, identifying the fixation point as a set of continuous points within a specific dispersion.

[0109] The process of extracting fixation points is as follows: First, the eye movement points are sorted chronologically, and then a sliding window is used to scan consecutive eye movement points. The initial size of the sliding window is determined by a duration threshold; in this embodiment, the initial window size is set to 35ms. The I-DT algorithm calculates the dispersion D of the eye movement points within the sliding window, using the following formula:

[0110] D=[max(x)-min(x)]+[max(y)-min(y)]

[0111] Where max(x) and min(x) represent the x-coordinates of the leftmost and rightmost eye-moving points in the plane, and max(y) and min(y) represent the y-coordinates of the topmost and bottommost eye-moving points in the plane.

[0112] If the dispersion is less than the threshold, the next eye-tracking point is added to the sliding window until the dispersion is greater than the threshold. At this point, the average coordinates of the eye-tracking points in the sliding window are calculated, and the result is the coordinates of a fixation point. When the dispersion is greater than the threshold, the window is moved to the right, and the separation is recalculated. This process is repeated until all eye-tracking points have been scanned.

[0113] Step 4-2-2: Based on the gaze points extracted in Step 4-2-1, calculate the Euclidean distance between each pair of gaze points, and use a density clustering algorithm to generate multiple regions of interest, as shown in the following formula:

[0114]

[0115] Among them, F i ,F j For two different fixation points, (x i ,y i ), (x j ,y j (F) represents the fixation point F. i F j The coordinates. If the distance d between two fixation points is less than the threshold d. ∈ The coordinates of the two fixation points are averaged and then merged to generate a new fixation point.

[0116] Step 4-2-3: A stimulus image may generate multiple regions of interest (ROIs). The region most likely to be the target needs to be selected. Since visual search is a process, the subject is more likely to focus on the target over time. Therefore, the last ROI generated is selected as the target region.

[0117] Step 4-2-4: Target localization based on human vision may have biases, meaning the region of interest may not accurately outline the target. Therefore, combining the Faster R-CNN target detection model described in Step 1-1, the bounding box provided by the model is compared with the target region determined by human vision. The bounding box that is closest to the target region determined by human vision is selected as the final target region and added to the candidate target set. The image is then cropped based on the coordinates of the bounding box, and the cropped image is saved locally, named as: candidate target_current number.jpg.

[0118] Step 4-3: Among the candidate targets obtained in Step 4-2, there may be false alarms, meaning that non-targets are identified as targets. Therefore, it is necessary to manually select a number of target images from the candidate target set that will ultimately be retrieved.

[0119] Step 4-4: Use the re-identification model described in Step 1-2 to extract features from each target image obtained in Step 4-3. Compare the extracted features with the feature library saved in Step 2-3 to calculate a similarity score. A higher score indicates greater similarity between the two feature vectors. The similarity between feature vectors is measured using cosine similarity, as shown in Equation 5.

[0120]

[0121] Here, A and B are two 512-dimensional feature vectors, and ||·|| represents the magnitude of the vector.

[0122] The similarity scores are sorted in descending order, and image features from the database with a similarity greater than 90% to any target image feature are selected. Using the dictionary described in step 2-3, the image names corresponding to all selected image features are found. Then, following the naming convention in step 2-2, the video name and current frame number are extracted from the image names. The video name and frame number are used to trace back to the corresponding video segment, thus achieving target localization and tracking.

Claims

1. A method for recognizing complex targets in massive video datasets based on human-machine collaboration, characterized in that: Includes the following steps: Step 1: Build and train the object detection model and the re-identification model; Step 2: Preprocessing of surveillance video; Extract video frame images from the retrieved surveillance video data; The target detection model is used to detect and analyze video frame images, and the targets in the video frame images are framed and cropped to obtain the cropped image; Use a re-identification model to extract features of the target in the cropped image; Step 3: Construct and calibrate the EEG classification model; the EEG classification model determines whether the user whose EEG signals were collected observed the target; Step 4: Human-machine collaborative video target retrieval Step 4-1: Use the re-identification model to pre-screen the cropping images identified by the target detection model based on the coarse-grained features of the target; The re-identification model selects based on confidence level. Crop images; extract the corresponding video frame images from each crop image; Step 4-2: Provide the subjects with the video frame images selected in Step 4-1; while the subjects are watching the video frame images, record the EEG signals and eye movement information simultaneously; input the obtained EEG signals into the EEG classification model after preprocessing; when the EEG classification model determines that the subject has observed a target, process the subject's eye movement data to extract the region of interest (ROI) of the subject's video frame images; extract the candidate target set from each ROI. Step 4-3: Select several target images from the candidate target set obtained in Step 4-2 for final retrieval; Step 4-4: Use the re-identification model to extract features from each target image obtained in Step 4-3, and perform similarity matching between the features extracted from each target image and the features of the video frame images in the retrieved surveillance video data; take the video frame images corresponding to features with similarity exceeding the threshold as images containing the target.

2. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 1, characterized in that: The object detection model described in step 1 uses a Faster R-CNN model that has been retrained on the CoCo dataset.

3. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 1, characterized in that: The re-identification model described in step 1 uses the ResNet50 model as the re-identification model; the average pooling layer in the ResNet50 model is replaced with an adaptive average pooling layer.

4. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 1, characterized in that: The feature vector of the re-identification model has a dimension of 512.

5. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 1, characterized in that: The specific process of step 3 is as follows: Step 3-1: Prepare stimulation images for several test blocks in advance. Each test block contains multiple trials, and each trial contains multiple images. Step 3-2: Present stimulus images to the subjects and collect EEG data; Step 3-3: Preprocess the acquired EEG data; Steps 3-4: Construct a deep learning-based EEG classification model to perform binary classification of EEG signals; The EEG classification model is trained by decoupling representation learning, and the training of the model is divided into two stages: the first stage trains the feature extractor and the second stage trains the classifier.

6. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 5, characterized in that: In step 3-1, the ratio of the number of stimulus images containing the target to the number of stimulus images not containing the target in each trial is 1:

10.

7. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 5, characterized in that: In step 3-2, there is a 5-second fixation time before each trial begins; after the trial begins, each stimulus image is presented for 500ms; after each trial ends, the subject rests for 10 seconds; the subject can rest for any length of time between two trials; when presenting stimulus images to the subject, there must be at least one stimulus image that does not contain the target between two adjacent stimulus images containing the target.

8. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 5, characterized in that: The preprocessing process in step 3-3 is as follows: use a 2-40Hz bandpass filter to remove voltage drift and high-frequency noise in the EEG signal, and downsample the EEG signal to 100Hz; segment the downsampled EEG signal, taking the presentation of each stimulus image as the time starting point, and extract the EEG signal with a duration of 1s as a sample.

9. A method for recognizing complex targets in massive video data based on human-machine collaboration as described in claim 5, characterized in that: The process of training the EEG classification model in steps 3-4 is as follows: Step 3-4-1: Construct triplet samples; In order to train the feature extractor using the triplet loss function, triplet samples need to be constructed. ; Samples and The samples belong to the same category. The sample belongs to other classes; Step 3-4-2: The feature extractor is trained to extract spatiotemporal information from EEG data; the feature extractor consists of a multilayer perceptron and a recurrent network layer; the multilayer perceptron is constructed from four fully connected layers, an ELU activation function, and residual connections; the recurrent network layer uses a long short-term memory network; Features output by the feature extractor for: ； in, , , , The weights of the fully connected layer, For ELU activation function, For each element in the triplet sample, i = , , ; These are the intermediate layer output and the final output of the multilayer perceptron, respectively. Features Flatten the features and use a projection layer. Mapping to a low-dimensional sample space yields the feature projection. as follows: ； in, The weights of the projection layer are used to calculate the projection of features from different samples. Similarity between samples; similarity between feature projections a and b of any two samples in a triplet. The expression is as follows: ； in, It is a preset parameter; It is a vector of all ones; Next, the feature extractor is trained using a triplet loss function. The definition is as follows: ； in, This represents the minimum distance between positive and negative samples; Step 3-4-3: Resample the training samples; Step 3-4-4: Train the classifier; the classifier consists of a fully connected layer, defined as follows: ； in, The predicted labels for the classifier; This is the weight matrix; For bias terms; The classifier is trained by minimizing the cross-entropy loss function, as shown in the following equation: ； in, Let be the predicted label of the classifier for the i-th sample. Let i be the true label of the i-th sample; For the resampled EEG data sample space The number of samples; the classifier is trained using the Adam optimizer.

10. The method for recognizing complex targets in massive videos based on human-machine collaboration according to claim 1, characterized in that: The specific process of eye-tracking data processing described in step 4-2 is as follows: Step 4-2-1: Extract fixation points from the raw eye-tracking data and identify them as a set of continuous points within a specific dispersion. The process of extracting fixation points is as follows: first, the eye movement points are sorted chronologically; then, a sliding window is used to scan consecutive eye movement points; the initial size of the sliding window is determined by a duration threshold, and the dispersion of eye movement points within the sliding window is calculated. The calculation formula is as follows: ； Where max(x) and min(x) represent the x-coordinates of the leftmost and rightmost eye-moving points in the plane, and max(y) and min(y) represent the y-coordinates of the topmost and bottommost eye-moving points in the plane. If the dispersion is less than the threshold, the next eye-tracking point is added to the sliding window until the dispersion is greater than the threshold. At this point, the average coordinates of the eye-tracking points in the sliding window are calculated as the coordinates of a fixation point. When the dispersion is greater than the threshold, the window is moved to the right and the separation is recalculated. This process is repeated until all eye-tracking points have been scanned. Step 4-2-2: Based on the gaze points extracted in Step 4-2-1, calculate the Euclidean distance between each pair of gaze points, and use a density clustering algorithm to generate multiple regions of interest, as shown in the following formula: ； in, For two different fixation points, , Each is a fixation point , The coordinates; if the distance between the two fixation points Less than the threshold The coordinates of the two fixation points are averaged and then merged to generate a new fixation point; Step 4-2-3: For each stimulus image, select the region of interest generated at the end as the target region determined by human vision. Step 4-2-4: Compare the bounding box region identified by the object detection model with the target region determined by human vision, select the bounding box region that is closest to the target region determined by human vision as the final target region, and add it to the candidate target set.

Images