Model training method, cross-modal retrieval method, device and equipment

Abstract

This disclosure provides a model training method, a cross-modal retrieval method, an apparatus, and a device. The model training method includes: in a cross-modal alignment model, filtering the initial EEG signal samples and initial image samples in the paired samples by frequency components to obtain target EEG signal samples and target image samples; performing global feature extraction, frequency domain feature decoupling, and feature alignment under the same frequency components on the target EEG signal samples and target image samples respectively to obtain the training loss value for the current training; and adjusting the parameters of the cross-modal alignment model based on the training loss value. Thus, the frequency domain information of EEG signals and images is fully utilized for training the cross-modal alignment model, improving the training effect and robustness of the cross-modal alignment model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, a cross-modal retrieval method, an apparatus, and a device. Background Technology

[0002] Cross-modal alignment technology of EEG signals and images has become a key research direction in many cutting-edge fields such as neurocognitive science and artificial intelligence, and has important theoretical significance and practical application value.

[0003] Currently, cross-modal alignment techniques between EEG signals and images mostly adopt a shared embedding space modeling approach based on contrastive learning. This involves mapping the temporal features of EEG signals and the spatial semantic features of images to a shared embedding space, and using a loss function to narrow the feature distance between EEG signals and images in the same paired sample, thereby achieving cross-modal feature alignment.

[0004] However, the cross-modal alignment model trained in the above manner has insufficient capabilities. Summary of the Invention

[0005] This disclosure provides a model training method, a cross-modal retrieval method, an apparatus, and a device to address the problem of insufficient model capabilities in cross-modal alignment models.

[0006] In a first aspect, embodiments of this disclosure provide a model training method, comprising: obtaining paired samples required for the current training from a training dataset, the paired samples including an initial EEG signal sample and an initial image sample corresponding to the initial EEG signal sample; inputting the paired samples into a cross-modal alignment model, in which the frequency components of the initial EEG signal sample are filtered to obtain a target EEG signal sample, and the frequency components of the initial image sample are filtered to obtain a target image sample; performing global feature extraction on the target EEG signal sample and the target image sample respectively to obtain EEG global features and image global features; performing frequency domain decoupling on the EEG global features and the image global features respectively to obtain multiple EEG local features and multiple image local features, different EEG local features corresponding to different frequency components, and different image local features corresponding to different frequency components; determining the training loss value for the current training by performing feature alignment on the multiple EEG local features and the multiple image local features under the same frequency component; and adjusting the parameters of the cross-modal alignment model according to the training loss value to obtain the cross-modal alignment model after the current training.

[0007] Secondly, embodiments of this disclosure provide a cross-modal retrieval method, comprising: acquiring an EEG query signal and multiple candidate images; performing frequency component screening and global feature extraction on the EEG query signal and multiple candidate images using a cross-modal alignment model to obtain EEG query features and multiple candidate image features, wherein the cross-modal alignment model is trained according to the model training method described in the first aspect; and determining a target image matching the EEG query signal among the multiple candidate images by performing similarity analysis on the EEG query features and multiple candidate image features.

[0008] Thirdly, embodiments of this disclosure provide a model training apparatus, comprising: an acquisition unit, configured to acquire paired samples required for the current training from a training dataset, the paired samples including an initial EEG signal sample and an initial image sample corresponding to the initial EEG signal sample; a frequency domain filtering unit, configured to input the paired samples into a cross-modal alignment model, and in the cross-modal alignment model, filter the frequency components of the initial EEG signal sample to obtain a target EEG signal sample, and filter the frequency components of the initial image sample to obtain a target image sample; and a feature extraction unit, configured to perform global feature extraction on the target EEG signal sample and the target image sample respectively to obtain a brain... The system comprises: global EEG features and global image features; a frequency domain decoupling unit, used to decouple the global EEG features and the global image features in the frequency domain to obtain multiple local EEG features and multiple local image features, where different local EEG features correspond to different frequency components, and different local image features correspond to different frequency components; a feature alignment unit, used to align the multiple local EEG features and multiple local image features under the same frequency component to obtain the training loss value for the current training iteration; and a parameter adjustment unit, used to adjust the parameters of the cross-modal alignment model according to the training loss value to obtain the cross-modal alignment model after the current training iteration.

[0009] Fourthly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the model training method as described in the first aspect above or the cross-modal retrieval method as described in the second aspect.

[0010] Fifthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the model training method described in the first aspect or the cross-modal retrieval method described in the second aspect.

[0011] Sixthly, embodiments of this disclosure provide a computer program product comprising computer execution instructions, which, when executed by a processor, implement the model training method described in the first aspect or the cross-modal retrieval method described in the second aspect.

[0012] The model training method, cross-modal retrieval method, apparatus, and device provided in this disclosure introduce frequency component screening at the input end of the cross-modal alignment model to suppress interference from noise frequency components and weakly related information that are unrelated to EEG-image alignment, thereby improving the robustness, robustness, and feature extraction quality of the cross-modal alignment model. Furthermore, the disclosure introduces frequency domain decoupling of global features and feature alignment under the same frequency component at the feature end of the cross-modal alignment model to improve the targeting and effectiveness of EEG-image semantic alignment. Therefore, this disclosure utilizes the temporal information of EEG signals, the spatial information of images, and the frequency domain information of both modalities in EEG-image cross-modal alignment, resulting in a cross-modal alignment model with stronger noise resistance, stronger training stability, and better generalization ability. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A schematic flowchart illustrating the model training method provided in this embodiment of the disclosure; Figure 2 A training example diagram of the cross-modal alignment model provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating the cross-modal retrieval method provided in this embodiment of the disclosure; Figure 4 Example diagrams for cross-modal retrieval provided in embodiments of this disclosure; Figure 5 This is a structural block diagram of the model training device provided in the embodiments of this disclosure; Figure 6 A structural block diagram of the cross-modal retrieval device provided in the embodiments of this disclosure; Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0016] Cross-modal alignment techniques between EEG signals and images often employ a shared embedding space modeling approach based on contrastive learning. However, this approach currently focuses on the temporal information of EEG signals and the spatial information of images, lacking explicit utilization of the frequency domain information of these two modalities.

[0017] Taking representative methods such as Uncertainty-aware Blur Prior (UBP), Visual-EEG Semantic Decoupling Framework (VE-SDN), and Brain-Inspired Prompt Tuning for EEG-to-Image Multimodal Contrastive Learning (NeuroCLIP) as examples, this paper illustrates that cross-modal alignment techniques for EEG signals and images lack explicit utilization of frequency domain information from these two modalities.

[0018] The Unified Backpropagation (UBP) method addresses the perceptual and stochastic gaps between EEG signals and visual stimuli by introducing a fuzzy prior to address uncertain perception. It dynamically blurs high-frequency details in the original image, reducing inconsistencies between these details and the EEG representation, thus improving cross-modal alignment. However, UBP essentially approximates the lack of high-frequency information in images and does not explicitly utilize the frequency domain information of both EEG signals and images. Furthermore, the fuzzy prior in UBP still represents a simplified model of the visual system, lacking a complete learnable mechanism. It fails to establish an explicit and unified frequency domain modeling and alignment mechanism for both EEG signals and images.

[0019] The VE-SDN method reprojects image features and EEG signal features into a joint semantic space, explicitly decomposing them into semantically relevant and domain-specific components. It improves semantic consistency between images and EEG signals through strategies such as maximizing or minimizing mutual information, cyclic reconstruction, and intra-class geometric consistency. However, the VE-SDN method primarily addresses the semantic consistency problem within the joint semantic space, focusing on semantically relevant feature extraction, mutual information constraints, and set consistency modeling, rather than the explicit utilization of frequency domain information in cross-modal alignment of EEG signals and images.

[0020] The NeuroCLIP method, building upon the CLIP framework, introduces two-stream visual embedding, dynamic filtering, token-level fusion based on cross-attention, cue learning, and an improved cross-modal alignment loss to enhance the image encoder's adaptability to EEG signal modalities and improve EEG signal-image alignment capabilities. However, the NeuroCLIP method does not treat the frequency components of EEG information and the frequency domain information of the image as independent modeling objects for cross-modal alignment of these two modalities.

[0021] Cross-modal alignment techniques for EEG signals and images suffer from the following shortcomings due to the lack of explicit utilization of frequency domain information: First, irrelevant oscillatory components in EEG signals and weakly correlated high-frequency details in images are directly introduced into the encoding and alignment process, easily introducing noise and task-weakly related information into cross-modal learning; Second, treating the feature information of EEG signals and images as a whole vector for unified constraint makes it difficult to distinguish frequency components with different characteristics, and difficult to accurately align the feature information of EEG signals and images, especially making it difficult to focus on aligning components that are truly conducive to the establishment of cross-modal shared semantics; Third, the lack of hierarchical modeling and differentiated constraint mechanisms for different frequency components makes it difficult to achieve the optimal balance between "shared semantic consistency" and "preservation of modality-specific differences".

[0022] The aforementioned shortcomings result in insufficient robustness, inadequate robustness, and limited generalization ability of cross-modal alignment models of EEG signals and images.

[0023] The model training method provided in this disclosure performs frequency component screening on the matching samples input to the cross-modal alignment model to suppress irrelevant oscillatory components in EEG signals and interference from weakly correlated high-frequency details in images, thereby improving the robustness, robustness, and feature extraction quality of the cross-modal alignment model. By performing frequency domain decoupling and feature alignment under the same frequency component on global EEG features and global image features, a training loss value is obtained, thereby performing targeted feature alignment and constraints on different frequency components, rather than treating the feature information of EEG signals and image features as a whole vector for unified constraint, thus improving the accuracy of the training loss value.

[0024] Therefore, it can be seen that in the EEG-image cross-modal alignment, the embodiments of this disclosure utilize the temporal information of the EEG signal, the spatial information of the image, and the frequency domain information of these two modalities, so that the cross-modal alignment model has stronger noise resistance, stronger training stability, and stronger generalization ability.

[0025] Figure 1 This is a schematic flowchart illustrating the model training method provided in an embodiment of this disclosure. Figure 1 As shown, the model training method includes: S101, Obtain the paired samples required for the current training from the training dataset. The paired samples include the initial EEG signal samples and the initial image samples corresponding to the initial EEG signal samples.

[0026] The training dataset includes multiple paired samples, each containing both an EEG signal sample and an EEG image sample. For ease of distinction, these are referred to as the initial EEG signal sample and the initial image sample in this embodiment. The initial EEG signal sample is a multi-channel time-series signal, so it can be represented as a two-dimensional data matrix, where one dimension represents the signal channel and the other represents the time point.

[0027] In the same paired sample, the original EEG signal sample and the original image sample constitute a supervised pairing relationship. The EEG signal response sample generated by using the EEG image sample as a visual stimulus can be preprocessed to obtain the original EEG signal sample. The preprocessing can include the following operations: signal truncation according to a preset time window, correction based on a preset baseline, and downsampling.

[0028] For example, for each EEG signal response sample, a local signal within a time window of 0-1000ms after the image stimulus is presented can be selected, and baseline correction and downsampling can be performed on the local signal to obtain multiple raw EEG signal samples. Each raw EEG signal sample has a unique visual stimulus corresponding to it. Each raw EEG signal sample can be paired with the original image sample that generates the visual stimulus to obtain multiple paired samples, so that the raw EEG signal samples and the original image samples form a supervised pairing relationship.

[0029] For example, the initial EEG signal sample is a 17-channel time-series signal, with each time-series signal corresponding to 250 time points. The initial EEG signal sample can be represented as follows: .

[0030] In this embodiment, when starting the current training iteration, a batch of paired samples is obtained from the training dataset and used for the current training iteration.

[0031] S102, input the paired samples into the cross-modal alignment model. In the cross-modal alignment model, the frequency components of the initial EEG signal samples are filtered to obtain the target EEG signal samples. The frequency components of the initial image samples are filtered to obtain the target image samples.

[0032] In this embodiment, the initial EEG signal samples and initial image samples from all paired samples are input into the cross-modal alignment model. Before feature extraction from the initial EEG signal samples and initial image samples, the initial EEG signal samples and initial image samples can be transformed to the frequency domain to obtain the frequency domain information of the initial EEG signal samples and the frequency domain information of the initial image samples. Frequency component screening is performed on the frequency domain information of the initial EEG signal samples and the frequency domain information of the initial image samples to obtain the screened frequency domain information of the initial EEG signal samples and the screened frequency domain information of the initial image samples. Inverse transformation is performed on the screened frequency domain information of the initial EEG signal samples and the screened frequency domain information of the initial image samples to obtain the target EEG signal samples and the target image samples.

[0033] In the process of frequency component screening of the frequency domain information of the initial EEG signal samples and the initial image samples, all or part of the task-irrelevant noise frequency components and the task-weakly related frequency components are removed in order to increase the proportion of task-related frequency components in the initial EEG signal samples and the initial image samples, and to suppress the interference of task-irrelevant noise frequency components and the task-weakly related frequency components on the training of cross-modal alignment models.

[0034] S103, global feature extraction is performed on the target EEG signal sample and the target image sample respectively to obtain global EEG features and global image features.

[0035] In this embodiment, the cross-modal alignment model includes an EEG feature extraction branch and an image feature extraction branch. The EEG feature extraction branch can be used to extract global features from the target EEG signal sample, yielding global EEG features; similarly, the image feature extraction branch can be used to extract global features from the target image sample, yielding global image features. This allows for parallel extraction of EEG signal features and image signal features without interference, improving the feature extraction efficiency of the cross-modal alignment model. Furthermore, since frequency component screening is performed on the target EEG signal sample and target image sample before performing global feature extraction on each, the feature quality of both the global EEG features and the global image features is improved.

[0036] S104, frequency domain decoupling is performed on the global features of EEG and the global features of the image to obtain multiple local features of EEG and multiple local features of the image. Different local features of EEG correspond to different frequency components, and different local features of the image correspond to different frequency components.

[0037] In this embodiment, cross-modal embedding, namely EEG global features and image global features, is no longer regarded as an indivisible single global vector, but rather as a one-dimensional discrete signal along the feature dimension, and is split in the frequency domain: the EEG global features and image global features can be transformed to the frequency domain to obtain EEG frequency domain features and image frequency domain features; in the frequency domain, the EEG frequency domain features and image frequency domain features are split according to multiple frequency components to obtain multiple EEG local features and multiple image local features, so that different EEG local features correspond to different frequency components, and different image local features correspond to different frequency components, thereby realizing the feature differentiation of different frequency components.

[0038] S105: By aligning multiple EEG local features and multiple image local features with the same frequency component, the training loss value for the current training iteration is obtained.

[0039] In this embodiment, different EEG local features correspond to different frequency components, and different image local features correspond to different frequency components. Under each frequency component, based on a feature alignment method designed for that frequency component, the EEG local feature and the image local feature corresponding to that frequency component can be aligned to obtain the training loss value for the current training. Feature alignment refers to cross-modal alignment of EEG local features and image local features in the same feature space, and the resulting training loss value is the alignment loss value of cross-modal alignment. Thus, by distinguishing the feature information of different frequency components, the feature information of EEG signal samples and image samples can be accurately aligned, achieving hierarchical modeling and differentiated constraints for different frequency components.

[0040] S106. Based on the training loss value, adjust the parameters of the cross-modal alignment model to obtain the cross-modal alignment model after the current training.

[0041] In this embodiment, based on the training loss value, the parameters of the cross-modal alignment model are adjusted using optimization algorithms, such as gradient descent and adaptive learning rate optimization algorithms, to obtain the cross-modal alignment model after the current training. If the cross-modal alignment model requires multiple training iterations, the steps provided in this embodiment can be followed to train the cross-modal alignment model multiple times until the training termination condition is met, resulting in a well-trained cross-modal alignment model.

[0042] This embodiment introduces frequency domain correlation processing at both the input and feature ends of the cross-modal alignment model. At the input end, frequency domain components are screened from the initial EEG signals and initial image samples to suppress the interference of irrelevant and weakly correlated frequency domain information on cross-modal alignment. At the feature end, global EEG features and global image features are decoupled to obtain multiple local EEG features and multiple local image features, enabling feature differentiation of different frequency components, rather than treating global EEG features and global image features as an inseparable whole. Feature alignment is performed on multiple local EEG features and multiple local image features under the same frequency component, improving the accuracy of feature alignment for EEG signal samples and image samples, and enabling hierarchical modeling and differentiated constraints for different frequency components. Thus, through the full utilization of frequency domain information, the cross-modal alignment model possesses stronger noise resistance, stronger training stability, and stronger generalization ability.

[0043] Below, we provide some examples for frequency domain component screening of initial EEG signal samples.

[0044] In some embodiments, the frequency components of an initial EEG signal sample are screened to obtain a target EEG signal sample. This includes: transforming the time-series signals of each signal channel in the initial EEG signal sample from the time domain to the frequency domain to obtain the frequency domain signals corresponding to each signal channel; based on a smooth frequency band selection mechanism, the frequency domain signals corresponding to each signal channel are screened to obtain the selected signals corresponding to each signal channel; and the selected signals corresponding to each signal channel are transformed from the frequency domain to the time domain to obtain the target EEG signal sample. Thus, by replacing direct hard truncation with a smooth frequency band selection mechanism, ringing effects and oscillation distortions caused by abrupt changes in frequency band boundaries during the frequency domain component screening process are avoided, resulting in a smoother waveform for the target EEG signal sample.

[0045] In this embodiment, the time-series signals of each signal channel in the initial EEG signal sample can be subjected to Discrete Fourier Transform to obtain the frequency domain signals corresponding to each signal channel; the frequency domain signals corresponding to each signal channel can be filtered by the frequency band selection function on the EEG side to obtain the selection signals corresponding to each signal channel. The frequency band selection function adopts a smooth frequency band selection mechanism; the selection signals corresponding to each signal channel can be subjected to Discrete Fourier Transform to obtain the target EEG signal sample.

[0046] In some embodiments, based on a smooth frequency band selection mechanism, the frequency domain signals corresponding to each signal channel in the initial EEG signal sample are filtered to obtain selection signals for each signal channel. This includes: constructing a frequency band selection function for the EEG signal based on a set upper cutoff frequency, lower cutoff frequency, transition bandwidth, and a cosine smooth transition function; and processing the frequency domain signals corresponding to each signal channel in the initial EEG signal sample using the frequency band selection function to obtain selection signals for each signal channel. This achieves smooth frequency band selection of the initial EEG signal, making the waveform of the target EEG signal sample smoother. The upper cutoff frequency, lower cutoff frequency, and transition bandwidth ensure more accurate signal filtering and guarantee the relevance of the target EEG signal sample to visual semantics.

[0047] Transition bandwidth refers to the bandwidth of the frequency band that needs to be reserved.

[0048] In this embodiment, at the input of the cross-modal alignment model, a specific frequency band needs to be preserved, such as the 3 Hz to 40 Hz band of the EEG signal sample, while attenuating components outside this band. If a hard truncation is performed using an ideal rectangular filter, Gibbs oscillations will occur in the time domain, causing spectral leakage and affecting the stability of subsequent feature extraction. To avoid this problem, a raised cosine function can be used as the cosine smoothing function, and a smoothing filter mask constructed based on the raised cosine function can be used as the frequency band selection function. This frequency band selection function does not make abrupt changes near the cutoff frequencies (upper and lower cutoff frequencies), but rather smoothly transitions from 0 to 1, or from 1 to 0, within a frequency range centered on the cutoff frequency and with a frequency width of Δ, according to the raised cosine curve. The frequency width Δ of this frequency range is the transition bandwidth.

[0049] The value of the transition bandwidth affects the frequency selectivity and signal fidelity of the band selection function: if the value is too small, the band selection function approaches an ideal rectangle, reintroducing Gibbs oscillations and spectral leakage; if the value is too large, it will attenuate the effective frequency components falling within the transition region. Therefore, a balance can be achieved between frequency selectivity and effective information preservation by reasonably determining the value of the transition bandwidth.

[0050] The value of the transition bandwidth can be determined experimentally.

[0051] For example, the frequency domain signal of a single signal channel is The length of this frequency domain signal is The discrete Fourier transform of this frequency domain signal can be expressed as follows: ; in, Indicates frequency index, The symbol represents the imaginary unit, and t represents the index of the time-domain sampling point. The value of t ranges from 0 to T-1. express The corresponding frequency domain signal. These are complex spectral coefficients, with their magnitude corresponding to amplitude information and their argument corresponding to phase information.

[0052] Apply the frequency band selection function from the EEG side to The results are then subjected to inverse discrete Fourier transform to obtain the frequency band controlled signal corresponding to a single signal channel: ; in, This represents the frequency band selection function. This represents the inverse discrete Fourier transform. This represents the frequency band controlled signal corresponding to a single signal channel. Finally, the frequency band controlled signals corresponding to multiple signal channels form the target EEG signal sample. Through the above processing, what enters the EEG feature extraction branch is no longer the undifferentiated raw EEG signal sample, but the target EEG signal sample after frequency domain screening and reconstruction, making the subsequent feature processing more focused on the EEG oscillation structure related to visual semantics.

[0053] The above example of constructing the band selection function is as follows: First, we introduce the cosine smooth transition function: ; in: ; Next, based on the above cosine smooth transition function, the frequency band selection function is constructed: ; in, and These represent the lower cutoff frequency and the upper cutoff frequency, respectively. Indicates transition bandwidth. The target cutoff frequency is a variable that can be used in actual calculations. Substitute into each place and For example, it can be and Set to 3Hz and 40Hz, Set to 2Hz.

[0054] Below, some examples are provided for frequency domain component screening of initial image signal samples.

[0055] In some embodiments, filtering the frequency components of an initial image sample to obtain a target image sample includes: performing a two-dimensional frequency domain transformation and centering rearrangement on the initial image sample to obtain an initial spectrum corresponding to the initial image sample; processing the initial spectrum through a radial frequency gating mask to obtain a target spectrum; performing an inverse centering operation on the target spectrum, and transforming the target spectrum after the inverse centering operation from the frequency domain to the spatial domain to obtain the target image sample. This improves the accuracy of frequency component filtering of the initial image sample and ensures that the target image sample retains information related to visual semantics.

[0056] Among them, the radial frequency gated mask is constructed with the spectral center of the initial spectrum as the origin, and the two-dimensional frequency selection mask is constructed based on the radial distance from each frequency point in the initial spectrum to the spectral center.

[0057] In this embodiment, the image is a two-dimensional spatial signal. A two-dimensional Fourier transform is performed on the initial image sample to convert it from the spatial domain to the frequency domain. The image spectrum output by the two-dimensional Fourier transform is then centered and rearranged to obtain an initial spectrum. This initial spectrum is a circularly distributed spectrum, with low-frequency components concentrated at the center and high-frequency components distributed in the outer region. A radial frequency gating mask is used to filter frequency points in the initial spectrum whose radial distance from the center does not meet a set threshold, thus obtaining the target spectrum. An inverse centering operation (restoring the frequency arrangement after the Fourier transform to the centered spectrum) and a two-dimensional Fourier transform are then performed on the target spectrum to transform it from the frequency domain to the spatial domain, obtaining the target image sample. This achieves accurate selection of the frequency components of the initial image sample.

[0058] For example, a two-dimensional Fourier transform of the initial image samples can be represented as: ; in, Represents the initial image sample. Represents a two-dimensional Fourier transform. The spectrum represents the image obtained by the two-dimensional Fourier transform, where i and j represent the two-dimensional coordinates.

[0059] For example, the centering rearrangement of the image spectrum can be represented as: ; in, This represents a centralized rearrangement, i.e., a spectrum centralization operation. This represents the initial spectrum corresponding to the initial image sample.

[0060] For example, processing the initial spectrum using a radial frequency-gated mask can be represented as follows: ; in, For radial frequency gated masks, This indicates a reverse centralization operation. This represents the two-dimensional inverse Fourier transform. This represents the real part of the reconstruction result (i.e., the output of the two-dimensional inverse Fourier transform).

[0061] In some embodiments, the process of constructing a radial frequency gate mask may include: normalizing the radial distances from multiple frequency points in the initial spectrum to the center of the spectrum to obtain multiple normalized radii; dividing the initial spectrum into frequency regions according to the multiple normalized radii to obtain multiple frequency regions; and constructing a radial frequency gate mask according to the positional distribution of the multiple frequency regions on the initial spectrum.

[0062] In this embodiment, the radial distances from multiple frequency points to the spectrum center can be normalized based on the maximum value among these distances, resulting in multiple normalized radii. Based on these normalized radii, the initial spectrum can be divided into a low-frequency center region, a mid-frequency ring band, and a high-frequency peripheral region. A radial frequency gating mask can be constructed based on the positional distribution of these regions. For example, if the low-frequency center region and the mid-frequency ring band are distributed within a normalized radius of less than 0.2, a radial frequency gating mask can be constructed to remove frequency components with a normalized radius greater than or equal to 0.2 and retain those with a normalized radius less than 0.2. Then, the radial frequency gating mask is used to process the initial spectrum to obtain the target spectrum.

[0063] For example, the radial distances from multiple frequency points to the center of the spectrum are normalized and represented as follows: ; in, This represents the maximum radial distance among multiple frequency points to the center of the spectrum. , (i,j) represents the two-dimensional coordinates of the center of the initial spectrum, and (i,j) represents the two-dimensional coordinates of a frequency point on the initial spectrum. This represents the normalized radius.

[0064] It should be noted that the frequency component screening of the initial EEG signal samples and the frequency component screening of the initial image samples can be performed in parallel to improve the efficiency of the overall training process and the data processing efficiency of the model.

[0065] Below, we provide some examples of extracting global features from electroencephalograms (EEGs) and images.

[0066] In some embodiments, the EEG feature extraction branch in the cross-modal alignment model may include a linear perturbation module and an EEG signal encoder; global feature extraction of the target EEG signal sample to obtain global EEG features may include: linearly enhancing the target EEG signal sample through the linear perturbation module, and then performing global feature extraction on the linearly enhanced target EEG signal sample through the EEG signal encoder to obtain global EEG signal features. Thus, the linear perturbation module makes the extracted global EEG signal features more robust and diverse, which helps prevent model overfitting.

[0067] Among them, the parameters of the linear perturbation module and the parameters of the EEG signal encoder are trainable parameters.

[0068] For example, linear enhancement of a target EEG signal sample using a linear perturbation module is represented as follows: ; in, For trainable parameters, This represents element-wise multiplication. Indicates the target EEG signal sample, This represents the target EEG signal sample after linear enhancement.

[0069] The EEG signal encoder may include a single fully connected layer and a normalized layer.

[0070] In some embodiments, the image feature extraction branch in the cross-modal alignment model may include a parameter-frozen image encoder and a feature projection layer. The input word sequence of the image encoder is interpolated with shared-level cue words, which are trainable parameters. Global feature extraction of the target image sample to obtain global image features may include: generating an input word sequence from the target image sample and the shared-level cue words; and extracting features from the input word sequence through the image encoder and the feature projection layer to generate global image features. Thus, while keeping the parameters of the image encoder unchanged, cross-modal adaptation of EEG signals and the visual representation of images is achieved through efficient learning of shared-level cue words, improving the ability to align features between images and EEG signals while maintaining the stability of the visual semantic space.

[0071] The parameters of the feature projection layer are trainable.

[0072] It should be noted that global feature extraction of target EEG signal samples and global feature extraction of target image samples can be performed in parallel to improve the efficiency of the overall training process and the feature extraction efficiency of the model.

[0073] Below, we provide some examples of feature alignment between local EEG features and local image features.

[0074] In some embodiments, multiple EEG local features include low-frequency EEG features and high-frequency EEG features, and multiple image local features include low-frequency image features and high-frequency image features. Frequency domain decoupling is performed on the global EEG features and global image features to obtain EEG local features and image local features under multiple frequency components, including: transforming the global EEG features and global image features to the frequency domain to obtain EEG frequency domain features and image frequency domain features; processing the EEG frequency domain features using low-frequency and high-frequency masks to obtain a first EEG frequency domain sub-feature and a second EEG frequency domain sub-feature; processing the image frequency domain features using low-frequency and high-frequency masks to obtain a first image frequency domain sub-feature and a second image frequency domain sub-feature; and performing inverse transforms on the first EEG frequency domain sub-feature, the second EEG frequency domain sub-feature, the first image frequency domain sub-feature, and the second image frequency domain sub-feature to obtain low-frequency EEG features, high-frequency EEG features, low-frequency image features, and high-frequency image features. Therefore, by treating global EEG features and global image features as one-dimensional discrete signals along the feature dimension, and splitting them into low-frequency and high-frequency components in the frequency domain, rather than treating them as indivisible single overall features, this provides a data foundation for subsequent feature alignment by distinguishing frequency components. Moreover, it enables the cross-modal alignment model to explicitly distinguish stable low-frequency components dominated by shared semantics from high-frequency components containing detailed perturbations and modal differences. This enhances the cross-modal alignment model's ability to finely model the internal structure of cross-modal features, and improves the pertinence and effectiveness of semantic alignment between EEG signals and images.

[0075] Low-frequency masks can be used to preserve low-frequency components, while high-frequency masks can be used to preserve high-frequency components.

[0076] In this embodiment, EEG frequency domain features and image frequency domain features are obtained by performing Discrete Fourier Transform on the global features of EEG and the global features of the image, respectively. Then, the frequency domain features of EEG are decoupled in the frequency domain by using low-frequency masks and high-frequency masks to obtain first EEG frequency domain sub-features and second EEG frequency domain sub-features. The first EEG frequency domain sub-features and second EEG frequency domain sub-features are respectively the local features of the low-frequency component and the local features of the high-frequency component of EEG in the frequency domain. The first EEG frequency domain sub-features and second EEG frequency domain sub-features are respectively subjected to Inverse Discrete Fourier Transform on the first EEG frequency domain sub-features and second EEG frequency domain sub-features to obtain low-frequency features and high-frequency features of EEG. The image frequency domain features are decoupled in the frequency domain by using low-frequency masks and high-frequency masks to obtain first image frequency domain sub-features and second image frequency domain sub-features. The first image frequency domain sub-features and second image frequency domain sub-features are respectively the local features of the low-frequency component and the local features of the high-frequency component of the image in the frequency domain. The first image frequency domain sub-features and second image frequency domain sub-features are respectively subjected to Inverse Discrete Fourier Transform on the first image frequency domain sub-features and second image frequency domain sub-features to obtain low-frequency features and high-frequency features of the image.

[0077] For example, performing Discrete Fourier Transform on the global features of EEG and the global features of the image respectively yields the frequency domain features of EEG and the frequency domain features of the image, represented as follows: ; in, Indicates global brainwave characteristics, Represents global features of the image. Indicates the frequency domain characteristics of electroencephalography (EEG). Represents the frequency domain features of an image. This represents the Discrete Fourier Transform.

[0078] For example, frequency domain decoupling and inverse discrete Fourier transform are performed on the frequency domain features of EEG to obtain low-frequency and high-frequency EEG features, as follows: ; in, Indicates a low-frequency mask. Indicates a high-frequency mask. Indicates low-frequency characteristics of brainwaves. This indicates low-frequency characteristics of brainwaves.

[0079] For example, by performing frequency domain decoupling and inverse discrete Fourier transform on the image frequency domain features, we obtain the low-frequency features and high-frequency features of the image, as shown below: ; in, Indicates low-frequency features of the image. This represents the low-frequency features of the image.

[0080] In some embodiments, a low-frequency retention length is determined based on a preset low-frequency retention ratio and shared feature dimension, and low-frequency masks and high-frequency masks are constructed based on the low-frequency retention length. This improves the accuracy of the low-frequency and high-frequency masks.

[0081] Before frequency domain decoupling, the global features of EEG and the global features of the image have been mapped to the same feature dimension. Therefore, the frequency domain features of EEG and the frequency domain features of the image have the same feature dimension, and the shared feature dimension is equal to the feature dimension of the frequency domain features of EEG or the feature dimension of the frequency domain features of the image.

[0082] For example, the low-frequency retention ratio is Shared feature dimension is The low-frequency retention length is: ; in, Indicates the low-frequency retention length.

[0083] For example, a low-frequency mask is represented as: ; High-frequency masks are represented as: ; in, This indicates that all characteristic portions with frequencies less than K are extracted.

[0084] Below, based on multiple EEG local features, including low-frequency and high-frequency EEG features, and multiple image local features, including low-frequency and high-frequency image features, some embodiments are provided for feature alignment of multiple EEG local features and multiple image local features.

[0085] In some embodiments, feature alignment of multiple EEG local features and multiple image local features under the same frequency component is performed to obtain the training loss value for the current training iteration. This includes: obtaining the low-frequency loss value for the current training iteration by performing feature alignment of low-frequency EEG features and low-frequency image features under the low-frequency component; obtaining the high-frequency loss value for the current training iteration by performing feature alignment of high-frequency EEG features and high-frequency image features under the high-frequency component; and determining the training loss value by weighting the low-frequency loss value and the high-frequency loss value according to preset low-frequency weight coefficients and preset high-frequency weight coefficients; wherein the low-frequency weight coefficient is greater than the high-frequency weight coefficient.

[0086] In this embodiment, considering that low-frequency features are the main carriers of cross-modal shared semantics, strong consistency constraints should be applied; high-frequency features are more likely to contain details and perturbations, so a relatively flexible alignment method should be adopted. For multiple EEG local features and multiple image local features, feature alignment to distinguish high-frequency components and low-frequency components is achieved. That is, EEG low-frequency features are aligned with image low-frequency features to obtain low-frequency loss, and EEG high-frequency features are aligned with image high-frequency features to obtain high-frequency loss values.

[0087] This improves the accuracy of feature alignment between multiple EEG local features and multiple image local features, facilitating the application of contrastive learning constraints of varying intensities to low-frequency and high-frequency features. It applies strong consistency constraints to low-frequency features and relatively flexible weak consistency constraints to high-frequency features, enabling differentiated modeling and optimization of shared semantic information in low frequencies and detailed differences in high frequencies. This avoids the over-constraint problem caused by rigidly aligning all frequency components with the same intensity, and helps to retain certain modality-specific information while enhancing cross-modal shared semantic consistency. Furthermore, in the process of obtaining the training loss value by weighting the low-frequency and high-frequency loss values, the low-frequency weight coefficient is greater than the high-frequency weight coefficient, reflecting the differentiated constraints on low-frequency and high-frequency components.

[0088] Furthermore, it enhances the semantic expressiveness and generalization ability of the cross-modal alignment model, thereby improving the cross-modal retrieval accuracy when the cross-modal alignment model is applied to cross-modal retrieval.

[0089] In some embodiments, feature alignment is performed between EEG low-frequency features and image low-frequency features to obtain the low-frequency loss value for the current training iteration. This includes: analyzing the similarity between EEG low-frequency features and image low-frequency features according to a first similarity function to determine a first similarity value; and analyzing the bidirectional contrast loss between EEG low-frequency features and image low-frequency features according to the first similarity value and a pre-constructed low-frequency bidirectional contrast loss function to obtain the low-frequency loss value. Thus, through similarity analysis and the low-frequency bidirectional contrast loss function, feature alignment between EEG low-frequency features and image low-frequency features is achieved, i.e., contrastive learning constraints are implemented on EEG low-frequency features and image low-frequency features, improving the accuracy of the low-frequency loss value.

[0090] The first similarity function is a pre-constructed function.

[0091] In some embodiments, before analyzing the similarity between low-frequency EEG features and low-frequency image features to determine the first similarity value, the low-frequency EEG features and low-frequency image features can be normalized separately to improve the accuracy of the similarity analysis.

[0092] For example, the normalization of low-frequency features in EEG or images can be represented as: ; in, This indicates low-frequency features in EEG or low-frequency features in images. Represents the normalized result , express The L2 norm.

[0093] In some embodiments, the first similarity function includes a first coefficient; analyzing the similarity between low-frequency features of EEG and low-frequency features of image according to the first similarity function to determine a first similarity value may include: multiplying the reciprocal of the low-frequency features of EEG with the low-frequency features of image by matrix multiplication to obtain a first product matrix, and dividing the first product matrix by the first coefficient to obtain the first similarity value.

[0094] The first similarity value is the similarity matrix.

[0095] For example, the formula for calculating the first similarity value is expressed as: ; There are multiple paired samples. This indicates that it is based on the i-th low-frequency EEG feature (obtained from the original EEG signal sample in the i-th paired sample); This indicates that the low-frequency features of the j-th image are used (obtained from the original image samples in the j-th paired samples). Indicates the first coefficient. This represents the similarity value between the i-th low-frequency EEG feature and the j-th low-frequency image feature, and is also the first similarity value.

[0096] In some embodiments, the low-frequency bidirectional contrast loss function includes an EEG-image low-frequency contrast loss function and an image-EEG low-frequency contrast loss function. The low-frequency loss value includes a first contrast loss value from EEG low-frequency features to image low-frequency features and a second contrast loss value from image low-frequency features to EEG low-frequency features. Based on a first similarity value and the pre-constructed low-frequency bidirectional contrast loss function, the bidirectional contrast loss between EEG low-frequency features and image low-frequency features is analyzed to obtain the low-frequency loss value. This includes: determining a first contrast loss value based on the first similarity value and the EEG-image low-frequency contrast loss function; determining a second contrast loss value based on the first similarity value and the image-EEG low-frequency contrast loss function; and combining the first and second contrast loss values ​​to obtain the low-frequency loss value. Thus, the accuracy of the low-frequency loss value is improved through the bidirectional contrast constraint between image low-frequency features and EEG low-frequency features.

[0097] For example, the low-frequency contrast loss function of EEG-image is expressed as: ; The image-EEG low-frequency contrast loss function is expressed as: ; Where B represents the number of paired samples used in the current training iteration. This represents the similarity value between the i-th low-frequency EEG feature and the i-th low-frequency EEG feature (obtained based on the original EEG signal sample in the i-th paired sample). This represents the first comparison loss value. This represents the second comparative loss value.

[0098] For example, the low-frequency loss value is: ; in, This represents the low-frequency loss value.

[0099] In some embodiments, feature alignment is performed between high-frequency EEG features and high-frequency image features to obtain the high-frequency loss value for the current training iteration. This includes: analyzing the similarity between the high-frequency EEG features and the high-frequency image features according to a first similarity function to determine a second similarity value; and analyzing the bidirectional contrast loss between the high-frequency EEG features and the high-frequency image features according to the first similarity value and a pre-constructed high-frequency bidirectional contrast loss function to obtain the high-frequency loss value. Thus, through similarity analysis and the high-frequency bidirectional contrast loss function, feature alignment between the high-frequency EEG features and the high-frequency image features is achieved, i.e., the contrastive learning constraint between the high-frequency EEG features and the high-frequency image features is implemented, improving the accuracy of the high-frequency loss value.

[0100] The second similarity function is a pre-constructed function.

[0101] In some embodiments, before analyzing the similarity between high-frequency EEG features and high-frequency image features to determine the second similarity value, the high-frequency EEG features and high-frequency image features can be normalized separately to improve the accuracy of the similarity analysis.

[0102] The normalization processing of high-frequency EEG features and high-frequency image features can be referred to the normalization processing of low-frequency EEG features and low-frequency image features mentioned above, and will not be repeated here.

[0103] In some embodiments, the second similarity function includes a second coefficient; analyzing the similarity between high-frequency features of EEG and high-frequency features of image according to the second similarity function to determine the second similarity value may include: multiplying the reciprocal of the high-frequency features of EEG with the high-frequency features of image by matrix multiplication to obtain a second product matrix, and dividing the second product matrix by the second coefficient to obtain the second similarity value.

[0104] The second similarity value is the similarity matrix.

[0105] For example, the formula for calculating the second similarity value is expressed as follows: ; There are multiple paired samples. This indicates that it is based on the i-th high-frequency EEG feature (obtained from the original EEG signal sample in the i-th paired sample); This indicates that the high-frequency features of the j-th image are used (obtained from the original image samples in the j-th paired samples). Indicates the second coefficient. This represents the similarity value between the i-th high-frequency EEG feature and the j-th high-frequency image feature, and is also the second similarity value.

[0106] In some embodiments, the high-frequency bidirectional contrast loss function includes an EEG-image high-frequency contrast loss function and an image-EEG high-frequency contrast loss function. The high-frequency loss value includes a third contrast loss value from EEG high-frequency features to image high-frequency features and a fourth contrast loss value from image high-frequency features to EEG high-frequency features. Based on a second similarity value and the pre-constructed high-frequency bidirectional contrast loss function, the bidirectional contrast loss between EEG high-frequency features and image high-frequency features is analyzed to obtain the high-frequency loss value. This includes: determining a third contrast loss value based on the second similarity value and the EEG-image high-frequency contrast loss function; determining a fourth contrast loss value based on the second similarity value and the image-EEG high-frequency contrast loss function; and combining the third and fourth contrast loss values ​​to obtain the high-frequency loss value. Thus, the accuracy of the high-frequency loss value is improved through the bidirectional contrast constraint between image high-frequency features and EEG high-frequency features.

[0107] For example, the high-frequency contrast loss function of EEG-image is expressed as: ; The image-EEG high-frequency contrast loss function is expressed as: ; in, This represents the similarity value between the i-th high-frequency EEG feature and the i-th high-frequency EEG feature (obtained based on the original EEG signal sample in the i-th paired sample). This represents the third comparison loss value. This represents the fourth comparison loss value.

[0108] For example, the high-frequency loss value is: ; in, This represents the high-frequency loss value.

[0109] For example, the training loss value for the current training iteration can be expressed as: ; in, This represents the training loss value for the current training iteration. This represents the preset low-frequency weighting coefficient. This represents the preset high-frequency weighting coefficient. Greater than Therefore, through and The influence of low-frequency and high-frequency loss values ​​on the training loss value is controlled separately, by... Greater than This allows for higher constraint strength in the low-frequency components.

[0110] In some embodiments, the first coefficient in the first similarity function is less than the second coefficient in the second similarity function. The first coefficient and the second coefficient are temperature coefficients. Specifically, refer to the aforementioned example. By making the first coefficient less than the second coefficient, the sharpness of the contrastive learning constraint of the high-frequency component is reduced, and the contrastive learning constraint of the low-frequency component and the high-frequency component is differentiated.

[0111] Example, Figure 2 This is a training example diagram of the cross-modal alignment model provided in an embodiment of this disclosure. (See diagram below.) Figure 2 As shown, the training process of the cross-modal alignment model includes: First, EEG signal samples and image samples are input into the cross-modal alignment model. The frequency domain constraints at the input end of the cross-modal alignment model include input frequency domain constraints on the EEG side and input frequency domain constraints on the image side. Frequency components of EEG signal samples are filtered through input frequency domain constraints on the EEG side, and frequency components of image samples are filtered through input frequency domain constraints on the image side.

[0112] Next, the EEG signal samples that have been filtered by frequency components enter the EEG feature extraction branch. Through the linear perturbation module and the EEG signal encoder, global EEG features are extracted. The image samples that have been filtered by frequency components and the shared-level cue words enter the image feature extraction branch. Through the image encoder and the feature projection layer, global image features are extracted.

[0113] Next, frequency domain decoupling is performed on the global EEG features to obtain high-frequency and low-frequency EEG features; frequency domain decoupling is also performed on the global image features to obtain high-frequency and low-frequency image features; the high-frequency EEG features are compared with the high-frequency image features to obtain the high-frequency loss value; the low-frequency EEG features are compared with the low-frequency image features to obtain the low-frequency loss value; the high-frequency and low-frequency loss values ​​are weighted to obtain the training loss value for the current training iteration.

[0114] Finally, the trainable parameters of the cross-modal alignment model are adjusted based on the training loss value. Among them, the parameters of the linear perturbation module, the parameters of the EEG signal encoder, the parameters of the feature projection layer, and the shared-level cue lexical units are trainable parameters.

[0115] Figure 3 This is a flowchart illustrating the cross-modal retrieval method provided in this embodiment of the disclosure. Figure 3 As shown, cross-modal detection methods include: S301, acquire EEG query signals and multiple candidate images.

[0116] In this embodiment, it is necessary to retrieve images that match the EEG query signal from multiple candidate images. The EEG query signal and multiple candidate images can be obtained through one or more of the following methods: from a database, from user input information, or through other means.

[0117] For example, obtaining the EEG query signal to be retrieved. and candidate image set .in, Indicates the first A brainwave query signal, This represents different candidate images.

[0118] S302 uses a cross-modal alignment model to perform frequency component screening and global feature extraction on EEG query signals and multiple candidate images, thereby obtaining EEG query features and features of multiple candidate images.

[0119] The cross-modal alignment model is trained according to the model training method provided in any of the foregoing embodiments. The model training method provided in any of the foregoing embodiments improves the robustness, robustness, and generalization ability of the cross-modal alignment model, thereby improving the accuracy of its cross-modal retrieval.

[0120] In this embodiment, the EEG query signal and multiple candidate images can be input into the cross-modal alignment model. In the cross-modal alignment model: frequency component filtering is performed on the EEG query signal and multiple candidate images to obtain frequency component-filtered EEG query signal and multiple frequency component-filtered candidate images; global feature extraction is performed on the frequency component-filtered EEG query signal through the EEG feature extraction branch to obtain EEG query features; global feature extraction is performed on the multiple frequency component-filtered candidate images through the image feature extraction branch to obtain multiple candidate image features.

[0121] The frequency component screening and feature extraction processes of the EEG query signal can be the same as those of the initial EEG signal sample screening and feature extraction processes described in the foregoing embodiments; the frequency component screening and feature extraction processes of the candidate image can be the same as those of the initial image sample screening and feature extraction processes described in the foregoing embodiments; these will not be elaborated upon here.

[0122] For example, in a cross-modal alignment model, EEG query signals After frequency domain transformation, frequency filtering, and inverse transformation reconstruction, the EEG query signal with frequency components filtered is obtained. At the same time, candidate images After undergoing two-dimensional frequency domain transformation, radial frequency filtering, and inverse transform reconstruction, candidate images with frequency components were obtained. , , ...

[0123] S303, by performing similarity analysis on EEG query features and multiple candidate image features, determines the target image matching the EEG query signal among multiple candidate images.

[0124] In this embodiment, by performing similarity analysis on EEG query features and multiple candidate image features, similarity scores between the EEG query features and multiple candidate image features can be obtained. Based on the similarity scores between the EEG query features and multiple candidate image features, the target image matching the EEG query signal is determined from the multiple candidate images.

[0125] For example, the candidate image with the highest similarity score can be determined as the target image; or, multiple candidate images can be sorted according to the similarity scores between the EEG query features and the features of multiple candidate images, and the candidate images ranked in the top N from largest to smallest similarity can be determined as the target image; or, candidate images with similarity scores greater than a set score threshold can be determined as the target image.

[0126] For example, the similarity calculation between EEG query features and multiple candidate image features is expressed as follows: ; in, This represents the similarity calculation function. Indicates EEG query signal EEG query characteristics, , , Representing candidate images Candidate image features, , , They represent and Similarity score and Similarity score and The similarity score.

[0127] In this embodiment of the disclosure, the training effect of the cross-modal alignment model is improved, thereby enhancing the accuracy of retrieving matching images for EEG signals.

[0128] It should be noted that in the application of cross-modal alignment models, frequency domain decoupling and feature alignment at the feature ends can be omitted, and cross-modal matching can be performed using the extracted global features.

[0129] It should be noted that the cross-modal alignment model can also be used in other applications, not limited to cross-modal retrieval.

[0130] As an example, Figure 4 Example diagrams of cross-modal retrieval provided in embodiments of this disclosure, such as... Figure 4 As shown, the cross-modal retrieval process based on the cross-modal alignment model is as follows: In the input frequency domain constraint of the cross-modal alignment model, the frequency components of the EEG query signal are filtered through the input frequency domain constraint on the EEG side, and multiple candidate images are filtered for frequency components through the input frequency domain constraint on the image side. The filtered EEG query signal enters the EEG feature extraction branch to obtain the EEG query features. The selected candidate images are then processed by the image feature extraction branch to obtain the features of the candidate images. , , Calculate EEG query features Each with multiple candidate image features , , The similarity scores are obtained, for example, 0.1, 0.2, and 0.7 respectively. The candidate image corresponding to the similarity score of 0.7 is determined as the target image.

[0131] Corresponding to the model training method in the above embodiments, Figure 5 This is a structural block diagram of a model training apparatus provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 5 The model training device 500 includes: The acquisition unit 501 is used to acquire the paired samples required for the current training from the training dataset. The paired samples include the initial EEG signal samples and the initial image samples corresponding to the initial EEG signal samples.

[0132] The frequency domain filtering unit 502 is used to input paired samples into the cross-modal alignment model. In the cross-modal alignment model, the frequency components of the initial EEG signal samples are filtered to obtain the target EEG signal samples, and the frequency components of the initial image samples are filtered to obtain the target image samples.

[0133] The feature extraction unit 503 is used to extract global features from the target EEG signal sample and the target image sample respectively, so as to obtain global EEG features and global image features.

[0134] The frequency domain decoupling unit 504 is used to decouple the global features of EEG and the global features of the image in the frequency domain, respectively, to obtain multiple local features of EEG and multiple local features of the image. Different local features of EEG correspond to different frequency components, and different local features of the image correspond to different frequency components.

[0135] The feature alignment unit 505 is used to align multiple EEG local features and multiple image local features under the same frequency component to obtain the training loss value of the current training.

[0136] The parameter adjustment unit 506 is used to adjust the parameters of the cross-modal alignment model according to the training loss value, so as to obtain the cross-modal alignment model after the current training.

[0137] In some embodiments, the initial EEG signal sample includes time-series signals on multiple signal channels. The frequency domain filtering unit 502 is specifically used to: transform the time-series signals on each signal channel in the initial EEG signal sample from the time domain to the frequency domain to obtain the frequency domain signals corresponding to each signal channel; filter the frequency domain signals corresponding to each signal channel based on a smooth frequency band selection mechanism to obtain the selection signals corresponding to each signal channel; and transform the selection signals corresponding to each signal channel from the frequency domain to the time domain to obtain the target EEG signal sample.

[0138] In some embodiments, the frequency domain filtering unit 502 is specifically used to: construct a frequency band selection function for the EEG signal side based on the set upper cutoff frequency, lower cutoff frequency, transition bandwidth and cosine smooth transition function; and process the frequency domain signals corresponding to each signal channel through the frequency band selection function to obtain the selection signal corresponding to each signal channel.

[0139] In some embodiments, the frequency domain filtering unit 502 is specifically used to: perform two-dimensional frequency domain transformation and centering rearrangement on the initial image sample to obtain the initial spectrum corresponding to the initial image sample; process the initial spectrum through a radial frequency gate mask to obtain the target spectrum; perform an inverse centering operation on the target spectrum, and transform the target spectrum after the inverse centering operation from the frequency domain to the spatial domain to obtain the target image sample.

[0140] In some embodiments, the multiple EEG local features include EEG low-frequency features and EEG high-frequency features, and the multiple image local features include image low-frequency features and image high-frequency features; the frequency domain decoupling unit 504 is specifically used to: transform the EEG global features and image global features to the frequency domain, respectively, to obtain EEG frequency domain features and image frequency domain features; process the EEG frequency domain features through low-frequency masks and high-frequency masks to obtain a first EEG frequency domain sub-feature and a second EEG frequency domain sub-feature; process the image frequency domain features through low-frequency masks and high-frequency masks to obtain a first image frequency domain sub-feature and a second image frequency domain sub-feature; and perform inverse transformations on the first EEG frequency domain sub-feature, the second EEG frequency domain sub-feature, the first image frequency domain sub-feature, and the second image frequency domain sub-feature, respectively, to obtain EEG low-frequency features, EEG high-frequency features, image low-frequency features, and image high-frequency features.

[0141] In some embodiments, the feature alignment unit 505 is specifically used to: obtain the low-frequency loss value of the current training by performing feature alignment of the low-frequency features of EEG and the low-frequency features of the image under the low-frequency component; obtain the high-frequency loss value of the current training by performing feature alignment of the high-frequency features of EEG and the high-frequency features of the image under the high-frequency component; and determine the training loss value by weighting the low-frequency loss value and the high-frequency loss value according to the preset low-frequency weight coefficient and the preset high-frequency weight coefficient; wherein the low-frequency weight coefficient is greater than the high-frequency weight coefficient.

[0142] In some embodiments, the feature alignment unit 505 is specifically configured to: analyze the similarity between low-frequency features of EEG and low-frequency features of image according to a first similarity function to determine a first similarity value; analyze the bidirectional contrast loss between low-frequency features of EEG and low-frequency features of image according to the first similarity value and a pre-constructed low-frequency bidirectional contrast loss function to obtain a low-frequency loss value; analyze the similarity between high-frequency features of EEG and high-frequency features of image according to the first similarity function to determine a second similarity value; and analyze the bidirectional contrast loss between high-frequency features of EEG and high-frequency features of image according to the first similarity value and a pre-constructed high-frequency bidirectional contrast loss function to obtain a high-frequency loss value.

[0143] The model training device provided in this embodiment can be used to execute the technical solutions of the above-described model training method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0144] Corresponding to the cross-modal retrieval method in the above embodiment, Figure 6 This is a structural block diagram of a cross-modal retrieval device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 6 The cross-modal detection device 600 includes: The acquisition unit 601 is used to acquire an EEG query signal and multiple candidate images; the feature extraction unit 602 is used to perform frequency component screening and global feature extraction on the EEG query signal and multiple candidate images through a cross-modal alignment model to obtain EEG query features and features of multiple candidate images; the similarity analysis unit 603 is used to determine the target image matching the EEG query signal among multiple candidate images by performing similarity analysis on the EEG query features and features of multiple candidate images. The cross-modal alignment model is trained using the model training method provided in any of the above embodiments.

[0145] The cross-modal detection device provided in this embodiment can be used to execute the technical solutions of the above-described cross-modal detection method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0146] refer to Figure 7The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0147] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0148] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0149] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0150] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0151] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0152] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0153] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not necessarily limiting in certain circumstances; for example, an acquisition unit can also be described as "a unit that acquires the page image and page description text of a webpage to be detected".

[0156] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0157] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0158] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0159] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0160] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A model training method, characterized in that, include: From the training dataset, obtain the paired samples required for the current training, the paired samples including the initial EEG signal samples and the initial image samples corresponding to the initial EEG signal samples; The paired samples are input into a cross-modal alignment model. In the cross-modal alignment model, the frequency components of the initial EEG signal samples are filtered to obtain target EEG signal samples. The frequency components of the initial image samples are also filtered to obtain target image samples. Global feature extraction is performed on the target EEG signal sample and the target image sample respectively to obtain global EEG features and global image features; Frequency domain decoupling is performed on the global EEG features and the global image features to obtain multiple local EEG features and multiple local image features. Different local EEG features correspond to different frequency components, and different local image features correspond to different frequency components. The training loss value for the current training iteration is determined by aligning multiple EEG local features and multiple image local features with features of the same frequency component. Based on the training loss value, the parameters of the cross-modal alignment model are adjusted to obtain the cross-modal alignment model after the current training.

2. The model training method according to claim 1, characterized in that, The step of filtering the frequency components of the initial EEG signal sample to obtain the target EEG signal sample includes: The time-series signals of each signal channel in the initial EEG signal sample are transformed from the time domain to the frequency domain to obtain the frequency domain signals corresponding to each signal channel. Based on a smooth frequency band selection mechanism, the frequency domain signals corresponding to each of the signal channels are filtered to obtain the selection signals corresponding to each of the signal channels. The selection signals corresponding to each of the signal channels are transformed from the frequency domain to the time domain to obtain the target EEG signal sample.

3. The model training method according to claim 2, characterized in that, The smooth-based frequency band selection mechanism filters the frequency domain signals corresponding to each signal channel to obtain the selection signal corresponding to each signal channel, including: Based on the set upper cutoff frequency, lower cutoff frequency, transition bandwidth, and cosine smooth transition function, a frequency band selection function for the EEG signal is constructed. The frequency domain signals corresponding to each signal channel are processed by the frequency band selection function to obtain the selection signal corresponding to each signal channel.

4. The model training method according to any one of claims 1 to 3, characterized in that, The step of filtering the frequency components of the initial image samples to obtain target image samples includes: The initial image sample is subjected to a two-dimensional frequency domain transformation and a centering rearrangement to obtain the initial spectrum corresponding to the initial image sample. The initial spectrum is processed using a radial frequency-gated mask to obtain the target spectrum; The target spectrum is subjected to inverse centering operation, and the target spectrum after inverse centering operation is transformed from the frequency domain to the spatial domain to obtain the target image sample.

5. The model training method according to any one of claims 1 to 3, characterized in that, The multiple EEG local features include low-frequency EEG features and high-frequency EEG features, and the multiple image local features include low-frequency image features and high-frequency image features; The frequency domain decoupling of the global EEG features and the global image features yields multiple local EEG features and multiple local image features, including: The global features of the electroencephalogram (EEG) and the global features of the image are transformed to the frequency domain to obtain the frequency domain features of the EEG and the frequency domain features of the image. The EEG frequency domain features are processed by low-frequency and high-frequency masks to obtain the first EEG frequency domain sub-feature and the second EEG frequency domain sub-feature. The image frequency domain features are processed using the low-frequency mask and the high-frequency mask to obtain the first image frequency domain sub-feature and the second image frequency domain sub-feature. The first EEG frequency domain sub-feature, the second EEG frequency domain sub-feature, the first image frequency domain sub-feature, and the second image frequency domain sub-feature are respectively inversely transformed to obtain the EEG low-frequency feature, the EEG high-frequency feature, the image low-frequency feature, and the image high-frequency feature.

6. The model training method according to claim 5, characterized in that, The step of determining the training loss value for the current training iteration by aligning multiple EEG local features and multiple image local features under the same frequency component includes: The low-frequency loss value for the current training iteration is obtained by aligning the low-frequency features of the EEG and the low-frequency features of the image with the low-frequency components. The high-frequency loss value for the current training iteration is obtained by aligning the high-frequency features of the EEG and the high-frequency features of the image under the high-frequency components. The training loss value is determined by weighting the low-frequency loss value and the high-frequency loss value according to the preset low-frequency weight coefficient and the preset high-frequency weight coefficient. The low-frequency weighting coefficient is greater than the high-frequency weighting coefficient.

7. The model training method according to claim 6, characterized in that, The step of aligning the low-frequency features of the EEG and the low-frequency features of the image at the low-frequency component level to obtain the low-frequency loss value for the current training iteration includes: Based on the first similarity function, the similarity between the low-frequency features of the EEG and the low-frequency features of the image is analyzed to determine the first similarity value; Based on the first similarity value and the pre-constructed low-frequency bidirectional contrast loss function, the bidirectional contrast loss between the EEG low-frequency features and the image low-frequency features is analyzed to obtain the low-frequency loss value. The step of aligning the high-frequency features of the EEG and the high-frequency features of the image under the high-frequency components to obtain the high-frequency loss value of the current training includes: Based on the first similarity function, the similarity between the high-frequency features of the EEG and the high-frequency features of the image is analyzed to determine the second similarity value; Based on the first similarity value and the pre-constructed high-frequency bidirectional contrast loss function, the bidirectional contrast loss between the high-frequency features of the EEG and the high-frequency features of the image is analyzed to obtain the high-frequency loss value.

8. A cross-modal retrieval method, characterized in that, include: Acquire EEG query signals and multiple candidate images; The frequency components of the EEG query signal and multiple candidate images are filtered and global features are extracted using a cross-modal alignment model to obtain EEG query features and multiple candidate image features. The cross-modal alignment model is trained using the model training method according to any one of claims 1 to 7. By performing similarity analysis on the EEG query features and multiple candidate image features, the target image matching the EEG query signal is determined among the multiple candidate images.

9. A model training device, characterized in that, include: The acquisition unit is used to acquire the paired samples required for the current training from the training dataset. The paired samples include the initial EEG signal samples and the initial image samples corresponding to the initial EEG signal samples. The frequency domain filtering unit is used to input the paired samples into the cross-modal alignment model, and in the cross-modal alignment model, to filter the frequency components of the initial EEG signal samples to obtain target EEG signal samples, and to filter the frequency components of the initial image samples to obtain target image samples. The feature extraction unit is used to perform global feature extraction on the target EEG signal sample and the target image sample respectively to obtain global EEG features and global image features; The frequency domain decoupling unit is used to decouple the global EEG features and the global image features in the frequency domain to obtain multiple local EEG features and multiple local image features. Different local EEG features correspond to different frequency components, and different local image features correspond to different frequency components. The feature alignment unit is used to perform feature alignment of multiple EEG local features and multiple image local features under the same frequency component to obtain the training loss value of the current training. The parameter adjustment unit is used to adjust the parameters of the cross-modal alignment model according to the training loss value, so as to obtain the cross-modal alignment model after the current training.

10. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the model training method as described in any one of claims 1 to 7, or to perform the cross-modal retrieval method as described in claim 8.

Images