Pet brain machine character translation method and device based on graph-to-text model

Abstract

A pet brain machine character translation method and device based on a graph-to-text model, and the method comprises the following steps: S1, collecting biological signals of a pet in real time, the biological signals comprising an electroencephalogram signal, a heart rate signal and image information based on pet vision; s2, preprocessing the collected biological signal of the pet; s3, performing character conversion on the acquired image information based on a graph-to-text model TransBLIP: S3-1, establishing a self-attention mechanism from a sequence to sequence prediction angle through an encoder Transform, and extracting image features in the image information; s3-2, extracting text features from the input text through a language model BERT; s3-3, realizing interaction between the image features and the text features by using a cross-modal interaction layer; (S3-4) generating a corresponding text according to the image features extracted in the step (S3-1); and S4, mapping the collected pet electroencephalogram signals and the characters generated in the step S3 by adopting a double-flow network architecture. Namely, the electroencephalogram information of the pet is translated in a text form by taking visual information as a medium, technical support can be provided for pet behavior interpretation in a brain-computer interface, the pet electroencephalogram signals are translated into languages and characters which can be understood by human beings, a pet owner is helped to accurately judge the emotion, intention and demand of the pet, and the communication experience between people and the pet is improved.

Description

Technical Field

[0001] The present invention relates to the fields of communication technology and artificial intelligence, and in particular to a pet brain-computer text translation method and device based on a picture-to-text model. Background Art

[0002] In recent years, the application of brain-computer interface (BCI) technology in animal research has received widespread attention. BCI provides a new way to directly understand the cognition and behavior of animals by recording and interpreting the electrophysiological signals of the animal brain. For example, in the latest study, Miiamaaria V. Kujala collected the EEG signals of dogs in real time through the pet brain-computer interface, and used machine learning algorithms to accurately identify the emotional state of dogs. In the invention patent CN202010045533.7, a device and method for using brain-computer pet language translation are proposed. The pet brain-computer device can learn and identify common signals of happiness, excitement, fear, and hunger of pets. Therefore, collecting the EEG signals of pets through pet BCI can help interpret and translate pet behavior patterns.

[0003] The above patent solution uses a BCI device to collect pet brain electrical signals and uses a computer system to implement the corresponding functions. However, the specific hardware facilities involved are not explained. In addition, the translation method used in the process is not clear, and only deep learning and big data analysis are mentioned in general.

[0004] However, although pet BCI provides a new way for humans to understand pet behavior patterns, the devices used to collect and store pet EEG information still cannot meet the needs in terms of comprehensiveness, accuracy and practicality. In particular, in the field of combining EEG signals with visual information for real-time translation, there is a lack of a device that can synchronously collect and process EEG information and visual data and express them in an intuitive text form.

[0005] In recent years, with the rapid development of graph-to-text models, this technology has been proficiently used to process various tasks such as image subtitle generation, image-text retrieval, and visual question and answer. For example, in the invention patent CN202311641472.0, an intelligent video surveillance system based on AI vision is disclosed. With the help of AI graph-to-text capabilities, the device performs real-time analysis of surveillance videos at the back end and triggers relevant warning rules through generated natural language instructions. Therefore, the BCI device is expected to integrate EEG signals, visual data, and text data with the help of advanced graph-to-text models to explore the feasibility of translating pet EEG signals into text. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a pet brain-computer text translation method and device based on a picture-to-text model.

[0007] To achieve the above object, the present invention provides the following technical solutions: A pet brain-computer text translation method based on a picture-based text model comprises the following steps: S1. Real-time collection of biological signals of the pet, wherein the biological signals include electroencephalogram signals, heart rate signals, and image information based on the pet's vision; S2, preprocessing the collected biological signals of the pet; S3, based on the image-to-text model TransBLIP, the collected image information is converted into text. S3-1. Through the encoder Transformer, a self-attention mechanism is established from the perspective of sequence-to-sequence prediction to extract image features from image information. S3-2. Extract text features from input text through the language model BERT; S3-3. Use the cross-modal interaction layer to realize the interaction between image features and text features; S3-4, generating corresponding text according to the image features extracted in S3-1; S4, a two-stream network architecture is used to map the collected pet EEG signals with the text generated in S3. The dual-stream architecture includes two independent streams, one for EEG signal processing and the other for text data processing, and finally achieves cross-modal alignment in a shared embedding space to complete the mapping between the pet EEG signal and the text generated in S3.

[0008] The separate streams for EEG signal processing include: The first input layer is used to input the preprocessed EEG signal; The time-frequency feature extraction layer converts the time-domain EEG signal into a time-frequency domain representation through short-time Fourier transform to generate a time and frequency feature map; The first feature learning layer processes the time and frequency feature maps to capture the local and global features of the EEG signal; The first projection layer projects the local and global features of the EEG signal into a fixed-length vector representation.

[0009] Separate streams for text data processing include: The second input layer is used to input preprocessed text data; Text embedding layer, which converts the text generated in S3 into text embedding; The second feature learning layer processes the text embedding through the self-attention mechanism of the encoder Transformer to further extract text features and ensure that semantic and time synchronization information is captured; The second projection layer projects the extracted text features into a vector representation of fixed length.

[0010] The continuous EEG signal is segmented into time windows, and each segment is processed synchronously with the generated text.

[0011] Based on the previous image generation and behavior description, the text data corresponding to the EEG signal is collected, and the text is segmented, stop words are removed, and syntactic analysis is performed to ensure the accurate expression of the text semantics.

[0012] The features extracted by the two independent streams are projected into the same shared embedding space, and the features in the two independent streams are aligned through contrastive learning or joint embedding strategies.

[0013] The steps for preprocessing EEG signals in S2 are as follows: S2-1 For EEG signal matrix To standardize, is the mean vector of each channel;

[0014] S2-2 calculates the covariance matrix C of the standardized EEG signal, where n is the number of time points:

[0015] S2-3 solves the eigenvalues ​​and eigenvectors of the covariance matrix C, where λ is the eigenvalue and ν is the corresponding eigenvector:

[0016] S2-4 selects the eigenvectors corresponding to the first k eigenvalues ​​according to the size of the eigenvalues ​​to form the characteristic matrix w, and then projects the original EEG data into the new principal component space, where Y is the EEG signal moment after dimensionality reduction: .

[0017] A pet translation device is an integrated BCI collection device, the integrated BCI collection device comprising: A pet helmet, wherein a camera device for acquiring visual image information of the pet is disposed on the top of the pet helmet, and an eight-channel cortical electrode for collecting electroencephalographic signals of different functional cortical areas of the pet is disposed on the inner side of the pet helmet; A pet collar equipped with a heart rate sensor for obtaining the pet's heart rate in real time; The controller obtains the biological signals of the pet collected by the pet helmet and the pet collar respectively, and realizes the mutual conversion between the pet EEG signals and text based on the above-mentioned pet brain-computer text translation method based on the image-to-text model.

[0018] The pet helmet is also provided with a noise reduction module for noise reduction processing of signals collected by eight-channel cortical electrodes.

[0019] The noise reduction module comprises: 50Hz oscillator, used to remove power frequency interference from pet EEG signals; Instrumentation amplifier, used to amplify weak pet EEG signals; The right leg driving circuit is used to reduce common-mode signal interference and enhance the signal-to-noise ratio; A low-pass filter is used to remove noise signals outside the frequency range.

[0020] Beneficial effects of the present invention: Pet EEG information translation technology based on a two-stage deep learning algorithm. In the first stage, the collected visual information is converted into human natural language through the graph-to-text model TransBLIP, and in the second stage, the mutual conversion between natural language and EEG information can be achieved through a dual-stream network architecture. That is, the pet's EEG information is translated into text form using visual information as a medium, which can provide technical support for the interpretation of pet behavior in brain-computer interfaces. By translating pet EEG signals into human-understandable language and text, it helps pet owners accurately judge the pet's emotions, intentions and needs, and improves the communication experience between people and pets. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the structure of the device provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the use status of the device provided by the present invention.

[0023] Figure 3 It is a schematic diagram of the graph-to-text model TransBLIP of the present invention.

[0024] Figure 4 Schematic diagram of the dual-stream network architecture model of the present invention.

[0025] Figure 5 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0028] The present invention provides a pet brain-computer text translation method based on a picture-to-text model, which comprises the following steps: S1. Real-time collection of biological signals of the pet, wherein the biological signals include electroencephalogram signals, heart rate signals, and image information based on the pet's vision; EEG signals can be used to monitor the pet's brain activity to help determine its emotions, cognitive state, and behavioral patterns; visual information can be used to record the pet's perspective, which can help better understand the pet's activity habits, things of interest, and potential environmental stressors. Heart rate can be used to monitor the pet's physiological state to help determine the pet's health status, physical fitness level, or stress response.

[0029] S2, preprocessing the collected biological signals of the pet; In order to ensure the temporal sequence alignment of the image, EEG, and heart rate signals, the EEG signal is first subjected to preliminary noise reduction by the noise reducer in the device to remove power frequency interference, reduce common mode signals, and amplify weak EEG signals. The EEG after preliminary noise reduction is uploaded to the cloud for algorithmic noise reduction, using algorithms such as bandpass filtering and principal component analysis to remove various artifacts such as ECG and electrooculogram, and reconstruct high-quality EEG signals.

[0030] S3, based on the image-to-text model TransBLIP, the collected image information is converted into text. In order to ensure that the image can reflect the pet's attention, the present invention proposes TransBLIP, a picture-to-text model based on the pet's perspective. Transformer can be used as a powerful encoder for image segmentation tasks, enhancing finer details by restoring local spatial information. It establishes a self-attention mechanism from the perspective of sequence-to-sequence prediction, and thus has a powerful global information capture capability. BLIP introduces a new visual language pre-training paradigm that can arbitrarily combine and fully utilize two pre-trained visual encoders and LLMs without the need to pre-train the entire architecture end-to-end. Therefore, TransBLIP enhances image detail recovery by introducing the Transformer encoder in the BLIP model to generate highly consistent text descriptions.

[0031] In order to improve the recognition accuracy of image details, the Transformer is modified in the traditional image-to-text model BLIP to extract features from the image as an encoder. With the help of the generation mechanism of the BLIP model, the visual features are input into the text generation module, and the descriptive text is generated through the decoder of the Transformer. And the TransBLIP model will be fine-tuned through a specific data set to adapt to the image-to-text conversion that pets are highly interested in.

[0032] That is, TransBLIP utilizes the self-attention mechanism of Transformer and the pre-training paradigm of BLIP, aiming to achieve image segmentation and image-text interaction.

[0033] Among them, S3-1, through the encoder Transformer, establishes a self-attention mechanism from the perspective of sequence-to-sequence prediction to extract image features from image information; S3-2. Extract text features from input text through the language model BERT; S3-3. Use the cross-modal interaction layer to realize the interaction between image features and text features; S3-4. Generate corresponding text based on the image features extracted in S3-1.

[0034] The TransBLIP proposed in the present invention mainly introduces the self-attention mechanism of Transformer on the basis of the basic BLIP model, thereby restoring strong image details and generating highly consistent text descriptions.

[0035] like Figure 3 As shown, TransBLIP includes: 1.1 Input Module Image input: Use a powerful visual encoder Transformer to extract image features, focusing on restoring local spatial information to enhance image details. The input image is resized to an appropriate size (224x224) and normalized.

[0036] Text input: The pre-trained language model BERT is used to encode the text to ensure that the text sequence is effectively represented. The text is segmented, special tags are added, and converted into a fixed-length sequence (32).

[0037] 1.2 Cross-modal interaction layer The self-attention mechanism is used to realize the interaction between image features and text features. This layer can capture global information and combine the details of the image with the contextual information of the text.

[0038] 1.3 Output Module Text generation: Generate corresponding description text based on the image.

[0039] 2. Data Preparation 2.1 Dataset Selection Choose appropriate datasets such as COCO and PASCAL VOC, and ensure that the datasets contain images and their corresponding labels and descriptions.

[0040] 2.2 Data Preprocessing Image processing: Scale the image to the model input size and perform data augmentation (flipping, cropping) to improve the generalization ability of the model.

[0041] Text processing: perform word segmentation and filling to ensure that the format of the input text is unified.

[0042] 3. Training steps 3.1 Model Training Use the pre-trained BLIP paradigm and the visual encoder Transformer for training 3.2 Fine-tuning Fine-tune different hyperparameters to optimize the model training effect.

[0043] 4. Evaluation The model performance is evaluated using standard metrics, IoU (Intersection over Union): evaluates the accuracy of segmentation: BLEU and CIDEr: evaluates the quality of generated text.

[0044] S4, a two-stream network architecture is used to map the collected pet EEG signals with the text generated in S3.

[0045] The dual-stream network architecture includes two independent streams, one for EEG signal processing and the other for text data processing, and finally achieves cross-modal alignment in a shared embedding space to complete the mapping between the pet EEG signal and the text generated in S3.

[0046] In order to achieve cross-modal alignment and fusion of two different modal data, electroencephalogram (EEG) data and text data, the present invention builds a dual-stream network architecture of EEG signals and text. First, the EEG signal is subjected to time-frequency analysis of short-time Fourier transform to extract time-frequency features, and then a temporal convolutional neural network is used to capture the temporal dependency and local features in the EEG signal to output a high-dimensional EEG stream; at the same time, the text is converted into an embedding vector through the pre-trained language model BERT, and a high-dimensional text stream is output; after extracting high-dimensional feature vectors from the EEG stream and the text stream respectively, the features of the two modalities are projected into the same shared embedding space through a fully connected layer to ensure that the features of different modalities have the same dimension and semantic consistency in this space.

[0047] The dual-stream network architecture model takes the image-to-text results of the TransBLIP model as input, and trains the matching EEG data, ultimately achieving the mapping conversion between pet EEG information and text data.

[0048] like Figure 5 As shown, the present invention provides a two-stream network architecture, which realizes the mapping between pet EEG signals and generated text through cross-modal alignment. The entire two-stream network architecture contains two independent streams: one processes EEG signals and the other processes text data, and finally performs cross-modal alignment in a shared embedding space.

[0049] 1.1 EEG stream (signal processing) Input layer: Input the processed EEG signal, the duration of which can be adjusted according to the experimental design (each segment is 3 seconds, the sampling rate is 128Hz).

[0050] Time-frequency feature extraction layer: Use short-time Fourier transform (STFT) to convert the time-domain EEG signal into a representation in the time-frequency domain to generate a time and frequency feature map. STFT performs a local Fourier transform on the EEG signal through a sliding window to capture the frequency information of the signal changing over time.

[0051] Feature learning layer: Convolutional neural network (CNN) or recurrent neural network (such as LSTM) is used to process the time-frequency graph to capture the local and global features of the EEG signal. The convolution layer can effectively extract spatial features, while LSTM captures temporal dynamics.

[0052] Projection layer: Projects the features to a fixed-length vector representation so that they are aligned with the text features in the embedding space.

[0053] 1.2 Text Stream (Text Processing) Input layer: Receives preprocessed text data, and the text content is a generated language description synchronized with the EEG signal.

[0054] Text embedding layer: Use the pre-trained language model BERT to convert text into vector embeddings. These embeddings can capture the context and semantic information of the text.

[0055] Feature learning layer: The text embedding is processed through Transformer to further extract text features to ensure that semantic and time synchronization information is captured.

[0056] Projection layer: Similar to the EEG stream, text features are also projected into a fixed-length vector representation.

[0057] 1.3 Cross-modal alignment Shared embedding space: The features of the EEG stream and the text stream are projected into the same shared embedding space. Through contrastive learning or joint embedding strategies, the model gradually aligns the features in the two streams during training.

[0058] 2. Data processing 2.1 EEG data processing Data Collection: The pet’s EEG signals are collected through our dedicated equipment to ensure that the signals are free of noise and artifacts.

[0059] Segmented processing: The continuous EEG signal is segmented into time windows (such as 5 seconds), and each segment of the signal is processed synchronously with the generated text.

[0060] Time-frequency analysis: Use short-time Fourier transform to perform time-frequency analysis on each EEG signal and obtain the time-frequency diagram as input.

[0061] 2.2 Text Data Processing Text collection: Based on the previous image generation and behavior description, collect text data corresponding to the EEG signal.

[0062] Text preprocessing: segment text, remove stop words, and perform syntactic analysis to ensure accurate expression of text semantics.

[0063] Embedding representation: BERT is used to embed the processed text and generate a text vector of fixed length.

[0064] 3. Training steps 3.1 Training Objectives Loss function: A cross-modal contrastive loss is adopted to ensure that the similarity between EEG signals and the corresponding texts is maximized while distinguishing mismatched EEG-text pairs.

[0065] 3.2 Optimization Strategy Optimizer: Use Adam optimizer, set learning rate (1e-4), and combine weight decay to avoid overfitting.

[0066] Training process: The model is updated through batch training. The model first processes the EEG stream and text stream data, and then calculates the contrast loss in the shared embedding space.

[0067] 4. Model Evaluation 4.1 Evaluation Criteria Alignment accuracy: measures the matching degree between EEG signal and text, ensuring the quality of cross-modal alignment.

[0068] Reconstruction quality: Evaluate the accuracy of EEG signals or text reconstructed from the shared embedding space, using mean squared error (MSE) and cosine similarity as evaluation metrics.

[0069] The steps for preprocessing EEG signals in S2 are as follows: S2-1 For EEG signal matrix To standardize, is the mean vector of each channel;

[0070] S2-2 calculates the covariance matrix C of the standardized EEG signal, where n is the number of time points:

[0071] S2-3 solves the eigenvalues ​​and eigenvectors of the covariance matrix C, where λ is the eigenvalue and ν is the corresponding eigenvector:

[0072] S2-4 selects the eigenvectors corresponding to the first k eigenvalues ​​according to the size of the eigenvalues ​​to form the characteristic matrix w, and then projects the original EEG data into the new principal component space, where Y is the EEG signal moment after dimensionality reduction: .

[0073] like Figure 1 and Figure 2 As shown, the present invention also provides a pet translation device, which is an integrated BCI collection device, and the integrated BCI collection device includes: A pet helmet is adapted to the pet's head shape. A camera device for obtaining visual image information of the pet is arranged on the top of the pet helmet. The camera device is consistent with the pet's visual direction, thereby being able to capture the pet's first visual image. An eight-channel cortical electrode for collecting EEG signals of different functional cortical areas of the pet is arranged on the inside of the pet helmet. When the pet wears the pet helmet, the eight-channel cortical electrode can contact the cortex of the corresponding position of the pet, thereby obtaining the most accurate EEG signal. A pet collar, which is worn around the pet's neck and has a heart rate sensor for obtaining the pet's heart rate in real time; The controller is a central controller, which is configured as a medium for preprocessing and storing pet brain waves, storing heart rate signals, storing pictures or video signals, and for cloud interaction and device switch control, respectively obtaining the biological signals of the pet collected by the pet helmet and the pet collar, and realizing the mutual conversion between the pet brain wave signals and text based on the above-mentioned pet brain-computer text translation method based on the image-to-text model.

[0074] The Linux board of the central controller is used to store the collected pet biological signals. At the same time, the central controller communicates with the analog-to-digital converter through the serial peripheral interface, transmits the processed eight-channel EEG signals and original heart rate signals, and digitizes them. They are then stored in the ROM, and all the data in the ROM is transmitted to the Linux board at one time after a period of time. At the same time, the image data of the sports camera can be obtained by the Linux board through the MIPI interface. Finally, the Linux board stores each data in different areas of its own memory card.

[0075] The eight-channel cortical electrodes can collect the pet's cortical EEG signals as comprehensively as possible. By arranging eight electrodes in different special cortical areas of the pet's head, high-precision monitoring of activities in different brain areas can be achieved, capturing more brain wave details. Based on the different functional cortical areas of the pet, eight-channel non-interfering pet brain-computer interface electrodes are used to ensure that the EEG signals between the electrodes do not interfere with each other, and the EEG signals in different functional areas do not interfere with each other, thereby obtaining more accurate and comprehensive EEG signals. The collected EEG signals are denoised by hardware and software algorithms to improve the signal-to-noise ratio and reconstruct high-quality EEG signals.

[0076] The pet helmet is also provided with a noise reduction module for noise reduction processing of signals collected by eight-channel cortical electrodes.

[0077] The noise reduction module comprises: 50Hz oscillator, used to remove power frequency interference from pet EEG signals; Instrumentation amplifier, used to amplify weak pet EEG signals; The right leg driving circuit is used to reduce common-mode signal interference and enhance the signal-to-noise ratio; A low-pass filter is used to remove noise signals outside the frequency range.

[0078] After the EEG information is uploaded to the cloud, a bandpass filtering algorithm is used to remove noise signals outside the frequency range, and an independent component analysis algorithm is used to remove artifacts such as electrooculogram (EOG) and electromyography (EMG), to reconstruct high-quality pet EEG signals.

[0079] It also supports cloud interaction functions, wirelessly transmitting data to the cloud to facilitate remote monitoring and big data analysis, and achieve real-time behavior tracking.

[0080] like Figure 5 As shown, the present invention comprehensively collects pet EEG signals through an eight-channel brain-computer device, performs software and hardware noise reduction on it, reconstructs high-quality EEG signals, and uploads them to the cloud. The cloud performs EEG-text mapping through a trained dual-stream network architecture, and translates the EEG signals into language and text, which can be displayed on a display screen, which can be displayed on a terminal or mobile device such as a mobile phone.

[0081] The embodiments should not be regarded as limiting the present invention, but any improvements made based on the spirit of the present invention should be within the protection scope of the present invention.

Claims

1. A pet brain-computer text translation method based on a picture-to-text model, characterized in that: It includes the following steps: S1. Real-time collection of biological signals of the pet, wherein the biological signals include electroencephalogram signals, heart rate signals, and image information based on the pet's vision; S2, preprocessing the collected biological signals of the pet; S3, based on the image-to-text model TransBLIP, the collected image information is converted into text. S3-1. Through the encoder Transformer, a self-attention mechanism is established from the perspective of sequence-to-sequence prediction to extract image features from image information. S3-2. Extract text features from input text through the language model BERT; S3-3. Use the cross-modal interaction layer to realize the interaction between image features and text features; S3-4, generating corresponding text according to the image features extracted in S3-1; S4, a two-stream network architecture is used to map the collected pet EEG signals with the text generated in S3. The dual-stream architecture includes two independent streams, one for EEG signal processing and the other for text data processing, and finally achieves cross-modal alignment in a shared embedding space to complete the mapping between the pet EEG signal and the text generated in S3.

2. According to claim 1, a pet brain-computer text translation method based on a picture-to-text model is characterized in that: The separate streams for EEG signal processing include: The first input layer is used to input the preprocessed EEG signal; The time-frequency feature extraction layer converts the time-domain EEG signal into a time-frequency domain representation through short-time Fourier transform to generate a time and frequency feature map; The first feature learning layer processes the time and frequency feature maps to capture the local and global features of the EEG signal; The first projection layer projects the local and global features of the EEG signal into a fixed-length vector representation.

3. The pet brain-computer text translation method based on the image-to-text model according to claim 1 is characterized in that: Separate streams for text data processing include: The second input layer is used to input preprocessed text data; Text embedding layer, which converts the text generated in S3 into text embedding; The second feature learning layer processes the text embedding through the self-attention mechanism of the encoder Transformer to further extract text features and ensure that semantic and time synchronization information is captured; The second projection layer projects the extracted text features into a vector representation of fixed length.

4. The pet brain-computer text translation method based on the image-to-text model according to claim 2 is characterized in that: The continuous EEG signal is segmented into time windows, and each segment is processed synchronously with the generated text.

5. The pet brain-computer text translation method based on the image-to-text model according to claim 3 is characterized by: Based on the previous image generation and behavior description, the text data corresponding to the EEG signal is collected, and the text is segmented, stop words are removed, and syntactic analysis is performed to ensure the accurate expression of the text semantics.

6. A pet brain-computer text translation method based on a picture-to-text model according to any one of claims 1 to 5, characterized in that: The features extracted by the two independent streams are projected into the same shared embedding space, and the features in the two independent streams are aligned through contrastive learning or joint embedding strategies.

7. The pet brain-computer text translation method based on the image-to-text model according to claim 1 is characterized in that: The steps for preprocessing EEG signals in S2 are as follows: S2-1 For EEG signal matrix To standardize, is the mean vector of each channel; S2-2 calculates the covariance matrix C of the standardized EEG signal, where n is the number of time points: S2-3 solves the eigenvalues ​​and eigenvectors of the covariance matrix C, where λ is the eigenvalue and ν is the corresponding eigenvector: S2-4 selects the eigenvectors corresponding to the first k eigenvalues ​​according to the size of the eigenvalues ​​to form the characteristic matrix w, and then projects the original EEG data into the new principal component space, where Y is the EEG signal moment after dimensionality reduction: 。 8. A pet translation device, characterized in that: It is an integrated BCI collection device, which includes: A pet helmet, wherein a camera device for acquiring visual image information of the pet is disposed on the top of the pet helmet, and an eight-channel cortical electrode for collecting electroencephalographic signals of different functional cortical areas of the pet is disposed on the inner side of the pet helmet; A pet collar equipped with a heart rate sensor for obtaining the pet's heart rate in real time; The controller obtains the biological signals of the pet collected by the pet helmet and the pet collar respectively, and realizes the mutual conversion between the pet EEG signals and text based on the pet brain-computer text translation method based on the image-to-text model described in any one of claims 1 to 7.

9. A pet translation device according to claim 8, characterized in that: The pet helmet is also provided with a noise reduction module for noise reduction processing of signals collected by eight-channel cortical electrodes.

10. A pet translation device according to claim 9, characterized in that: The noise reduction module comprises: 50Hz oscillator, used to remove power frequency interference from pet EEG signals; Instrumentation amplifier, used to amplify weak pet EEG signals; The right leg driving circuit is used to reduce common-mode signal interference and enhance the signal-to-noise ratio; A low-pass filter is used to remove noise signals outside the frequency range.

Images