A method and system for generating human brain cognitive text description of natural images

Abstract

The present invention relates to the field of computer vision and cognitive neuroscience, and in particular to a method and system for generating human brain cognitive text description of natural images. The present invention pre-constructs an fMRI-Image mapping model that maps fMRI signals with image feature vectors and text feature vectors obtained by CLIP for various complex scenes, obtains fMRI signals generated by the human brain based on stimulus images, and pre-processes the obtained fMRI signals to obtain processed fMRI signals, inputs the pre-processed fMRI signals into the fMRI-Image mapping model to obtain image feature vectors and text feature vectors, and inputs the image feature vectors and text feature vectors output by the fMRI-Image mapping model into a pre-trained language mapping model to generate image text descriptions, thereby being able to extract visual semantic features from complex natural scenes and facilitating accurate generation of text descriptions of natural images.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and cognitive neuroscience, and in particular to a method and system for generating human brain cognitive text descriptions of natural images. Background Art

[0002] In recent years, with the rapid development of neuroimaging technology, researchers have been able to explore the visual information processing mechanism of the human brain more deeply. Functional Magnetic Resonance Imaging (fMRI), as a non-invasive technology with high spatial resolution, can indirectly monitor neural activity by detecting changes in blood oxygenation level dependent (BOLD) signals. This makes it an important tool for studying the cognitive function of the human brain. Among them, using these neural signals to decode the content of external visual stimuli, especially generating text descriptions of natural images, is an important research direction in the current field of cognitive neuroscience and computer vision.

[0003] Traditional decoding methods mostly use simple images or artificially synthesized images as stimulus sources, which results in limited semantic information for decoding and makes it difficult to cope with complex natural scenes. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method and system for generating human brain cognitive text description of natural images, which can extract visual semantic features from complex natural scenes and help to accurately generate text descriptions of natural images.

[0005] In order to implement the above technical solution, in a first aspect, the present invention provides a method for generating a human brain cognitive text description of a natural image, comprising the following steps:

[0006] Step 1: Pre-build and train the fMRI-Image mapping model;

[0007] Step 2: Pre-build and train the language mapping model;

[0008] Step 3, obtaining fMRI signals generated by the human brain based on the stimulus image, and preprocessing the obtained fMRI signals to obtain processed fMRI signals;

[0009] Step 4: input the preprocessed fMRI signal into the constructed and trained fMRI-Image mapping model to obtain an image feature vector and a text feature vector for describing the stimulus image;

[0010] Step 5: Input the image feature vector and text feature vector output by the fMRI-Image mapping model into the constructed and trained language mapping model to generate a text description for the stimulus image.

[0011] Furthermore, the step 1 comprises:

[0012] Constructing a CNN network model, and training the CNN network model to obtain an fMRI-Image mapping model;

[0013] collecting a signal set including fMRI signals, and preprocessing the fMRI signals in the signal set to eliminate useless signals;

[0014] Acquire a stimulus image corresponding to each fMRI signal in the signal set and a text description matching the stimulus image, and input each acquired stimulus image and text description matching the stimulus image into a text-image comparison pre-training model to obtain an actual image feature vector and an actual text feature vector for each stimulus image;

[0015] The CNN model is trained based on the actual image feature vector and actual text feature vector of each fMRI signal in the dataset and its corresponding stimulus image to obtain the fMRI-Image mapping model.

[0016] Furthermore, the training of the CNN model based on the actual image feature vector and text feature vector of each fMRI signal in the data set and its corresponding stimulus image includes:

[0017] The maximum normalized contrast loss of information in the CNN network model is defined as follows:

[0018]

[0019] in, is the mathematical expectation, Respectively The predicted value and true value of samples; B is the number of samples, and B is a value greater than 1, The value of is from 1 to all values ​​in B; For the The true value of samples; is a non-negative hyperparameter selected by parameter search;

[0020] Based on the defined maximum normalized contrast loss, for the image feature vector of each stimulus image, the CNN network model uses the following loss function for loss calculation during training:

[0021] in, represents the Euclidean distance, represents the cosine similarity, Indicates The actual image feature vector of the image sample is the predicted image feature vector output by the CNN network model; , , is a non-negative hyperparameter selected by parameter search;

[0022] Based on the defined maximum normalized contrast loss, for the text feature vector of each stimulus image, the CNN network model uses the following loss function for loss calculation during training:

[0023]

[0024] in, Indicates The actual text feature vector of the text samples, It is the predicted text feature vector output by the CNN network model.

[0025] Furthermore, the step five comprises:

[0026] The Transformer network projects the image feature vector and text feature vector output by the fMRI-Image mapping model into the latent space of the text generation model;

[0027] The text generation model generates text descriptions based on the image feature vector and text feature vector projected into the latent space.

[0028] In a second aspect, the present invention provides a system for generating human brain cognitive text descriptions of natural images, the system comprising: a first construction and training module for pre-constructing and training an fMRI-Image mapping model, wherein the fMRI-Image mapping model adopts a CNN network model;

[0029] The second construction and training module is used to construct and train a language mapping model; wherein the language mapping model includes a Transformer network and a text generation model;

[0030] A signal acquisition module, used for acquiring fMRI signals generated by the human brain based on stimulus images;

[0031] A signal processing module, used for preprocessing the acquired fMRI signals;

[0032] The fMRI-Image mapping model is used to receive each fMRI signal and output an image feature vector and a text feature vector of a stimulus image corresponding to each fMRI signal;

[0033] The language mapping model is used to generate a text description for the stimulus image based on the image feature vector and text feature vector output by the fMRI-Image mapping model.

[0034] Furthermore, the system further comprises:

[0035] A signal collection module, used for collecting a signal set including fMRI signals;

[0036] An image acquisition module, used for acquiring a stimulus image corresponding to each fMRI signal in the collected signal set;

[0037] The text-image comparison pre-trained model is used to extract the actual image feature vector and the text feature vector for each stimulus image obtained.

[0038] In a third aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for generating human brain cognitive text description of natural images described above.

[0039] The beneficial effects of the present invention are:

[0040] An fMRI-Image mapping model is pre-constructed and trained based on publicly available fMRI signals, image feature vectors of stimulus images that match the publicly available fMRI signals, and text feature vectors for the stimulus images. The acquired fMRI signals generated by the human brain based on the target stimulus images are then processed and input into the pre-constructed fMRI-Image mapping model to obtain image feature vectors and text feature vectors. The obtained image feature vectors and text feature vectors are input into the constructed and pre-trained language mapping model to generate a text description for the target stimulus image, thereby being able to extract visual semantic features from complex natural scenes, and facilitating accurate generation of text descriptions of natural images.

[0041] (2) The language mapping model constructed by the present invention includes a Transformer network and a text generation model, so that the language mapping model can project the image feature vector and the text feature vector into the latent space of the text generation model through the Transformer network. At the same time, the Transformer network uses the self-attention mechanism to capture the contextual relationship between the image feature vector and the text feature vector projected into the latent space of the text generation model, so that the text generation model can generate text features that conform to semantic logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 The present invention is a flowchart of the preprocessing stage of a method for generating human brain cognitive text descriptions of natural images.

[0044] Figure 2 The present invention is a flowchart of the processing stage of a method for generating human brain cognitive text description of natural images.

[0045] Figure 3 It is a schematic diagram of the CNN network model structure of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present invention. Unless otherwise specified, each technical and scientific term used in this embodiment has the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0050] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. Relevant scientific research or technical personnel in this field can determine the specific meanings of the above terms in the present invention according to specific circumstances, and they should not be understood as limiting the present invention.

[0051] Embodiment 1:

[0052] like Figures 1 to 3 As shown, this embodiment provides a method for generating a human brain cognitive text description of a natural image, comprising the following steps:

[0053] (I) Preprocessing stage (e.g. Figure 1 as shown).

[0054] S1: Collect a signal set including multiple fMRI signals, and preprocess the fMRI signals in the signal set to eliminate useless signals.

[0055] For ease of explanation and understanding, the fMRI signal is recorded as .

[0056] Among them, the fMRI signal is a signal that has been made public. Each fMRI signal covers a large amount of voxel BOLD response information. The BOLD response information of each voxel reflects the overall response of neurons in the corresponding millimeter-level volume area in the brain to external stimuli. The response information of some brain regions may be irrelevant. For the BOLD response information collected from key areas such as the visual cortex of the brain, these irrelevant response information will produce interfering noise signals. Therefore, the fMRI data set in the fMRI data set is standardized to remove the interfering noise signals and correct the noise such as motion artifacts.

[0057] It should be noted that the specific steps for generating voxel BOLD response information are: confirm and select the ROI (Region Of Interest) in the subject's brain through a mask image of the brain area related to the brain's visual function pointed out in neuroscience-related research. When the subject views the relevant stimulus image, the voxels in the selected ROI area respond to the visual information and generate relevant voxel BOLD response information.

[0058] S2: Obtain the stimulus image and text description corresponding to each fMRI signal in the signal set, and input each stimulus image and text description obtained into the Contrastive Language-Image Pre-Training (CLIP) model to obtain the actual image feature vector and text feature vector of each stimulus image.

[0059] S3: Construct an fMRI-Image mapping model and train the constructed fMRI-Image mapping model based on each fMRI signal in the dataset and the actual image feature vector and text feature vector corresponding to each fMRI signal.

[0060] Specifically, a CNN network model is constructed, the constructed CNN network model is used as an fMRI-Image mapping model, and the fMRI-Image mapping model is trained. The CNN network model consists of multiple layers of residual blocks and convolutional layers, and finally outputs a one-dimensional prediction result.

[0061] Specifically, Figure 3 As shown, the CNN network model structure includes:

[0062] A) Input layer, which consists of a convolutional layer (Conv1d) and a ReLU activation function, and the input dimension is 15724. The convolutional layer performs a convolution operation and the output dimension is (64, 7862), indicating 64 channels and 7862 eigenvalues ​​per channel.

[0063] B) Multiple residual blocks (ResBlock_1d), each residual block consists of a series of convolutional layers and ReLU activation functions with skip connections.

[0064] C) Fully connected layer block (FC Block), which is behind the residual block and contains two convolutional layers and two ReLU activation functions. The convolutional layers and ReLU activation functions are connected alternately. The output dimension of the fully connected layer block becomes (768, 1), indicating that the final output has 768 eigenvalues.

[0065] D) Output layer: The output layer is a fully connected layer that outputs a one-dimensional prediction result.

[0066] The training of the constructed fMRI-Image mapping model based on the actual image feature vector and text feature vector of each fMRI signal in the data set and its corresponding stimulus image specifically comprises the following steps:

[0067] S3.1: Define the maximum normalized contrast loss in the CNN network model.

[0068] In the CNN network model, for the first samples, the maximum normalized contrast loss is defined as:

[0069]

[0070] Among them, among them, is the mathematical expectation, Respectively The predicted value and true value of samples; B is the number of samples, and the B value is greater than 1. The value of is from 1 to all values ​​in B. For the The true value of samples; is a non-negative hyperparameter selected via parameter search.

[0071] S3.2: Based on the above definition, for the image feature vector, the following loss function is used in the CNN network model training:

[0072]

[0073] in represents the Euclidean distance, represents the cosine similarity, Indicates The actual image feature vector of image samples, It is the predicted image feature vector output by the CNN network model; , , is a non-negative hyperparameter selected via parameter search.

[0074] S3.3: Based on the above definition, for the text feature vector, the following loss function is used in the CNN network model:

[0075]

[0076] in, Indicates The actual text feature vector of the text samples, The predicted text feature vector output by the CNN network model; , , is a non-negative hyperparameter selected via parameter search.

[0077] More specifically, during the training of the CNN network model, the input fMRI signal is only used in the output layer of the CNN network, that is, the last layer to compare the image feature vector and the text feature vector. The training goal is to make the loss function L shown in the formula img After the training of the constructed fMRI-Image mapping model is completed, in the testing (inference) stage, the fMRI signal is output through the CNN network, and the result is the mapping of the fMRI signal to the image feature vector and the text feature vector.

[0078] It should also be noted that mapping fMRI to image feature vectors and text feature vectors are two independent processes, that is, two networks need to be trained separately. The structures of these two networks are the same, both are customized CNN structures, but the parameters are different because the training data are different, one uses image data and the other uses text data.

[0079] S4: construct a language mapping model and train the constructed language mapping model.

[0080] Specifically include the following steps:

[0081] S4.1: Use the Transformer network and a text generation model (e.g., the GTP-3 model) to build a language mapping model.

[0082] Among them, the Transformer network structure includes:

[0083] Attention module: Multi-head self-attention mechanism, calculates query, key and value, and performs self-attention calculation. The self-attention mechanism can capture the relationship between different parts of the image feature vector and text feature vector extracted from the stimulus image and the text description of the stimulus image through the CLIP model, thereby enhancing the modeling ability of complex image information and complex text information.

[0084] Multilayer Perceptron Module: It is used for feed-forward operation in the Transformer layer and contains two linear layers and activation functions. The Multilayer Perceptron Module helps to further nonlinearly transform and refine the features, so that the features after the self-attention module can better capture complex patterns and relationships.

[0085] Transformer layer: Contains the basic Transformer layer of the attention mechanism and feedforward network, as well as layer normalization and residual connection. The Transformer network structure consists of 11 Transformer layers. The Transformer layer can refine the input features layer by layer, so that the CNN network model can learn the potential correlation between the image feature vector and the text feature vector extracted by the CLIP model at multiple levels. The residual connection and layer normalization ensure that the information is effectively transmitted in the deep network. Through the gradual refinement of multiple layers of Transformer layers, the image feature vector and text feature vector extracted by the CLIP model can be adjusted and processed more finely in each layer, and the image features and text features of CLIP are adaptively aligned and adjusted to make them more consistent with the feature space of downstream tasks (such as the GPT-3 language model).

[0086] The training of the constructed language mapping model comprises the following steps:

[0087] S4.2: Keep the text generation model parameters fixed and only train the Transformer network. This can avoid training the text generation model parameters, thereby greatly reducing the number of trainable parameters.

[0088] Specifically, the text generation model is kept fixed and the Transformer network is used as the framework of the mapping model. The Transformer network allows global attention between input tokens while reducing the number of parameters for long sequences. In addition, the attention mechanism helps capture the contextual relationship between the image feature vector and the text feature vector and the language model generation, so that the generated text is not only grammatically coherent, but also highly semantically matched with the input visual information.

[0089] The image feature vector and text feature vector obtained by the CLIP model are used as input, and the goal is to convert them into feature representations suitable for the implicit space of the language model (such as GPT-3). The training samples include the image feature vector and the corresponding text feature vector extracted by the CLIP model.

[0090] Specifically, the following steps are included:

[0091] A) The Transformer network adjusts the actual image feature vector and the actual text feature vector through a multi-layer structure (including a multi-head self-attention mechanism, a feedforward network, layer normalization, and residual connections) to make them more in line with the input requirements of the language model.

[0092] B) The image feature vector and text feature vector processed by the Transformer network are input into the text generation language model to generate a text description that matches the image.

[0093] C) Using the cross entropy loss function, compare the difference between the generated text description and the actual text description for the image, and adjust the parameters of the Transformer network through backpropagation and optimization algorithms (such as Adam).

[0094] (II) Text description generation stage (e.g. Figure 2 as shown).

[0095] Q1: Obtain fMRI signals generated by the human brain based on stimulus images, and preprocess the acquired fMRI signals to obtain processed fMRI signals.

[0096] Specifically, the test subjects are asked to watch the stimulus images, and the fMRI signals generated by the corresponding ROI area in the brain when the test subjects watch the stimulus images are obtained, and the obtained fMRI signals are preprocessed to remove interference signals.

[0097] Among them, the preprocessing of the acquired fMRI signal includes correcting noise such as motion artifacts.

[0098] Q2: Input the preprocessed fMRI signal into the fMRI-Image mapping model to obtain the image feature vector and text feature vector.

[0099] Q3: Input the image feature vector and text feature vector output by the fMRI-Image mapping model into the pre-trained language mapping model to generate image text description.

[0100] The specific steps include:

[0101] Q3.1: The Transformer network projects the image feature vector and text feature vector output by the fMRI-Image mapping model into the latent space of the text generation model.

[0102] Q3.2: The text generation model generates a text description based on the image feature vector and text feature vector projected into the latent space.

[0103] In this embodiment, the method further includes optimizing the language mapping model, which specifically includes the following steps:

[0104] The test image feature vector and text feature vector output by the fMRI-Image mapping model are used to train the language mapping model again and fine-tune the model to adapt to the characteristics of the test image feature vector and text feature vector output by the fMRI-Image mapping model, so as to avoid slight differences between the test image feature vector and text feature vector output by the fMRI-Image mapping model and the actual image feature vector and text feature vector output by the CLIP model.

[0105] Embodiment 2:

[0106] This embodiment provides a system for generating human brain cognitive text description of natural images, including:

[0107] The first construction and training module is used to pre-construct and train the fMRI-Image mapping model, wherein the fMRI-Image mapping model adopts a CNN network model.

[0108] The second construction and training module is used to construct and train a language mapping model; wherein the language mapping model includes a Transformer network and a text generation model.

[0109] The signal acquisition module is used to acquire fMRI signals generated by the human brain based on stimulation images.

[0110] The signal processing module is used to pre-process the acquired fMRI signals.

[0111] The fMRI-Image mapping model is used to receive each fMRI signal and output an image feature vector and a text feature vector of a stimulus image corresponding to each fMRI signal.

[0112] The language mapping model is used to generate a text description for the stimulus image based on the image feature vector and text feature vector output by the fMRI-Image mapping model.

[0113] The signal collection module is used to collect a signal set including fMRI signals.

[0114] The image acquisition module is used to acquire a stimulation image corresponding to each fMRI signal in the collected signal set.

[0115] The text-image comparison pre-trained model is used to extract the actual image feature vector and the text feature vector for each stimulus image obtained.

[0116] Embodiment 3:

[0117] This embodiment provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for generating a human brain cognitive text description of a natural image described in Example 1.

[0118] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, it should be noted that the flowchart in the accompanying drawings shows the method of the embodiment of the present disclosure. In the corresponding description in the flowchart or block diagram in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be performed substantially in parallel, and they can sometimes be performed in the opposite order, which can depend on the functions involved. Each block in the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating human brain cognitive text description of natural images, characterized in that: The following steps are involved: Step 1: Pre-build and train the fMRI-Image mapping model; Step 2: Pre-build and train the language mapping model; Step 3, obtaining fMRI signals generated by the human brain based on the stimulus image, and preprocessing the obtained fMRI signals to obtain processed fMRI signals; Step 4: input the preprocessed fMRI signal into the constructed and trained fMRI-Image mapping model to obtain an image feature vector and a text feature vector for describing the stimulus image; Step 5: Input the image feature vector and text feature vector output by the fMRI-Image mapping model into the constructed and trained language mapping model to generate a text description for the stimulus image; The step one comprises: Constructing a CNN network model, and training the CNN network model to obtain an fMRI-Image mapping model; collecting a signal set including fMRI signals, and preprocessing the fMRI signals in the signal set to eliminate useless signals; Acquire a stimulus image corresponding to each fMRI signal in the signal set and a text description matching the stimulus image, and input each acquired stimulus image and text description matching the stimulus image into a text-image comparison pre-training model to obtain an actual image feature vector and an actual text feature vector for each stimulus image; The CNN model is trained based on the actual image feature vector and actual text feature vector of each fMRI signal in the dataset and its corresponding stimulus image to obtain an fMRI-Image mapping model; The training of the CNN model based on each fMRI signal in the data set and the actual image feature vector and text feature vector of the corresponding stimulus image includes: The maximum normalized contrast loss of information in the CNN network model is defined as follows: in, is the mathematical expectation, B is the sample size, and the B value is greater than 1; Respectively The predicted value and true value of samples; The value of is from 1 to all values ​​in B. For the The true value of samples; is a non-negative hyperparameter selected by parameter search; Based on the defined maximum normalized contrast loss, for the image feature vector of each stimulus image, the CNN network model uses the following loss function for loss calculation during training: in, represents the Euclidean distance, represents the cosine similarity, Indicates The actual image feature vector of image samples, It is the predicted image feature vector output by the CNN network model; , , is a non-negative hyperparameter selected by parameter search; Based on the defined maximum normalized contrast loss, for the text feature vector of each stimulus image, the CNN network model uses the following loss function for loss calculation during training: in, Indicates The actual text feature vector of the text samples, It is the predicted text feature vector output by the CNN network model.

2. The method for generating human brain cognitive text description of natural images according to claim 1, characterized in that: The step five comprises: The Transformer network projects the image feature vector and text feature vector output by the fMRI-Image mapping model into the latent space of the text generation model; The text generation model generates text descriptions based on the image feature vector and text feature vector projected into the latent space.

3. A system for generating human brain cognitive text descriptions of natural images, characterized in that: The system comprises: a first construction and training module, which is used to pre-construct and train an fMRI-Image mapping model, wherein the fMRI-Image mapping model adopts a CNN network model; The second construction and training module is used to construct and train a language mapping model; wherein the language mapping model includes a Transformer network and a text generation model; A signal acquisition module, used for acquiring fMRI signals generated by the human brain based on stimulus images; A signal processing module, used for preprocessing the acquired fMRI signals; The fMRI-Image mapping model is used to receive each fMRI signal and output an image feature vector and a text feature vector of a stimulus image corresponding to each fMRI signal; A language mapping model, used to generate a text description for a stimulus image based on the image feature vector and text feature vector output by the fMRI-Image mapping model; The system further comprises: A signal collection module, used for collecting a signal set including fMRI signals; An image acquisition module, used for acquiring a stimulus image corresponding to each fMRI signal in the collected signal set; A text-image comparison pre-training model is used to extract the actual image feature vector and the text feature vector for each stimulus image obtained; The first construction and training module is also used to: train the CNN model based on the actual image feature vector and text feature vector of each fMRI signal in the data set and its corresponding stimulus image; The training of the CNN model based on each fMRI signal in the data set and the actual image feature vector and text feature vector of the corresponding stimulus image includes: The maximum normalized contrast loss of information in the CNN network model is defined as follows: in, is the mathematical expectation, B is the sample size, and the B value is greater than 1; Respectively The predicted value and true value of samples; The value of is from 1 to all values ​​in B. For the The true value of samples; is a non-negative hyperparameter selected by parameter search; Based on the defined maximum normalized contrast loss, for the image feature vector of each stimulus image, the CNN network model uses the following loss function for loss calculation during training: in, represents the Euclidean distance, represents the cosine similarity, Indicates The actual image feature vector of image samples, It is the predicted image feature vector output by the CNN network model; , , is a non-negative hyperparameter selected by parameter search; Based on the defined maximum normalized contrast loss, for the text feature vector of each stimulus image, the CNN network model uses the following loss function for loss calculation during training: in, Indicates The actual text feature vector of the text samples, It is the predicted text feature vector output by the CNN network model.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the method for generating a human brain cognitive text description of a natural image according to any one of claims 1 to 2.

Images