A method for modeling a brain cognitive system based on a deep neural network

Abstract

The invention discloses a method for modeling a brain cognitive system based on a deep neural network, belonging to the fields of artificial intelligence and neurobiology. Aiming at the deficiencies of the background and existing research results, this paper proposes a method for modeling a cognitive system based on a deep neural network, aiming to solve the problems that the existing technology models do not conform to the operation mechanism of the biological brain cognitive system, the modeling does not consider the high-dimensional information in the cognitive system, how to train the model using the temporal data set of cognitive tasks containing image information, and there is still a large gap between the model representation inferred by the existing technology and the biological brain representation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and neurobiology, and specifically relates to a cognitive system modeling method based on deep neural network. Background Art

[0002] Due to the limitations of measurement technology, it is difficult to directly detect the activity of neurons inside the biological brain. In recent years, thanks to the rapid development of deep neural networks, more and more researchers have used it as the main means of modeling the biological brain, and studied the operating mechanism of the biological brain by measuring the neuronal activity of deep neural networks. Among them, convolutional neural networks (CNNs) are often used to model the primary visual cortex of the biological brain to perform visual tasks. In addition to modeling the visual cortex, another important challenge of using deep neural networks to model the biological brain is how to model the cognitive system of the biological brain.

[0003] Unlike the image vision task of the visual cortex, the cognitive systems in the biological brain, such as working memory, are carried out in the time dimension, which makes modeling more difficult. Researchers usually use empirical data or behavioral data to infer the representation of the corresponding brain areas of the biological brain, but there is still a large gap between the representation patterns inferred by the above methods and the actual representation of the brain areas. Recently, some studies have used recurrent neural networks (RNNs) that are good at dealing with time series problems to model cognitive systems. However, most models are based on the assumption that there is a low-dimensional dynamic system in the model, which simplifies the modeling and does not consider the high-dimensional information in the cognitive system. In the study of the biological brain, it was found that the primary visual cortex and the advanced frontal and parietal cortex are all involved in cognitive behavior. Therefore, the modeling of the cognitive system should include the high-dimensional information of the visual cortex, which is not considered by the existing models. Therefore, how to build a deep neural network model that is more in line with the operating mechanism of the biological brain cognitive system, how to train with a cognitive task time series data set containing image information, and how to improve the similarity between the model representation and the biological brain representation are still urgent problems to be solved. Summary of the invention

[0004] In view of the shortcomings of the background and existing research results, this paper proposes a cognitive system modeling method based on deep neural networks, aiming to solve the problems that the existing technology models do not conform to the operating mechanism of the biological brain cognitive system, the modeling does not take into account the high-dimensional information in the cognitive system, how to use the cognitive task time series data set containing image information to train the model, and the large gap between the model representation inferred by the existing technology and the biological brain representation.

[0005] The technical solution of the invention is described as follows: A method for modeling a brain cognitive system based on a deep neural network, the method comprising:

[0006] Step 1: Build a cognitive system model;

[0007] Pre-train a complete CNNs using the ImageNet dataset. Specifically, the CNNs adopt the AlexNet model architecture; after completion, discard the fully connected layer, and sequentially add a flattening layer and a recurrent layer based on a continuous-time dynamical system after the convolutional layer to form a cognitive system model; where the output of the convolutional layer is the matrix C m×m , and the flattening layer is used to flatten C m×m into a one-dimensional vector for input to the recurrent layer;

[0008] The dynamical equation of the above-mentioned recurrent layer based on a continuous-time dynamical system is: The output equation of the recurrent layer is: where τ is the time constant, r(t) is the activation value of the neuron at time t, x(t) is the input of the recurrent layer, b r is the bias term, y(t) is the output of the recurrent layer, f(·) is the ReLu activation function, W r , W x , w fb , are the connection matrices of the corresponding variables respectively;

[0009] The connection matrices W r , W x , w fb ,

[0010] The recurrent layer consists of an input layer, a hidden layer and an output layer; where the number of neurons in the input layer is m×m + 1, and m×m of these neurons are used to receive the one-dimensional vector input by the flattening layer; the last neuron L is used to receive the status instruction. When L receives 2, it means the model is in the response stage of the cognitive task. When it receives 1, it means the model is in the stimulus display stage of the cognitive task. When it receives 0, it means it is in the interval stage of the cognitive task; the number of neurons in the hidden layer is u more than the number of neurons in the input layer, defined as i = m×m + 1 + u neurons; the number of neurons in the output layer is 3, and 3 labels are output, defined as 1, 0, -1; 1 means the output is a match in the response stage of the cognitive task, -1 means the output is a mismatch in the response stage of the cognitive task, and 0 means the output label in the non-response stage of the cognitive task. The non-response stage means the stimulus display stage and the interval stage;

[0011] Step 2: Train the model and extract the activation values of the model;

[0012] Abstractly generalize the cognitive task experimental paradigm adopted in biological experiments for training the cognitive system model;

[0013] Download the cognitive task stimulus image set and the corresponding neural response data set; among them, the image set is an image set containing N types. For one type n among them, randomly select 3 images, defined as stimulus 1, stimulus 2, and stimulus 3; during training, the inputs to the model are in turn:

[0014] Stimulus 1 is input into the convolutional layer of the cognitive system model, input t1 / τ times in turn, and at the same time, neuron L receives 1 each time it is input. This stage is the stimulus display stage, and t1 is the duration of this stage in the cognitive task experimental paradigm;

[0015] The value 0 is input into the convolutional layer of the cognitive system model, input t2 / τ times in turn, and at the same time, neuron L receives 0 each time it is input. This stage is the interval stage, and t2 is the duration of this stage in the cognitive task experimental paradigm;

[0016] Stimulus 2 is input into the convolutional layer of the cognitive system model, input t3 / τ times in turn, and at the same time, neuron L receives 1 each time it is input. This stage is the stimulus display stage, and t3 is the duration of this stage in the cognitive task experimental paradigm;

[0017] The value 0 is input into the convolutional layer of the cognitive system model, input t4 / τ times in turn, and at the same time, neuron L receives 0 each time it is input. This stage is the interval stage, and t4 is the duration of this stage in the cognitive task experimental paradigm;

[0018] Stimulus 3 is input into the convolutional layer of the cognitive system model, input t5 / τ times in turn, and at the same time, neuron L receives 1 each time it is input. This stage is the stimulus display stage, and t5 is the duration of this stage in the cognitive task experimental paradigm;

[0019] The value 0 is input into the convolutional layer of the cognitive system model, input 1 time, and at the same time, neuron L receives 2. This stage is the reaction stage;

[0020] In the above input process:

[0021] If stimulus 3 and stimulus 1 are the same, the model should output label 1 in the reaction stage, indicating a match; if stimulus 3 and stimulus 1 are different, the model should output label -1 in the reaction stage, indicating a mismatch; in the non-reaction stage, the model should output label 0;

[0022] For each of the N types of image sets, perform the above random extraction d times to form d above tasks for training the cognitive system model; during training, freeze the weights of the convolutional layer and only train the weights of the recurrent layer;

[0023] During training, set the batch size and the number of iterations; use the following formula method to iteratively update the recurrent layer. The formula is: where θ k represents the set of parameters of the recurrent layer at the k-th iteration, I r represents the network learning rate, and J(θ k ) represents the loss function; the network learning rate controls the amplitude of parameter update in each iteration. An overly high or low learning rate will reduce the training effect of the model. Decay the learning rate to improve the network training efficiency. The decay formula for the network learning rate I r is

[0024] After the iterative training is completed, for a certain type n, input the tasks of the same stimuli 1, stimuli 2, and stimuli 3 received by the biological subjects in the biological experiment into the cognitive system model, which is defined as the q-th task of type n, and extract the activation values of the corresponding recurrent layer neurons t is the duration of a single task. Specifically, t = t1 + t2 + t3 + t4 + t5;

[0025] Step 3: Calculate the similarity between the cognitive system model and the biological brain;

[0026] Calculate the mean of the activation values of a total of Q tasks with the same stimuli as the biological subjects as the overall activation value of type n Calculate the representational dissimilarity matrix of the cognitive system model, denoted as R N×N , where the element Pearson(·) is the Pearson correlation coefficient; according to the downloaded task stimulus image set and the corresponding neural response data set, calculate the RDM of the biological brain, denoted as S N ×N , where the element respectively represent the biological brain neural response matrices corresponding to type n1 and type n2; j is the dimension of the biological brain neural response matrix;

[0027] where the matrices R N×N and Q N×N are both symmetric matrices with diagonals being 0; expand the lower triangular elements of R N×N and Q N×N into one-dimensional vectors, calculate the Pearson correlation coefficient and the Spearman correlation coefficient of the two vectors, and obtain the mean of these two correlation coefficients to get the final similarity.

[0028] Finally, select the highest similarity as the similarity of the cognitive system model for the corresponding brain region, and record the cognitive system model obtained by the initialization method when the similarity is the highest for comparison with the biological cognitive system. Description of the Drawings

[0029] Figure 1 This is a flowchart of a cognitive system modeling method based on a deep neural network. Specific implementation

[0030] Step 1: Build a cognitive system model

[0031] Use Python to build a complete AlexNet model and pre-train it using the ImageNet dataset. After completion, discard the fully connected layer, and sequentially add a flattening layer and a recurrent layer based on a continuous-time dynamical system after the convolutional layer to form a cognitive system model. The output of the convolutional layer is the matrix C 6×6 , and the flattening layer is used to flatten C 6×6 into a one-dimensional vector for input to the recurrent layer.

[0032] The dynamical equation of the above recurrent layer based on a continuous-time dynamical system is as follows: The output equation of the recurrent layer is: where τ is the time constant, taken as 100 ms, r(t) is the activation value of the neuron at time t, x(t) is the input to the recurrent layer, b r is the bias term, y(t) is the output of the recurrent layer, f(·) is the ReLu activation function, W r , W x , w fb , are the connection matrices for the corresponding variables respectively.

[0033] Initialize the connection matrices W r , W x , w fb ,

[0034] The recurrent layer consists of an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is 6×6 + 1, where 6×6 neurons are used to receive the one-dimensional vector input from the flattening layer. The last 1 neuron L is used to receive the status instruction. When L receives 2, it means the model is in the response stage of the cognitive task. When it receives 1, it means the model is in the stimulus presentation stage of the cognitive task. When it receives 0, it means it is in the interval stage of the cognitive task. The number of neurons in the hidden layer is 27 more than that in the input layer, and there are 64 neurons in the hidden layer. The number of neurons in the output layer is 3, and 3 labels are output, defined as 1, 0, -1. 1 means the output is a match in the response stage of the cognitive task, -1 means the output is a mismatch in the response stage of the cognitive task, and 0 means the output label in the non-response stage (stimulus presentation stage and interval stage) of the cognitive task.

[0035] Step 2: Train the model and extract the model activation values

[0036] Abstract and generalize the cognitive task experimental paradigm adopted in biological experiments for training the cognitive system model. Download the cognitive task stimulus image set and the corresponding neural response data set in biological experiments, specifically the working memory public data set in the Human Connectome Project (HCP). The image set contains 4 types of image sets, namely the body image set (97 images), the face image set (102 images), the scene image set (98 images), and the tool image set (103 images). For one of the types, randomly select 3 images and define them as Stimulus 1, Stimulus 2, and Stimulus 3. During training, the inputs to the model are as follows:

[0037] Stimulus 1 is input into the convolutional layer of the cognitive system model and input times. At the same time, neuron L receives 1 each time it is input. This stage is the stimulus presentation stage, and 2000 ms is the duration of this stage in the cognitive task experimental paradigm;

[0038] The value 0 is input into the convolutional layer of the cognitive system model and input times. At the same time, neuron L receives 0 each time it is input. This stage is the interval stage, and 500 ms is the duration of this stage in the cognitive task experimental paradigm;

[0039] Stimulus 2 is input into the convolutional layer of the cognitive system model and input times. At the same time, neuron L receives 1 each time it is input. This stage is the stimulus presentation stage, and 2000 ms is the duration of this stage in the cognitive task experimental paradigm;

[0040] The value 0 is input into the convolutional layer of the cognitive system model and input times. At the same time, neuron L receives 0 each time it is input. This stage is the interval stage, and 500 ms is the duration of this stage in the cognitive task experimental paradigm;

[0041] Stimulus 3 is input into the convolutional layer of the cognitive system model and input times. At the same time, neuron L receives 1 each time it is input. This stage is the stimulus presentation stage, and 2000 ms is the duration of this stage in the cognitive task experimental paradigm;

[0042] The value 0 is input into the convolutional layer of the cognitive system model once, and at the same time neuron L receives 2. This stage is the response stage;

[0043] In the above input process, the corresponding output of the model should be:

[0044] If stimulus 3 is the same as stimulus 1, the model should output label 1 during the model response stage, indicating a match; if stimulus 3 is different from stimulus 1, the model should output label -1 during the model response stage, indicating a mismatch; during the non-response stage (stimulus presentation stage and interval stage), the model should output label 0.

[0045] For each of the 4 types of image sets, the above-mentioned random sampling is performed 2000 times to form 2000 above-mentioned tasks for training the cognitive system model. During training, the weights of the convolutional layer are frozen, and only the weights of the recurrent layer are trained.

[0046] During training, the batch size is 20 and the number of iterations is 200. The gradient descent algorithm is used to iteratively update the recurrent layer. The formula of the gradient descent algorithm is: where θ t represents the set of parameters of the recurrent layer at the k-th iteration, I r represents the network learning rate, and J(θ) represents the loss function. The network learning rate controls the amplitude of parameter update in each iteration. Too high or too low a learning rate will reduce the training effect of the model. In this method, the learning rate is decayed during the iterative training process to improve the network training efficiency. The decay formula of the network learning rate I r is

[0047] After the iterative training is completed, for a type n, the tasks with the same stimuli 1, 2, and 3 as those received by the biological subjects in the biological experiment are input into the cognitive system model, defined as the q-th task of type n, and the activation values of the corresponding recurrent layer neurons are extracted.

[0048] Step 3: Calculate the similarity between the cognitive system model and the biological brain

[0049] Calculate the mean of the activation values of a total of 32 tasks with the same stimuli as the biological subjects as the overall activation value of type n. Calculate the RDM of the cognitive system model, denoted as R 4×4 , where the elements According to the downloaded cognitive task stimulus image set of the biological experiment and the corresponding neural response data set, calculate the RDM of the biological brain, denoted as S 4×4 , where the elements represent the biological brain neural response matrices corresponding to type n1 and type n2 respectively. j is the dimension of the biological brain neural response matrix.

[0050] where the matrix R 4×4 and Q 4×4 are both symmetric matrices with a diagonal of 0. Let R 4×4 and Q 4×4The lower triangular elements are expanded into a one-dimensional vector, and the Pearson correlation coefficient and Spearman correlation coefficient of the two vectors are calculated, and the mean of the two correlation coefficients is obtained to get the final similarity.

[0051] Finally, the highest similarity is selected as the similarity of the cognitive system model for the corresponding brain region, and the cognitive system model obtained by recording the initialization method at the time of the highest similarity is compared with the biological cognitive system.

[0052] The experimental results of different traditional methods and this method are further compared, and the experimental results show that this method has the highest similarity with the biological brain.

[0053] Table 1 shows the similarity results of this method compared with different traditional methods.

[0054]

Claims

1. A method for modeling a brain cognitive system based on a deep neural network, the method comprising: Step 1: Build a cognitive system model; Pre-train a complete CNNs using the ImageNet dataset, and the CNNs adopt the AlexNet model architecture; After completion, discard the fully connected layer, and sequentially add a flattening layer and a recurrent layer based on continuous-time dynamical systems after the convolutional layer to form a cognitive system model; where the output of the convolutional layer is the matrix C m×m , and the flattening layer is used to flatten C m×m into a one-dimensional vector for input to the recurrent layer; The dynamic equation of the recurrent layer based on the continuous-time dynamic system is as follows: The output equation of the recurrent layer is: where τ is the time constant, r(t) is the activation value of the neuron at time t, x(t) is the input of the recurrent layer, b r is the bias term, y(t) is the output of the recurrent layer, f(·) is the ReLu activation function, W r , W x , w fb , are the connection matrices of the corresponding variables respectively; Initialize the connection matrix W using the all-zero initialization, uniform random initialization, or normal distribution initialization methods respectively r , W x , w fb , The recurrent layer consists of an input layer, a hidden layer, and an output layer; where the number of neurons in the input layer is m×m + 1, and m×m of these neurons are used to receive the one-dimensional vector input by the flattening layer; the last neuron L is used to receive the status instruction. When L receives 2, it means the model is in the response stage of the cognitive task. When it receives 1, it means the model is in the stimulus display stage of the cognitive task. When it receives 0, it means it is in the interval stage of the cognitive task; the number of neurons in the hidden layer is u more than the number of neurons in the input layer, defined as i = m×m + 1 + u neurons; the number of neurons in the output layer is 3, and 3 labels are output, defined as 1, 0, -1; 1 means the output is a match in the response stage of the cognitive task, -1 means the output is a mismatch in the response stage of the cognitive task, and 0 means the output label in the non-response stage of the cognitive task. The non-response stage means the stimulus display stage and the interval stage; Step 2: Train the model and extract the model activation values; Abstract and generalize the cognitive task experimental paradigm adopted in biological experiments for training the cognitive system model; Download the cognitive task stimulus image set and the corresponding neural response data set; where the image set is an image set containing N types. For one type n among them, randomly select 3 images, defined as stimulus 1, stimulus 2, and stimulus 3; during training, the inputs to the model are in turn: Stimulus 1 is input into the convolutional layer of the cognitive system model, input t1 / τ times in turn, and at the same time, neuron L receives 1 each time it is input. This stage is the stimulus display stage, and t1 is the duration of this stage in the cognitive task experimental paradigm; The value 0 is input into the convolutional layer of the cognitive system model, input t2 / τ times in turn, and at the same time, neuron L receives 0 each time it is input. This stage is the interval stage, and t2 is the duration of this stage in the cognitive task experimental paradigm; Stimulus 2 is input into the convolutional layer of the cognitive system model, input t3 / τ times in turn, and at the same time, neuron L receives 1 each time it is input. This stage is the stimulus display stage, and t3 is the duration of this stage in the cognitive task experimental paradigm; The value 0 is input into the convolutional layer of the cognitive system model, input t4 / τ times in turn, and at the same time, neuron L receives 0 each time it is input. This stage is the interval stage, and t4 is the duration of this stage in the cognitive task experimental paradigm; Stimulus 3 is input into the convolutional layer of the cognitive system model, input t5 / τ times in turn, and at the same time, neuron L receives 1 each time it is input. This stage is the stimulus display stage, and t5 is the duration of this stage in the cognitive task experimental paradigm; The value 0 is input into the convolutional layer of the cognitive system model, input 1 time, and at the same time, neuron L receives 2. This stage is the response stage; In the above input process: If stimulus 3 and stimulus 1 are the same, the model should output the label 1 in the response stage, indicating a match; if stimulus 3 and stimulus 1 are different, the model should output the label -1 in the response stage, indicating a mismatch; in the non-response stage, the model should output the label 0; For each of the N types of image sets, the above-mentioned random sampling is performed d times to form d above-mentioned tasks for training the cognitive system model; during training, the weights of the convolutional layer are frozen, and only the weights of the recurrent layer are trained; During training, set the batch size and the number of iterations; use the following formula method to iteratively update the recurrent layer, and the formula is: where θ k represents the set of parameters of the recurrent layer at the k-th iteration, I r represents the network learning rate, and J(θ k ) represents the loss function; the network learning rate controls the amplitude of parameter update in each iteration. Too high or too low a learning rate will reduce the training effect of the model. Decay the learning rate to improve the network training efficiency. The decay formula for the network learning rate I r is After the iterative training is completed, for a certain type n, the tasks of the same stimuli 1, stimuli 2, and stimuli 3 received by the biological subjects in the biological experiment are input into the cognitive system model, defined as the q-th task of type n, and the activation values of the corresponding recurrent layer neurons are extracted. t is the duration of a single task, and t = t1 + t2 + t3 + t4 + t5; Step 3: Calculate the similarity between the cognitive system model and the biological brain; Calculate the mean of the total Q task activation values of the same stimulus as the biological subject as the overall activation value of type n Calculate the representation dissimilarity matrix of the cognitive system model, denoted as R N×N , where the elements n1, n2 ∈ [1, N], Pearson(·) is the Pearson correlation coefficient; according to the downloaded task stimulus image set and the corresponding neural response data set, calculate the RDM of the biological brain, denoted as S N×N , where the elements respectively represent the biological brain neural response matrices corresponding to type n1 and type n2; j is the dimension of the biological brain neural response matrix; Among them, matrix R N×N and S N×N are both symmetric matrices with diagonals being 0; expand the lower triangular elements of R N×N and S N×N into one-dimensional vectors, calculate the Pearson correlation coefficient and Spearman correlation coefficient of the two vectors, and obtain the mean value of these two correlation coefficients to get the final similarity.

Images