Construction and analysis method and system based on feature interpretable visual coding model

Abstract

The invention discloses a construction and analysis method and system based on a feature interpretable visual coding model. The method comprises the steps of 1, constructing a visual language shared feature space according to image-text pair public data; 2, constructing a visual coding model according to the visual language shared feature space; 3, giving image stimulation to obtain a visual cortex response corresponding to the voxel, and inputting the image and the visual cortex response corresponding to the voxel into the visual coding model to obtain a natural language set of a visual cortex brain region corresponding to the voxel; and 4, performing semantic function analysis on the natural language set of the visual corticobrain region corresponding to the voxel to obtain semantic function characteristics of the visual corticobrain region corresponding to the voxel. According to the method, explainability of the visual coding model in the feature level is realized through the visual language sharing feature space and the visual coding model, and further technical support is provided for brain-computer interaction application research based on visual function analysis.

Description

Technical Field

[0001] The present invention relates to the field of visual information processing technology, and in particular to a method and system for constructing and analyzing a feature-based interpretable visual coding model. Background Art

[0002] The study of human visual information analysis aims to explore the information processing mechanism of the visual cortex of the brain on the content of visual scenes, analyze the representation mode and characteristics of the visual cortex neural activity on the content of visual scenes, and analyze the visual scene content in the visual cortex neural activity. It is a very important topic in the study of human visual function. Functional Magnetic Resonance Imaging (fMRI) provides a high spatial resolution, good reliability, and non-invasive neural activity monitoring technology for the study of brain visual cortex function, and has become an important tool for the study of human visual function.

[0003] Natural image scenes are complex and the types of objects are diverse. The fMRI visual information analysis of natural complex scenes is a cutting-edge and difficult problem, which usually includes two directions of research: visual encoding and decoding. Figure 1 As shown in the figure, the visual encoding direction studies how to build a computational model to simulate the information processing process of the brain's visual cortex and predict the fMRI voxel response of the subject's visual cortex under natural image stimulation, that is, "brain simulation". The visual decoding direction studies how to build a computational model to parse the content information of the natural image seen by the subject from the fMRI voxel response of the brain's visual cortex, that is, "brain reading". Among them, visual information encoding is a basic problem in visual information analysis.

[0004] The research on interpretable visual coding technology based on fMRI mainly constructs a computational model of the neural activity of the brain's visual cortex based on the fMRI data (output) of the visual cortex response obtained when humans view image stimuli (input). The focus is on obtaining interpretable feature information from it, revealing how visual cortical voxels and brain areas form their functional characteristics through information interaction and processing, and exploring and verifying the information processing mechanism of the visual cortex.

[0005] Deep neural networks are inspired by biological neural information processing mechanisms and have very efficient data representation capabilities. In recent years, research on visual information encoding based on deep networks has achieved a series of breakthrough results, greatly improving the encoding accuracy of the model and explaining the view that the hierarchical information processing structure of deep networks and visual cortex is relatively similar. At present, visual encoding models based on deep networks have been widely studied and focused in the field.

[0006] Current research has been answering the question "What are the functional characteristics of visual cortical voxels and brain regions?" mainly from a macroscopic level. By comparing and analyzing the correlation between deep network features and visual cortical activation, the hierarchical nature of visual cortical feature representation and feature receptive fields have been verified. However, more in-depth research is still needed on the detailed explanation of the functional characteristics of visual cortical voxels and brain regions. Summary of the invention

[0007] In order to at least partially solve the problem of insufficient interpretability of existing deep network features in interpretable visual coding technology, the present invention provides a method and system for constructing and analyzing a feature-based interpretable visual coding model. First, a joint representation model is constructed through paired data of images and natural language vocabulary descriptions (texts) to obtain a visual language shared feature space, which connects the image space and the natural language space; then, the feature is used as an intermediate bridge to construct a visual coding model, and a coding mapping from the visual language shared feature space to the visual area voxel space is constructed. Finally, the mapping correlation between the feature and the language space vocabulary description is used to interpret the semantic representation characteristics of the voxel using the statistical description of the language vocabulary, and further analyze the functional characteristics of the visual cortex brain area.

[0008] In order to achieve the above object, the technical solution of the present invention is:

[0009] The first aspect of the present invention proposes a method for constructing and analyzing a feature-based interpretable visual encoding model, comprising:

[0010] Step 1: Construct a shared feature space of visual language based on public data of image-text pairs to link images and texts;

[0011] Step 2: Construct a visual encoding model based on the shared feature space of visual language;

[0012] Step 3: Given an image stimulus, obtain the visual cortical response corresponding to the voxel, input the image and the visual cortical response corresponding to the voxel into the visual coding model, and obtain the natural language set of the visual cortical brain area corresponding to the voxel, so as to facilitate the matching of the voxel and the natural language;

[0013] Step 4: Perform semantic function analysis on the natural language set of the visual cortex brain area corresponding to the voxels to obtain the semantic function characteristics of the visual cortex brain area corresponding to the voxels.

[0014] Furthermore, the visual language shared feature space includes a plurality of feature vectors, each feature vector corresponds to a word in a natural language, so that the image and the natural language can be linked together through the feature vectors.

[0015] Furthermore, constructing a visual encoding model according to the shared feature space of the visual language specifically includes:

[0016] Construct a linear coding model and initialize the weight parameters in the linear coding model to facilitate model construction;

[0017] Calculate the average vector of all feature vectors under each image stimulus, and use the average vector as the feature vector matching the voxel in the linear encoding model;

[0018] The weight parameters of the feature vector are obtained according to the linear coding model and gradient descent method, which is convenient for model training;

[0019] The error is calculated according to the weight parameter and the specific linear coding model, and the feature vector with the smallest error is selected as the feature vector matching the visual cortical response to complete one iteration;

[0020] Perform multiple iterations to obtain the final weight parameter and the matrix H consisting of the eigenvectors matching the visual cortical response corresponding to the voxel j , complete the matching of voxels and feature vectors, and obtain the natural language set of the visual cortex brain area corresponding to the voxels.

[0021] Furthermore, the linear coding model is expressed according to the following formula:

[0022] y j =H j w+σ

[0023] Among them, y j is the voxel v j The visual cortex response to all image stimuli, H j is the voxel v j All related eigenvectors, w is the weight parameter, and σ is the error.

[0024] Further, the calculating the error according to the weight parameter specifically includes:

[0025] According to the weight parameter, the feature vectors under each image stimulus are brought into the specific linear coding model to calculate the error. The specific linear coding model is expressed according to the following formula:

[0026]

[0027] in, is the visual cortical response to the ith image stimulus, is the feature vector corresponding to the mth semantic word in the ith image, w is the weight parameter, and σ is the error.

[0028] Further, the semantic function analysis of the natural language set of voxels and the visual cortex brain area corresponding to the voxels specifically includes:

[0029] Construct a language vocabulary set to visualize each voxel of the visual cortex and perform interpretable analysis of the functional properties of the voxels;

[0030] We weighted each voxel, constructed a language vocabulary set for each brain region of the visual cortex, and analyzed the semantic functional characteristics of each brain region of the visual cortex.

[0031] Based on the spatial distance between voxels and the semantic functional characteristics, sub-regions with different functional characteristics in each visual brain area are obtained, thus achieving a refined interpretation of the semantic functional characteristics of the visual brain area.

[0032] The second aspect of the present invention proposes a feature-based interpretable visual encoding model construction and analysis system, comprising:

[0033] The visual language shared feature space module is used to build a visual language shared feature space based on public image-text pair data to link images and texts;

[0034] Visual encoding model module, used to build a visual encoding model based on the shared feature space of visual language;

[0035] A matching module is used to obtain the visual cortex response corresponding to the voxel given an image stimulus, input the image and the visual cortex response corresponding to the voxel into the visual coding model, and obtain the natural language set of the visual cortex brain area corresponding to the voxel, so as to facilitate the matching of the voxel and the natural language;

[0036] The analysis module is used to perform semantic function analysis on the natural language set of the visual cortex brain area corresponding to the voxels, so as to obtain the semantic function characteristics of the visual cortex brain area corresponding to the voxels.

[0037] The third aspect of the present invention proposes an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a method for constructing and analyzing a feature-based interpretable visual coding model as described in the first aspect above.

[0038] The fourth aspect of the present invention proposes a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for constructing and analyzing a feature-based interpretable visual coding model as described in the first aspect above.

[0039] Beneficial effects of the present invention:

[0040] (1) The present invention proposes a method for constructing and analyzing a feature-interpretable visual coding model. It obtains more interpretable deep network features through human semantic vocabulary, and innovatively generates a "language vocabulary feature map" to depict the semantic functional characteristics of visual cortical voxels and brain regions, thereby achieving interpretability of the coding model at the feature level. The interpretability of the model is enhanced, providing further technical support for brain-computer interaction application research based on visual function analysis.

[0041] (2) The present invention constructs a visual language shared feature space and a visual coding model to calculate the association between the visual cortical response corresponding to the voxel and the feature vector in the visual language shared feature space, and matches the visual cortical response corresponding to the voxel with the feature vector based on the association between the visual cortical response corresponding to the voxel and the feature vector in the visual language shared feature space, thereby finally obtaining a matching relationship between the voxel and the natural language describing the image, which facilitates a detailed explanation of the semantic functional characteristics of the visual cortical voxels and brain regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of two research directions in fMRI-based visual information analysis provided in an embodiment of the present invention.

[0043] Figure 2 A flowchart of a method for constructing and analyzing a feature-interpretable visual coding model provided in an embodiment of the present invention.

[0044] Figure 3 A schematic diagram of a visual language shared feature space provided by an embodiment of the present invention.

[0045] Figure 4 A schematic diagram of a method for constructing and analyzing a feature-interpretable visual encoding model provided in an embodiment of the present invention.

[0046] Figure 5 An architecture diagram of a feature-based interpretable visual coding model construction and analysis system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Example 1

[0049] like Figure 2As shown, a method for constructing and analyzing a feature-based interpretable visual encoding model includes the following steps:

[0050] S101: Construct a shared feature space of visual language based on public data of image-text pairs.

[0051] Specifically, Figure 3 As shown in the figure, both images and language can describe scene content to a certain extent. Based on a large amount of public data on image-text pairs, a visual language shared feature space can be constructed to establish the correspondence between images and natural language. Among them, each feature vector corresponds to a word in natural language through a decoder. Compared with abstract visual features, visual language shared features are closely related to language vocabulary. By constructing a visual encoding model based on deep network features, interpretable semantic vocabulary with strong correlation with voxels is obtained to achieve functional analysis of voxel representation characteristics. Pre-training technology is used based on a large number of image-text pair samples to achieve visual language shared representation. Afterwards, driven by the image description task, the visual language shared feature space is adjusted to obtain the final visual language shared feature space, which can predict the corresponding shared features and text descriptions based on the image.

[0052] S102: Construct a visual encoding model based on the shared feature space of visual language.

[0053] S103: Given an image stimulus, obtain a visual cortical response corresponding to the voxel, input the image and the visual cortical response corresponding to the voxel into a visual coding model, and obtain a natural language set of the visual cortical brain area corresponding to the voxel.

[0054] S104: Performing semantic function analysis on the natural language set of the visual cortex brain area corresponding to the voxels to obtain semantic function characteristics of the visual cortex brain area corresponding to the voxels.

[0055] The present invention constructs a shared feature space of visual language, links images and texts through feature vectors, and then constructs a visual coding model to match the visual cortical responses corresponding to voxels with the feature vectors, thereby obtaining the association between the visual cortical responses corresponding to the voxels and the text, thereby improving the interpretability of the model and facilitating the depiction of the semantic functional characteristics of visual cortical voxels and brain regions.

[0056] Example 2

[0057] Based on the above embodiment, the embodiment of the present invention provides a specific process of constructing a visual coding model, which specifically includes:

[0058] In the visual language shared feature space, for the i-th image stimulus s i , and obtain several visual language feature vectors Each feature vector The mth semantic word corresponding to the i-th image.

[0059] For the jth voxel v of the brain image to be analyzed j , in the i-th image stimulus s i The visual cortex response under Assume that the mth eigenvector The corresponding voxel response is predicted by a linear regression model. The specific linear encoding model is shown in formula (1). j To Stimulus i Which feature vector h is the visual cortical response of i The correlation cannot be determined in advance, that is, m is unknown, and for different image stimuli s i and different voxels v j , the sequence number m will also change.

[0060]

[0061] in, is the visual cortical response of the j-th voxel under the stimulation of the i-th image, is the mth semantic word corresponding to the i-th image, w is the weight, and σ is the error.

[0062] y j V j The visual cortical response vector for all image stimuli S, H j V j The matrix composed of all relevant eigenvectors is matched between voxels and eigenvectors through a linear coding model, and the linear coding model is shown in formula (2). Let the total number of image stimuli be k, then before the weight vector w is optimized and trained, k sequence values ​​need to be determined That is, we need to determine k voxels v j The matching feature vector.

[0063] y j =H j w+σ

[0064] Therefore, in the voxel v j Before encoding, the linear weights w and the selection of feature vectors M j Unknown. The feature vector selection not only affects the quality of the encoding, but also determines the subsequent interpretation of the voxel representation characteristics, that is, whether the H j The corresponding semantic word information W j Voxel v jFor the problem of two types of unknown quantities, we borrowed the idea of ​​EM algorithm and optimized the encoding model alternately. Finally, while completing the optimization of the weight parameters of the encoding model, we determined the visual language features of each voxel in each image, laying the foundation for obtaining the corresponding interpretable deep network features in the next step. j , the corresponding encoding algorithm process is as follows:

[0065] S201: Calculate the average vector of all feature vectors under each image stimulus as the vector corresponding to the voxel v j The matching feature vectors, thus H j initialization;

[0066] S202: According to formula (2), the weight w is solved by gradient descent iterative optimization;

[0067] S203: Based on the solved w, substitute the feature vector under each image stimulation into formula (1) to calculate the error σ, and select the feature vector with the smallest error as the feature vector corresponding to the voxel v j The most matching feature vector is determined by traversing all image stimuli. And update H j ;

[0068] S204: Repeat S02 and S203 until the maximum number of iterations L is reached to obtain the final w and H j .

[0069] The feature vector corresponding to the voxel is obtained, and then the natural language describing the image corresponding to the voxel is obtained.

[0070] Example 3

[0071] On the basis of the above embodiments, the embodiment of the present invention provides a specific process of performing semantic function analysis on a natural language set of visual cortical brain areas corresponding to voxels, specifically including:

[0072] In the process of building the encoding model, a set of language words related to each voxel was obtained. Since the language words themselves are directly interpretable, the essence is that the deep network features corresponding to each voxel are interpretable through the language words, reflecting the interpretable role of the joint representation of vision and language. With the help of this very intuitive language word set, the semantic functional characteristics of visual cortical voxels and brain areas can be explained in more detail from several perspectives.

[0073] By visualizing the language vocabulary set of each voxel in the visual cortex, an interpretable analysis of the functional characteristics of the voxels can be achieved. For each voxel, an open source language vocabulary visualization tool (wordcloud) is used to draw the statistical distribution characteristics of the voxel semantic vocabulary set through an image. First, the language vocabulary set is counted. The more times a language vocabulary appears, the larger the space it occupies in the image. Different words are drawn with different colors to distinguish them, so that the statistical distribution of the semantic vocabulary set can be displayed more clearly. This is called a "language vocabulary feature map". By directly analyzing this "language vocabulary feature map", the representational characteristics and functional characteristics of the voxels can be explained. In this way, an interpretive analysis of the functional characteristics of all significant voxels is achieved. In the early stage, basic preliminary experiments related to single voxel visualization analysis based on language vocabulary were carried out on the BOLD-5000 public dataset, which showed good interpretability. As Figure 4 As shown in the figure, for example, from the analysis of words such as next and close, it can be considered that the 138th voxel in the L-LOC visual area of ​​Subject 1 is more inclined to express spatial position relationship; the 20th voxel in the R-RSC visual area of ​​Subject 2 is related to man and people, that is, they are more interested in the representation of people, etc. The visualization of these voxels explains their semantic functional characteristics, which is difficult to obtain with existing encoding models.

[0074] By weighting each voxel, we can obtain the language vocabulary set of each brain area in the visual cortex, and realize the analysis of the semantic functional characteristics of each brain area in the visual cortex. For each visual area, the weight of each voxel is calculated according to the encoding performance of each voxel in the area. The larger the weight, the higher the accuracy of the expression of the voxel. By weighted summarizing the semantic vocabulary set of significant voxels, we can obtain a summary semantic vocabulary set of each visual brain area. By drawing its "language vocabulary feature map", we can realize the description and explanation of the functional characteristics of the visual brain area. At the same time, by comparing and analyzing the "language vocabulary feature map" of different visual areas, we can further analyze the differences in the representation of functional characteristics of each visual brain area, enhance the understanding of the representation characteristics and functional characteristics of the visual brain area, reveal the functional characteristics of the visual brain area, and what role each brain area plays.

[0075] According to the spatial distance and semantic functional characteristics between voxels, voxels with similar functional characteristics in the visual cortex are clustered and merged to obtain sub-regions with different functional characteristics in each visual brain area, so as to achieve a more refined understanding of the semantic functional characteristics of the visual brain area. For a single voxel, the deep network feature set H matching it is calculated. j The average eigenvalue of is used as the eigenvector describing the semantic functional characteristics of each voxel. The similarity of functional characteristics between different voxels is measured by calculating the correlation between the eigenvectors of different voxels.

[0076] On this basis, each voxel is regarded as a node, and the dissimilarity of the functional characteristics of adjacent voxel nodes is used as the edge between nodes according to the spatial position of the visual cortex fMRI. Thus, a graph is constructed according to the correlation between voxels and voxels. The visual brain area is divided into several sub-areas by using a greedy clustering algorithm based on a graph. First, the dissimilarity between each node and its three-dimensional spatial neighboring nodes is calculated, and the edges are arranged in ascending order. The smallest edges are selected from small to large for merging judgment. If the dissimilarity is less than the internal dissimilarity of the two (the largest edge in the sub-area), the two sub-areas are merged (a single voxel is the smallest sub-area unit), and the merging threshold and sub-area label of each area are updated. If the merging condition is not met, the next edge is merged. In this way, each sub-area is finally obtained, and the semantic functional characteristics of each sub-area are similar. The semantic vocabulary set is obtained by weighting the semantic vocabulary set of the voxels in each sub-area, and a more detailed explanation of the functional characteristics of each visual brain sub-area is achieved by drawing its "language vocabulary feature map".

[0077] Example 4

[0078] Corresponding to the above method, such as Figure 5 As shown, an embodiment of the present invention provides a feature-based interpretable visual coding model construction and analysis system, including:

[0079] The visual-linguistic shared feature space module is used to build a visual-linguistic shared feature space based on public image-text pair data.

[0080] The visual encoding model module is used to build a visual encoding model based on the shared feature space of visual language.

[0081] The matching module is used to obtain the visual cortical response corresponding to the voxel given the image stimulus, input the image and the visual cortical response corresponding to the voxel into the visual coding model, and obtain the natural language set of the visual cortical brain area corresponding to the voxel.

[0082] The analysis module is used to perform semantic function analysis on the natural language set of the visual cortex brain area corresponding to the voxels, so as to obtain the semantic function characteristics of the visual cortex brain area corresponding to the voxels.

[0083] It should be noted that the system for building and analyzing a feature-interpretable visual coding model provided in an embodiment of the present invention is for realizing a method for building and analyzing a feature-interpretable visual coding model in the above-mentioned embodiment. Its specific functions can be referred to the above-mentioned method embodiments and will not be repeated here.

[0084] Example 5

[0085] Corresponding to the above method, an embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements a method for constructing and analyzing a feature-based interpretable visual coding model as described in the above embodiment.

[0086] Preferably, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for constructing and analyzing a feature-based interpretable visual coding model as described in the above embodiment.

[0087] In summary, the present invention proposes a method for constructing and analyzing a feature-interpretable visual coding model, obtains more interpretable deep network features through human semantic vocabulary, and innovatively generates a "language vocabulary feature map" to depict the semantic functional characteristics of visual cortical voxels and brain regions, so as to achieve interpretability of the coding model at the feature level, enhance the interpretability of the model, and provide further technical support for brain-computer interaction application research based on visual function analysis. The present invention constructs a visual language shared feature space and a visual coding model, calculates the association between the visual cortical response corresponding to the voxel and the feature vector in the visual language shared feature space, and matches the visual cortical response corresponding to the voxel with the feature vector according to the association between the visual cortical response corresponding to the voxel and the feature vector in the visual language shared feature space, and finally obtains the matching relationship between the voxel and the natural language describing the image, which is convenient for detailed explanation of the semantic functional characteristics of the visual cortical voxels and brain regions.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing and analyzing a feature-based interpretable visual encoding model, characterized in that: include: Step 1: Construct a shared feature space of visual language based on public data of image-text pairs; Step 2: Construct a visual encoding model based on the shared feature space of visual language; Step 3: Given an image stimulus, obtain the visual cortical response corresponding to the voxel, input the image and the visual cortical response corresponding to the voxel into the visual encoding model, and obtain the natural language set of the visual cortical brain area corresponding to the voxel; Step 4: Perform semantic function analysis on the natural language set of the visual cortex brain area corresponding to the voxels to obtain the semantic function characteristics of the visual cortex brain area corresponding to the voxels.

2. A method for constructing and analyzing a feature-based interpretable visual coding model according to claim 1, characterized in that: The visual language shared feature space includes a plurality of feature vectors, each of which corresponds to a word in a natural language.

3. A method for constructing and analyzing a feature-based interpretable visual coding model according to claim 2, characterized in that: The step of constructing a visual encoding model based on a shared feature space of a visual language specifically includes: Construct a linear coding model and initialize the weight parameters in the linear coding model; Calculate the average vector of all feature vectors under each image stimulus, and use the average vector as the feature vector matching the voxel in the linear encoding model; Obtain the weight parameters of the feature vector according to the linear coding model and the gradient descent method; The error is calculated according to the weight parameter and the specific linear coding model, and the feature vector with the smallest error is selected as the feature vector matching the visual cortical response to complete one iteration; Perform multiple iterations to obtain the final weight parameter and the matrix H consisting of the eigenvectors matching the visual cortical response corresponding to the voxel j , complete the matching of voxels and feature vectors, and obtain the natural language set of the visual cortex brain area corresponding to the voxels.

4. A method for constructing and analyzing a feature-based interpretable visual coding model according to claim 3, characterized in that: The linear coding model is expressed according to the following formula: y j =H j w+s Among them, y j is the voxel v j The visual cortex response to all image stimuli, H j is the voxel v j All related eigenvectors, w is the weight parameter, and σ is the error.

5. The method for constructing and analyzing a feature-based interpretable visual coding model according to claim 3, characterized in that: The calculating of the error according to the weight parameter specifically includes: According to the weight parameter, the feature vectors under each image stimulus are brought into the specific linear coding model to calculate the error. The specific linear coding model is expressed according to the following formula: in, is the visual cortical response to the ith image stimulus, is the feature vector corresponding to the mth semantic word in the ith image, w is the weight parameter, and σ is the error.

6. The method for constructing and analyzing a feature-based interpretable visual coding model according to claim 1, characterized in that: The semantic function analysis of the natural language set of voxels and the visual cortex brain area corresponding to the voxels specifically includes: Construct a language vocabulary set to visualize each voxel of the visual cortex and perform interpretable analysis of the functional properties of the voxels; We weighted each voxel, constructed a language vocabulary set for each brain region of the visual cortex, and analyzed the semantic functional characteristics of each brain region of the visual cortex. Based on the spatial distance between voxels and the semantic functional characteristics, sub-regions with different functional characteristics in each visual brain area are obtained, thus achieving a refined interpretation of the semantic functional characteristics of the visual brain area.

7. A feature-based interpretable visual coding model construction and analysis system, characterized in that: include: The visual language shared feature space module is used to construct a visual language shared feature space based on public image-text pair data; Visual encoding model module, used to build a visual encoding model based on the shared feature space of visual language; A matching module is used to obtain the visual cortical response corresponding to the voxel given an image stimulus, and input the image and the visual cortical response corresponding to the voxel into the visual encoding model to obtain a natural language set of the visual cortical brain area corresponding to the voxel; The analysis module is used to perform semantic function analysis on the natural language set of the visual cortex brain area corresponding to the voxels, so as to obtain the semantic function characteristics of the visual cortex brain area corresponding to the voxels.

8. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements a method for constructing and analyzing a feature-based interpretable visual coding model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for constructing and analyzing a feature-based interpretable visual coding model as described in any one of claims 1 to 6.

Images