Method for generating cognitive training material, cognitive training method, apparatus, and medium
Patent Information
- Application Number
- PCT/CN2023/127496
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2023-10-30
- Publication Date
- 2024-11-21
AI Technical Summary
Existing cognitive training methods fail to establish semantic map-assisted positioning between cognitive training materials and brain region responses through magnetic resonance, and cannot accurately and efficiently stimulate brain functional cortical targets that need training.
By obtaining multimedia materials and their semantic information, as well as the characteristics of the fMR signal, we use deep learning models to establish association relationships, generate cognitive training materials, and use semantic maps to accurately locate and stimulate the target brain area.
The generation of individual-based cognitive training materials can accurately stimulate brain functional cortical targets and improve the efficiency and effectiveness of cognitive training.
Smart Images

Figure CN2023127496_21112024_PF_FP_ABST
Abstract
Description
Cognitive training material generation method, cognitive training method, device and medium
[0001] Related applications
[0002] This application claims priority to Chinese patent application number 202310538580.9, filed on May 15, 2023, entitled “Cognitive training material generation method, cognitive training method, device and medium,” the entire text of which is hereby incorporated by reference. Technical Field
[0003] The present application relates to the field of deep learning, and in particular to methods for generating cognitive training materials, cognitive training methods, devices, and media. Background Art
[0004] The human brain responds differently to different things. Magnetic resonance imaging (MRI) is a noninvasive method for measuring brain responses with high spatial resolution. The cerebral cortex can be divided into functional regions, such as the temporal lobe responsible for hearing and the occipital lobe for vision, and this division is shared across individuals. When a subject observes an image, in addition to stimulating the occipital cortex, the image's semantics also significantly stimulates the medial and lateral parietal cortex, the temporal cortex, and the lateral prefrontal cortex. Deep learning can be used to establish a mapping between sensory stimuli and brain MRI response signals, allowing sensory stimuli to be selected based on the cortical targets that require a response. Enhancing neural activity in these targets through cognitive training may be a potential way to improve corresponding sensory functions, such as attention, memory, and thinking.
[0005] Cognitive training is a method designed to enhance cognitive abilities, improve thinking processes, and enhance behavioral performance. It is a systematic cognitive intervention that improves thinking and behavioral performance through training and practicing a range of cognitive skills. Cognitive training can be tailored to individual cognitive abilities, such as enhancing attention, stimulating memory, and improving decision-making and problem-solving. Because cognitive training can improve cognitive abilities, it can theoretically be applied to people of all ages, including children, adolescents, and adults, and has significant implications for learning, work, and daily life.
[0006] In related technologies, cognitive training has not considered brain science but has been approached solely from an application perspective, such as designing a game to prevent Alzheimer's disease. In recent years, there have also been studies using transcranial electrical stimulation to stimulate known functional brain regions. However, the magnitude and effectiveness of the stimulation vary from person to person, and it is not possible to stimulate deep brain regions and implement individualized implementation plans. Therefore, current cognitive training has not yet established a brain semantic map to assist in the localization of cognitive training materials and brain region responses through magnetic resonance imaging. Before cognitive training, there is no measurable basis for whether the cognitive training materials can accurately and efficiently stimulate the functional cortical targets in need of training.
[0007] There is currently no effective solution to the problem in related technologies that it is impossible to obtain cognitive training materials that can accurately stimulate the brain functional cortical targets that need to be trained.
[0008] Summary of the Invention
[0009] According to various embodiments of the present application, a method for generating cognitive training materials, a cognitive training method, an apparatus, and a medium are provided.
[0010] First, in this embodiment, a method for generating cognitive training materials is provided, including:
[0011] Acquire a first feature and a second feature, wherein the first feature includes multimedia material and semantic information corresponding to the multimedia material, the second feature includes a magnetic resonance representation, and the first feature and the second feature are associated with each other;
[0012] Fitting the first feature and the second feature, obtaining a semantic map according to the fitting result and a preset brain map, and acquiring target semantic information corresponding to the target point according to the semantic map;
[0013] Using the first feature as an input of a deep learning model and the second feature as a constraint condition of the deep learning model, training the deep learning model, and determining a weight parameter of the deep learning model when the deep learning model meets a convergence condition;
[0014] Generate cognitive training materials based on the target semantic information and the weight parameters of the deep learning model.
[0015] In some embodiments, generating cognitive training materials includes:
[0016] Inputting a first cognitive training material generated according to the target semantic information and the weight parameters of the deep learning model into the deep learning model, and predicting a second feature corresponding to the first cognitive training material;
[0017] Determining whether a second feature corresponding to the first cognitive training material meets a preset condition;
[0018] The first cognitive training material is screened according to the judgment result to obtain the second cognitive training material.
[0019] In some embodiments, obtaining the first feature includes:
[0020] The features of the multimedia material are extracted through a convolutional neural network, and the semantic information of the multimedia material is encoded, and the features of the multimedia material and the encoded semantic information are used as the first features.
[0021] In some embodiments, obtaining the second feature includes:
[0022] Acquiring a functional magnetic resonance signal corresponding to the first feature, extracting features of the functional magnetic resonance signal, and obtaining a signal feature;
[0023] The signal feature is mapped to the cerebral cortex, and the mapped signal feature is used as the second feature.
[0024] In some embodiments, after using the mapped signal feature as the second feature, the method further includes:
[0025] Acquiring structural magnetic resonance signals and diffusion magnetic resonance signals;
[0026] The structural magnetic resonance signal and the diffusion magnetic resonance signal are used as the second feature.
[0027] In some embodiments, obtaining a semantic map based on the fitting result and a preset brain map includes:
[0028] Mapping the fitting result to the preset brain map, and decoding the mapped fitting result through an autoencoder;
[0029] According to the decoding results, the semantic information that generates the strongest magnetic resonance representation in each target point is selected to generate the semantic map.
[0030] In some embodiments, generating first cognitive training material according to the target semantic information and the weight parameters of the deep learning model includes:
[0031] Obtaining shallow weight parameters and deep weight parameters of the deep learning model;
[0032] First cognitive training material is generated based on the target semantic information as input information, the shallow weight parameters and the deep weight parameters.
[0033] In a second aspect, this embodiment provides a cognitive training method, comprising:
[0034] Obtain the target point that the user needs to perform cognitive training on, and determine the semantic information corresponding to the target point based on the semantic map;
[0035] Acquire cognitive training materials according to the semantic information, wherein the cognitive training materials are obtained based on the cognitive training material generation method described in the first aspect above;
[0036] The cognitive training material is presented to the user according to a preset duration.
[0037] In a third aspect, an electronic device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating cognitive training materials described in the first aspect is implemented.
[0038] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the method for generating cognitive training materials described in the first aspect is implemented.
[0039] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0041] FIG1 is a flow chart of a method for generating cognitive training materials according to an embodiment of the present application.
[0042] FIG2 is a flow chart of a cognitive training method according to an embodiment of the present application.
[0043] FIG3 is a flowchart of a cognitive training method based on magnetic resonance and graph convolution according to an embodiment of the present application.
[0044] FIG4 is a continuous recognition task experimental design paradigm according to an embodiment of the present application.
[0045] FIG5 is a schematic diagram of characteristic beta values of task-state functional magnetic resonance signals according to an embodiment of the present application.
[0046] FIG6 is a schematic diagram of a semantic graph according to an embodiment of the present application.
[0047] FIG. 7 is a schematic diagram of cognitive training materials according to an embodiment of the present application. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0050] In this embodiment, a method for generating cognitive training materials is provided. FIG1 is a flow chart of the method for generating cognitive training materials in this embodiment. As shown in FIG1 , the process includes the following steps:
[0051] Step S101 : obtaining a first feature and a second feature, wherein the first feature includes multimedia material and semantic information corresponding to the multimedia material, the second feature includes magnetic resonance representation, and the first feature and the second feature are associated with each other.
[0052] Multimedia materials include natural stimuli such as pictures, videos, and sounds that can produce visual, auditory, and other stimulations on the subjects, and the semantic information of the multimedia materials includes textual descriptions of the natural stimuli. Optionally, the features of the multimedia materials and the features of the semantic information of the multimedia materials are used as the first features.
[0053] The association between the first feature and the second feature includes: providing the first feature to the subject, and generating the second feature when the subject recognizes the first feature. Obtaining the second feature includes: obtaining the magnetic resonance signal generated by the subject during the process of identifying the natural stimulus. Optionally, the subject recognizes a series of natural stimuli, including visual and auditory stimuli, in the magnetic resonance imaging machine, and determines whether each stimulus has appeared before by pressing a button, thereby obtaining the magnetic resonance signal generated by the subject throughout the entire experiment. The features of the magnetic resonance signal are extracted to obtain the magnetic resonance representation.
[0054] Step S102: Fit the first feature and the second feature, obtain a semantic map based on the fitting result and a preset brain map, and obtain target semantic information corresponding to the target point based on the semantic map.
[0055] Multimedia material, its corresponding semantic information, and its magnetic resonance representation are fitted to obtain fitting weights. The fitted weights are then mapped to a pre-set brain map to determine the relationship between different categories of semantic information and the responses of various cortical targets, generating a semantic map. The target semantic category corresponding to the target point is obtained from the semantic map, and target semantic information related to the target semantic category is generated.
[0056] Step S103: Use the first feature as the input of the deep learning model, use the second feature as the constraint condition of the deep learning model, train the deep learning model, and determine the weight parameters of the deep learning model when the deep learning model meets the convergence condition.
[0057] Training the deep learning model includes using a portion of the first features and the corresponding second features as a training set, using the remaining first features and the remaining second features as a test set, comparing the similarity between the deep learning model output and the second features, and determining whether the deep learning model meets convergence conditions. The trained deep learning model can predict and output the magnetic resonance representation generated by the subject when identifying the multimedia material and the semantic information corresponding to the multimedia material based on the multimedia material and the semantic information corresponding to the multimedia material.
[0058] Step S104: Generate cognitive training materials based on the target semantic information and the weight parameters of the deep learning model.
[0059] First, the target semantic information category is obtained, and then the relevant description of the target semantic information category is obtained to obtain the target semantic information. The target semantic information is input into the neural network generation model, and the weight parameters of the deep learning model are used as the constraints of the neural network generation model. The neural network generation model outputs cognitive training materials related to the target semantic information. Exemplarily, the weight parameters of the deep learning model include shallow weight parameters and deep weight parameters. The shallow weight parameters are used to better constrain the detailed information of the generated material, and the deep weight parameters are used to better constrain the semantic information of the generated material.
[0060] Through the above steps, based on the first feature including multimedia materials and corresponding semantic information, a second feature including the response of the cerebral cortical target is obtained. The multimedia material provides auditory and visual stimulation in multiple aspects, which better stimulates the activity of the brain, making the semantic map established by the first and second features more accurate, and can accurately determine the corresponding semantic information according to the cerebral cortical target through the semantic map. The deep learning model is trained according to the first and second features, so that the deep learning model learns the relationship between the real cerebral cortical target response and the cognitive training material. In the process of generating cognitive training materials based on semantic information, the weight data of the deep learning model is used as a condition to better constrain the detailed information and semantic information of the cognitive training material, so that the generated cognitive training material can accurately and efficiently stimulate the brain area where the cerebral cortical target is located. Thereby solving the problem of not being able to obtain cognitive training materials that can accurately stimulate the brain functional cortical targets that need to be trained, and realizing the acquisition of cognitive training materials based on individual subjects and maximizing the training of the subject's brain.
[0061] In some embodiments, generating cognitive training materials includes: inputting first cognitive training materials generated based on target semantic information and weight parameters of a deep learning model into a deep learning model, and predicting a second feature corresponding to the first cognitive training material; determining whether the second feature corresponding to the first cognitive training material meets a preset condition; and filtering the first cognitive training material based on the determination result to obtain a second cognitive training material.
[0062] Inputting the first cognitive training material into the deep learning model includes: obtaining the first cognitive training material and corresponding semantic information; extracting features of the first cognitive training material, encoding the semantic information, and inputting the features of the first cognitive training material and the encoded semantic information into the deep learning model. The deep learning model predicts the response of the target point based on the first cognitive training material to obtain a second feature corresponding to the first cognitive training material; determining whether the magnitude of the second feature meets a preset condition, and screening the first cognitive training material.
[0063] Exemplarily, the deep learning model is a fusion model including a graph model, including a graph model and a data processing model, which is used to establish a mapping relationship between multimedia materials and corresponding semantic descriptions and representations of multimedia materials. Among them, the graph models include GCN (Graph Convolutional Networks), GAT (Graph Attention Networks) or GraphSage (Graph Sample and Aggregate) and other models. The graph model enables semantic information and magnetic resonance representation to better match and better represent semantic information. According to the data modality of the multimedia material, the corresponding data processing model is fused on the basis of the graph model. For image data, the optional models include resnet (Residual Network), Vit (Vision transformer, vision-based transformer model), mask-RCNN (Mask Region-based Convolutional Neural Network, mask-region-based graph neural network) and other models. The data processing of multimedia materials is realized through the data processing model.
[0064] In some embodiments, obtaining the first feature includes: extracting features of multimedia material through a convolutional neural network, encoding semantic information of the multimedia material, and using the features of the multimedia material and the encoded semantic information as the first feature.
[0065] Optionally, feature extraction is performed on multimedia materials through a residual convolutional network to obtain features of the multimedia materials. Multimedia materials include natural stimuli such as pictures, videos, and sounds that can produce visual, auditory, and other stimuli to the subjects, and include multiple modalities. Among them, the semantic information of the multimedia materials is the text modality of natural stimuli. The semantic information of the multimedia materials can be encoded by the encoding part of the autoencoder to obtain the features of the semantic information of the multimedia materials. According to the corresponding modal data, encoding methods such as unique hot code, image embedding, audio embedding, and word embedding can be selected. The features of the encoded semantic information can be combined with multiple features through splicing, dot product, or cross attention algorithm. The data obtained after the combination can be linearly or nonlinearly fitted with the magnetic resonance signal.
[0066] Obtaining the second feature includes: obtaining a functional magnetic resonance signal corresponding to the first feature, extracting features of the functional magnetic resonance signal to obtain a signal feature; mapping the signal feature to the cerebral cortex, and using the mapped signal feature as the second feature. After using the mapped signal feature as the second feature, the method further includes: obtaining a structural magnetic resonance signal and a diffusion magnetic resonance signal; and using the structural magnetic resonance signal and the diffusion magnetic resonance signal as the second feature.
[0067] The functional magnetic resonance (fMRI) signal is a magnetic resonance signal acquired during functional magnetic resonance imaging (fMRI), the structural magnetic resonance (SMR) signal is a magnetic resonance signal acquired during structural magnetic resonance imaging (SMRI), and the diffusion magnetic resonance (DMR) signal is a magnetic resonance signal acquired during diffusion magnetic resonance imaging (DMRI). Optionally, features of the fMRI signal are extracted using a generalized linear model, and a cortical surface standard template is used to map the features of the fMRI signal to the cerebral cortex.
[0068] A semantic map is generated based on the fitting results and a preset brain map, including: mapping the fitting results to the preset brain map, decoding the mapped fitting results through an autoencoder; and selecting the semantic information that generates the strongest magnetic resonance representation in each target point based on the decoding results to generate a semantic map. The fitting results after decoding the mapping through the autoencoder include: when generating the semantic map, decoding the features of the vertices through the decoding portion of the autoencoder. Optionally, before and after decoding by the autoencoder, noise reduction processing such as principal component analysis and the Topk algorithm can be used.
[0069] According to the target semantic information and the weight parameters of the deep learning model, the first cognitive training material is generated, including: obtaining the shallow weight parameters and deep weight parameters of the deep learning model; generating the first cognitive training material according to the target semantic information as input information, the shallow weight parameters and the deep weight parameters. Optionally, the cognitive training material is generated by a neural network generation model. The neural network generation model can be a fusion model including a diffusion model. The shallow weight parameters and the deep weight parameters are used as constraints and input into the neural network generation model. The shallow weight parameters are used to better constrain the detailed information of the generated material, and the deep weight parameters are used to better constrain the semantic information of the generated material.
[0070] In some embodiments, a cognitive training method is provided. FIG2 is a flow chart of the cognitive training method of this embodiment, including:
[0071] Step S201: obtaining a target point for cognitive training that the user needs to perform, and determining semantic information corresponding to the target point based on a semantic map;
[0072] Step S202: Acquire cognitive training materials according to the semantic information. The cognitive training materials are obtained based on the cognitive training material generation method described in any one of the above embodiments.
[0073] Step S203: displaying the cognitive training material to the user according to a preset duration.
[0074] Optionally, cognitive training materials are displayed to users for half an hour every day. Users can identify the cognitive training materials and stimulate the relevant brain areas corresponding to the targets, thereby achieving the effect of improving user cognition.
[0075] In this embodiment, a cognitive training method based on magnetic resonance and graph convolution is also provided. FIG3 is a flow chart of the cognitive training method based on magnetic resonance and graph convolution in this embodiment. As shown in FIG3 , the process includes:
[0076] Step S301, cognitive data collection and preprocessing. Taking image-based data collection and preprocessing as an example, the coco (Common Objects in Context) image dataset is obtained, and 10,000 natural images, labels and description information corresponding to the natural images are randomly obtained from the image dataset. For subjects who meet the experimental requirements, large-scale color natural images are continuously recognized, and the task functional magnetic resonance signals during the recognition process are collected. Optionally, the subjects who meet the experimental requirements can be: healthy subjects aged 18 to 30 years old, with normal or corrected-to-normal vision, all right-handed, without serious health problems, cognitive or mental diseases, such as stroke, epilepsy, heart disease, etc., without metal implants or pacemakers in the body, etc. The most important thing is that the subjects can understand the experimental process and requirements and can cooperate to complete the experimental tasks.
[0077] Figure 4 illustrates a continuous recognition task experimental design paradigm for this embodiment. As shown in Figure 4 , during the experiment, participants lay in an MRI machine, viewing natural images while simultaneously performing image recognition. Each participant completed a total of 40 MRI scans. During these scans, participants were required to focus on the images and, if an image had appeared before, to identify and press a key. 10,000 different images were randomly presented three times, with a uniform probability of appearance. The order of the images was fixed across participants. 90% of the images each participant viewed were different from those viewed by other participants, while 10% were shared with other participants. In one experiment, participants completed 12 runs, each with 75 trials. The first three and last four trials presented no images; the subject's interface remained blank. The remaining 68 trials were divided into 63 stimulus trials and 5 blank trials, with blank trials randomly placed within the run. Even-numbered runs consisted of 62 trials, with the 63rd trial being blank. Therefore, a total of 750 picture stimuli were presented in the one-time picture viewing and continuous recognition tasks. Optionally, the picture presentation frequency was 3 seconds per picture, and the interval between pictures was 1 second.
[0078] Step S302, obtain the input features and constraints of the deep learning model. The natural image is cropped and the features of the natural image are extracted through the residual convolutional network. The size of the cropped image is (425, 425, 3), where 425 corresponds to the length or width of the image, and 3 corresponds to the three color channels of red, yellow, and green. To encode the semantic information of the image, a UNet model, a Transformer model, a one-hot code, or other models containing codecs can be used. Taking one-hot code encoding as an example, the semantic information of the image is one-hot encoded to obtain a multi-classification one-hot code (multihots). Specifically, each natural image contains an 80-dimensional data, where 80 refers to the number of categories of specific things in the coco image set. When the image contains a certain category, the value of multihots in the category is set to 1, and if it does not contain it, the value of multihots in the category is set to 0. Because each image can contain multiple categories, multihots includes more than one 1. The extracted image features and multihots are used together as the input features of the deep learning model.
[0079] A response signal is generated when the subject views a set of pictures, wherein the response signal may be a task-state functional magnetic resonance signal. The features of the task-state functional magnetic resonance signal are extracted by a generalized linear model to obtain a beta value, wherein the beta value represents the degree of activation of each brain region of the subject. Figure 5 is a schematic diagram of the beta value of the task-state functional magnetic resonance signal feature in this embodiment. At this time, the beta value extracted is a voxel-based feature. A standard fsaverage template (cortical surface standard template) is selected to map the beta value to the cerebral cortex to obtain a beta feature. The beta feature obtained after the beta value is mapped to the cerebral cortex, and the mapped beta feature is used as a constraint condition for the deep learning model.
[0080] The input features and corresponding constraints of some deep learning models were used as the training set, and the remaining input features and constraints of the deep learning models were used as the test set. The training set consisted of the image features and response features corresponding to 9,000 different images viewed by each subject, while the test set consisted of the image features and response features corresponding to the 1,000 images viewed by all subjects.
[0081] Step S303, construct a deep learning model. Exemplarily, the deep learning model is a fusion model including a graph model, and a fusion model including a graph model is constructed based on the Vision GNN (ViG, a vision-based graph neural network) model. Among them, an MLP layer (Multi-Layer Perceptron) is added to the output of the ViG model to ensure that the shape of the output data is consistent with the shape of the beta feature, so that the fusion model including the graph model can predict the corresponding second feature based on the input first feature. Fsaverage7 and Fsaverage5 belong to a set of standard templates with different resolutions provided by the open source magnetic resonance data processing software FreeSurfer. The beta values of the model input are unified to the resolution of Fsaverage7. Considering the limitation of video memory size, the resolution of Fsaverage7 (high-resolution Fsaverage7 grid) of 327684 (including the left and right hemispheres) is first downsampled to Fsaverage5, so that the resolution is converted to the resolution of Fsaverage5 of 20484 (including the left and right hemispheres). The BERT model is selected to process the semantic information of the image. You can also add an MLP layer to the output of the BERT model to ensure that the shape of the output data is consistent with the shape of the beta features. Alternatively, the graph model can use a ResNet network, a Mask-RCNN network, or a diffusion model.
[0082] Acquire structural magnetic resonance signals and diffusion magnetic resonance signals. Process the structural magnetic resonance signals and diffusion magnetic resonance signals separately using a graph convolutional model to obtain structural magnetic resonance features and diffusion magnetic resonance features. The structural magnetic resonance features and diffusion magnetic resonance features have a shape of (20484,), i.e., a uniform resolution of fsaverage5. Optionally, when constructing a fusion model that includes a graph model, it can be determined whether to include structural magnetic resonance features and diffusion magnetic resonance features as prior information for the model based on actual conditions.
[0083] The image features output by the splicing graph model, the features of the image semantic information output by the bert model, the processed structural magnetic resonance features and the diffuse magnetic resonance features are used to obtain data with a shape of (20484, 4). Optionally, a cross-attention mechanism can be used to achieve the fusion of multiple features. The spliced data is processed through a convolutional layer to output data with a shape of (20484,). The obtained data is normalized to (-1, 1) through a softmax layer to obtain the similarity coefficient between the predicted value and the beta feature. The similarity coefficient is used to judge the effect of the prediction of the metric deep learning model and train the model to convergence. The loss function of the fusion model containing the graph model is as follows: Loss = 1-Corr(Pred softmax , β)
[0084] Among them, the Corr() function is used to obtain the similarity coefficient, Pred softmax is the predicted value output by the fusion model including the graphical model, β is the beta feature, and the Adam optimizer is selected as the model optimizer.
[0085] Grid search and random search methods, combined with cross-validation, can be used to tune hyperparameters such as the model learning rate, number of graph convolution layers, number of convolution layers, convolution kernel parameters, and number of channels of the fusion model containing the graphical model. Knowledge distillation can also be performed on the fusion model containing the graphical model, optionally introducing semantic information of the image as prior information during the distillation process. After distillation, the fusion model containing the graphical model can reduce the number of model weight parameters, thereby improving the model's prediction accuracy.
[0086] Step S304, generating a semantic map. Fit the multihots and the beta values mapped to the cortex, and correspond the fitting weights to the standard brain map. Optionally, the standard brain map can be Brainnetome Atlas. The fitting weights corresponding to the standard brain map are decoded. The decoding process is essentially to find the relationship between the semantics of the image and the response of the cerebral cortex. Optionally, multiple semantic categories are generated, each category includes multiple specific words, each word has a different color, and words of different colors correspond to the cerebral cortex responses of different brain regions. After decoding, the strongest expression semantic category is selected to obtain a semantic map. Figure 6 is a schematic diagram of the semantic map in this embodiment. As shown in Figure 6, the semantic information with strong responses in each brain region can be viewed.
[0087] Step S305: Select the corresponding semantics based on the target area. A target for cognitive training is selected from the semantic map based on the needs. For example, the fusiform face area (FFA) is selected. The semantic map shows that the semantic information category corresponding to the target is "person." The semantic map also provides descriptions related to this category, such as "happy person" or "college student."
[0088] Step S306, generating cognitive training materials. The descriptions "happy people" and "college students" associated with the semantic categories are input into the neural network generation model to generate a large number of semantically related picture materials. Figure 7 is a schematic diagram of the cognitive training materials in this embodiment. Optionally, the neural network generation model can be a stable diffusion (steady-state diffusion model) or a fusion model including a diffusion model. The neural network generation model is used to generate cognitive training materials. In addition to the semantic information corresponding to the target, its input can also introduce the shallow weight parameters and deep weight parameters of the deep learning model as constraint information.
[0089] Step S307, screening cognitive training materials. According to the method in step S302, feature extraction is performed on the cognitive training materials or other image materials used to train the deep learning model. The weight of the deep learning model constructed in step S303 is obtained. According to the extracted features and the weight of the trained deep learning model, the target response corresponding to the feature is predicted. The materials are further screened according to the size of the FFA target response signal of the predicted materials. Training the neural network generation model may include: adjusting the neural network generation model according to the prediction results of the deep learning model, or directly connecting the feature extraction and trained model weight parameters to the generation model after fixing them, and further optimizing the generation model.
[0090] Step S308: Targeted cognitive training is performed. The screened cognitive training materials are used for cognitive training, and the screened cognitive training materials can promote the peak activity of the target brain area to exceed the naturally occurring level.
[0091] The cognitive training process involved selecting parietal and frontal lobe targets on the semantic map, with the goal of improving participants' attention and memory. Image materials were generated and screened using the aforementioned model. Two groups of participants were recruited for a training experiment and a control experiment. The training group first conducted a short-term picture memory experiment. The training materials generated in the previous step were organized and shuffled to serve as stimuli. This experiment was conducted in an MRI machine. During the experiment, eight stimuli were randomly selected from the stimulus material and presented to the participants in sequence, pushing peak activity in the target brain regions above naturally occurring levels. Each stimulus was presented for a fixed duration. Two of the stimuli were then randomly removed, and the remaining six were replayed. Participants were asked to recall which stimuli were absent. After the stimuli were played back, they were given feedback via audio or a keystroke. The control group followed a similar experimental paradigm to the training experiment, except that the stimuli presented in the control experiment were natural images that had not been trained with graph convolution. After the images were displayed sequentially, participants were asked to identify which stimuli were absent upon replay.
[0092] MRI data were collected from both groups of subjects, and behavioral data were recorded. Accuracy comparisons were used to determine whether the experimental stimulation in the experimental group was more conducive to training concentration. Secondly, the activation results of the MRI data also served as key evidence for the effectiveness of the training experimental material. Specifically, by preprocessing the MRI data and performing general linear model analysis, and conducting statistical tests at the group level, it was found that the brain regions with greater activation in the experimental group than in the control group were located in the target locations of cognitive training, further proving the effectiveness of the experimental material.
[0093] Finally, the screened cognitive training materials will be shown to the subjects according to the preset duration, so that they can watch the training materials for half an hour every day to stimulate the subjects' brain areas related to attention and memory.
[0094] Through the above steps, a continuous recognition task based on natural stimuli is carried out to collect stimulus semantic information and acquire magnetic resonance signals. The natural stimuli and stimulus semantic information are encoded as input to a deep learning model, and features are extracted from the magnetic resonance signals as constraints for the deep learning model. These features are then fed into the constructed deep learning model for training, resulting in trained model weight parameters. The model inputs and model constraints are fitted, and the fitted weights are mapped to a standard brain map and then decoded to obtain a semantic map. The desired targets are selected based on the semantic map, and the semantic information corresponding to the targets is input into the constructed neural network generation model to generate cognitive training materials. Feature extraction and model prediction are performed on the cognitive training materials, and cognitive training materials are further selected based on target responses. The selected cognitive training materials are then used for cognitive training of the subjects. This method, using magnetic resonance imaging and graph convolution, enables non-invasive and measurable cognitive training of cortical targets.
[0095] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0096] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0097] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0098] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:
[0099] Step S101 : obtaining a first feature and a second feature, wherein the first feature includes multimedia material and semantic information corresponding to the multimedia material, the second feature includes magnetic resonance representation, and the first feature and the second feature are associated with each other.
[0100] Step S102: Fit the first feature and the second feature, obtain a semantic map based on the fitting result and a preset brain map, and obtain target semantic information corresponding to the target point based on the semantic map.
[0101] Step S104: Use the first feature as the input of the deep learning model, use the second feature as the constraint condition of the deep learning model, train the deep learning model, and determine the weight parameters of the deep learning model when the deep learning model meets the convergence condition.
[0102] Step S105: Generate cognitive training materials based on the target semantic information and the weight parameters of the deep learning model.
[0103] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.
[0104] In addition, in conjunction with the cognitive training material generation method provided in the above embodiments, a storage medium may also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the cognitive training material generation methods in the above embodiments is implemented.
[0105] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0106] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.
[0107] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.
[0108] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.
Claims
1. A method for generating cognitive training materials, It is characterized in that include: Acquire a first feature and a second feature, wherein the first feature includes multimedia material and semantic information corresponding to the multimedia material, the second feature includes magnetic resonance representation, and the first feature and the second feature are associated with each other; Fitting the first feature and the second feature, obtaining a semantic map according to the fitting result and a preset brain map, and acquiring target semantic information corresponding to the target point according to the semantic map; Using the first feature as an input of a deep learning model, using the second feature as a constraint condition of the deep learning model, training the deep learning model, and determining a weight parameter of the deep learning model when the deep learning model meets a convergence condition; Generate cognitive training materials based on the target semantic information and the weight parameters of the deep learning model.
2. The method for generating cognitive training materials according to claim 1, in, Generate cognitive training materials including: Inputting a first cognitive training material generated according to the target semantic information and a weight parameter of the deep learning model into the deep learning model, and predicting a second feature corresponding to the first cognitive training material; Determining whether a second feature corresponding to the first cognitive training material meets a preset condition; The first cognitive training material is screened according to the judgment result to obtain the second cognitive training material.
3. The method for generating cognitive training materials according to claim 1, in, Get the first feature, including: The features of the multimedia material are extracted through a convolutional neural network, and the semantic information of the multimedia material is encoded, and the features of the multimedia material and the encoded semantic information are used as the first features.
4. The method for generating cognitive training materials according to claim 1, in, Get the second feature, including: Acquire a functional magnetic resonance signal corresponding to the first feature, extract the feature of the functional magnetic resonance signal, and obtain a signal feature; The signal feature is mapped to the cerebral cortex, and the mapped signal feature is used as the second feature.
5. The method for generating cognitive training materials according to claim 4, in, After taking the mapped signal feature as the second feature, the method further includes: Acquiring structural magnetic resonance signals and diffusion magnetic resonance signals; The structural magnetic resonance signal and the diffusion magnetic resonance signal are used as the second feature.
6. The method for generating cognitive training materials according to claim 1, in, The semantic map is obtained based on the fitting results and the preset brain map, including: Mapping the fitting result to the preset brain map, and decoding the mapped fitting result through an autoencoder; According to the decoding results, the semantic information that generates the strongest magnetic resonance representation in each target point is selected to generate the semantic map.
7. The method for generating cognitive training materials according to claim 1, in, Generating first cognitive training material according to the target semantic information and the weight parameters of the deep learning model includes: Obtaining shallow weight parameters and deep weight parameters of the deep learning model; A first cognitive training material is generated according to the target semantic information as input information, the shallow weight parameters and the deep weight parameters.
8. A cognitive training method, It is characterized in that include: Obtaining the target point that the user needs to perform cognitive training on, and determining the semantic information corresponding to the target point according to the semantic map; Acquire cognitive training materials according to the semantic information, wherein the cognitive training materials are obtained based on the cognitive training material generation method according to any one of claims 1 to 7; The cognitive training material is displayed to the user according to a preset duration.
9. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to run the computer program to execute the cognitive training material generation method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method for generating cognitive training materials according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Brain cognition model infusion method based on integrated learning
CN111714118A
Brain cognitive skill transplanting method
CN111772629A
Apparatus and method for utilizing a brain feature activity map database to characterize content
CN113164126A
Method and system for analysing sound
US20140307878A1
System and Method for Measuring Perceptual Experiences
US20200187841A1
Cited By
Geomagnetic navigation method and system based on vector data
CN119289971A
