Cognitive training system and method for image-text association

Abstract

The invention discloses an image-text associated cognitive training system and a method thereof, and relates to the technical field of cognitive training and artificial intelligence. According to the method, image memorability calculation, image-text semantic matching and user behavior driven personalized recommendation and cognitive ability evaluation mechanisms are fused, the problems that in the prior art, image selection is random, image-text pairing is extensive, training lacks personalized adaptability and feedback analysis and the like are solved, and personalized, self-adaptive, efficient and intelligent cognitive training is achieved. The system comprises the following core modules: an image memorability database module; an image automatic matching module; a personalized recommendation module; and a cognitive training module.

Description

Technical Field

[0001] This invention relates to the fields of cognitive training and artificial intelligence technology, specifically to a personalized cognitive training system based on a graph-text association mechanism and its implementation method. Background Technology

[0002] With the rapid development of neuroscience, psychology, and artificial intelligence, cognitive training systems are widely used in areas such as cognitive impairment prevention, educational intervention, and language learning. Among them, image-text association training, as a classic memory enhancement method, has been proven to have significant effects in promoting long-term memory formation and improving learning efficiency.

[0003] However, existing image and text recognition training systems mainly suffer from the following technical limitations:

[0004] 1. Lack of image memorability factor: Most current training systems rely mainly on human experience or random sampling in image selection, without considering the differences in the "memorability" of the images themselves, and ignoring the intrinsic influence of images on memory performance.

[0005] 2. The image-text matching method is crude: In training content, images and target memory objects (such as words and phrases) are often matched through simple rules or fixed templates. There is a lack of quantitative modeling and automatic matching mechanism for the semantic relevance of images and text, making it difficult to flexibly adjust strategies according to different training objectives (such as language learning scenarios requiring strong semantic connections, and other scenarios requiring weak semantic connections).

[0006] 3. Individual differences are not adequately considered: Due to varying life experiences among users, their ratings of semantic relevance may differ. Furthermore, image memorability refers to the likelihood of a group remembering the image; an image deemed highly memorable by the general public may not be easily remembered by an individual. Therefore, fixed image-text stimuli cannot adapt to individual cognitive characteristics, easily leading to inefficient or unstable training results.

[0007] 4. Separation between cognitive assessment and training systems: Most systems only provide training tasks without in-depth analysis of user behavior data (such as recall accuracy and reaction time), and lack quantitative feedback on cognitive abilities and dynamic optimization mechanisms for training effects.

[0008] Therefore, there is an urgent need for a text-image association cognitive training system with highly memorable image selection capabilities, intelligent matching capabilities based on semantic relevance between images and text, personalized recommendation capabilities, and cognitive assessment capabilities, in order to improve the scientific nature and effectiveness of cognitive training. Summary of the Invention

[0009] The purpose of this invention is to provide a cognitive training system and method based on image-text association, which integrates image memorability calculation, image-text semantic matching, user behavior-driven personalized recommendation and cognitive ability assessment mechanism to solve problems such as arbitrary image selection, crude image-text matching, lack of personalized adaptability and feedback analysis in the prior art, and realize personalized, adaptive and efficient intelligent cognitive training.

[0010] To achieve the above objectives, this invention designs a cognitive training system for image-text association, comprising the following core modules:

[0011] Image memory database module;

[0012] This module stores a collection of images with pre-calculated or manually labeled image memorability scores, and supports dynamic updates based on user feedback. It is responsible for building and maintaining an image memory feature database, characterized in that all images possess a memorability score, which can be automatically generated by a deep convolutional neural network (such as the ResMem model) or obtained through large-scale behavioral experiment annotation; the memorability score M... i ∈[0,1] is the image I i The probability of being remembered under the group average is denoted by the scoring function as follows: in This is the visual feature vector of the image (extracted from networks such as ResNet), σ is the activation function of the deep neural network (such as ReLU, sigmoid function, etc.), W is the weight of the deep convolutional neural network, b is the bias of the deep convolutional neural network, and the superscript Τ is the transpose of the matrix. This module supports personalized updates driven by user behavior feedback (such as memory accuracy).

[0013] Automatic image matching module;

[0014] This method is used to perform pairing calculations based on the semantic relevance between the target memory object (such as text, words, or phrases) and the image. It prioritizes images with high memorability scores as pairing cues. In language learning scenarios, images with high semantic relevance to the target memory object are selected, while in other scenarios, images with low semantic relevance to the target memory object are selected.

[0015] This module implements a semantic relevance matching mechanism between text targets and images. Its core task is to match the semantic relevance of a given target memory object T. j Under the condition of selecting an image I i Calculate the semantic relevance of image-text pairs. in It is the semantic vector of the text. These are the semantic vectors of an image, obtained through image and text feature encoders such as the CLIP model. Selecting an image that matches the target memory object can be modeled as a multi-objective optimization problem: Where α and β are weighting factors; when the application scenario is language learning, the optimization problem becomes In practical applications, based on experience, it is recommended that α = 1 and β = 1.

[0016] Personalized recommendation module;

[0017] A personalized recommendation module is used to make personalized recommendations for image-text pairings based on users' individual characteristics and historical training data. The personalized recommendation module includes:

[0018] (1) User cognitive feature acquisition unit, used to acquire individual variables such as user age, cognitive ability assessment results, and training records.

[0019] (2) Image Personalization and Memorability Assessment Unit, used to assess a user's prior training performance on a specific image I. i The recall accuracy of the paired text is used to assess the user's personalized memorability rating of the image. When the number of times the image is presented exceeds a threshold θ... N At that time, calculate the user's personalized memorability score for the image. in It is the recall stage that the user has a memory of image I i Paired text T j The number of times you can correctly recall Image I i The number of times it is presented during the recall phase. The image's memorability score M in the image memorability database module. i For comparison, when M i and The difference is lower than the threshold θ M When, then image I i Images marked as having low memorability for the user are given lower recommendation priority. In practice, based on experience, θ is recommended. N =5, θ M =0.2.

[0020] (3) Personalized correction unit for semantic relevance of images and text, used to correct the semantic relevance of images and text based on the user's score during the encoding stage. The semantic relevance score S calculated in the image automatic matching module ij Differences between Determine if the image and text pair exhibit personalized semantic bias. When bias exists... Exceeding the preset threshold θS At that time, This serves as a semantic relevance score for the user's text and image content, used for personalized recommendations. Based on experience, θ is recommended. S =0.2.

[0021] Cognitive training module;

[0022] This module is used to generate training tasks, push image-text matching stimuli to users in a specific presentation format, and record user behavioral data, including memory accuracy, encoding reaction time, recall reaction time, and personalized scores for image-text semantic relevance. The cognitive training module includes:

[0023] (1) Memory encoding unit, used to present the paired text and image pairs to the user; the number of text and image pairs presented, the presentation time (the user can freely control the presentation time, or fix the presentation time of each text and image pair), the interval time, and the presentation order can all be set by the user (for example, a total of 10 text and image pairs are presented, each for 4 seconds, and the order is random).

[0024] (2) Semantic relevance scoring unit, used to obtain a personalized semantic relevance score from the user for the image-text pairs presented in the memory encoding unit. It is used in the image-text semantic relevance personalization correction unit of the personalized recommendation module.

[0025] (3) Memory interference unit, used to interfere with the user's short-term memory, eliminate the recency effect, and increase the difficulty and effectiveness of training. The presentation method and presentation time of the memory interference unit can be set by the user (for example, it is presented for a total of 30 seconds, and the interference content is a mathematical calculation problem).

[0026] (4) Memory Unit: This unit presents images to the user and records the user's recall data of the text. The presentation method of the memory unit can be set by the user (e.g., forced selection or free recall).

[0027] Cognitive assessment module;

[0028] This module is used to statistically analyze user behavior data during training, evaluate cognitive indicators such as memory ability, and use the evaluation results as input for the personalized recommendation module to achieve closed-loop regulation. This module is responsible for transforming training behavior data into structured cognitive indicators for training effect tracking and personalized recommendations. These indicators include, but are not limited to: ① recall accuracy. Where N correct N is the number of times the user correctly recalled the information in the recall unit. totalThe evaluation results are: ① the total number of recalls by the user in the recall unit (number of correct recalls + number of incorrect recalls); ② the recall reaction time, which is the average time it takes for the user to make a correct recall; ③ optionally, when the presentation time of the image-text pair in the memory encoding unit is controlled by the user, the memory encoding time is the average time it takes for the user to memorize the image-text pair. The evaluation results are fed back to the user as a cognitive training report and also input into the personalized recommendation module, forming an adaptive optimization system based on a behavior-cognition closed loop.

[0029] In addition, the system can be deployed on mobile terminals, wearable devices or online platforms, supporting remote cognitive training and long-term tracking and evaluation, and is suitable for various application scenarios such as attention training, language learning, and cognitive ability maintenance.

[0030] This invention improves overall memory retention and learning efficiency by introducing an image memorability module to filter training stimuli. It utilizes a text-image semantic embedding model to achieve intelligent pairing, supporting flexible adjustment of text-image semantic relevance and enhancing the goal-orientation and diversity of training task design. A personalized recommendation mechanism based on individual behavioral data dynamically adapts training content to user cognitive characteristics, enhancing the stability and sustainability of training effects. A cognitive assessment module constructs a behavior-cognition closed-loop feedback system, enabling dynamic adjustment and fine-grained management of training content and difficulty. The system possesses strong portability and application scalability, suitable for multi-terminal deployment and multi-field promotion, and has high scientific research value and promising industrial application prospects. Attached Figure Description

[0031] Figure 1 The overall architecture diagram of the system of the present invention includes the relationship between the image memory database module, the automatic matching module, the personalized recommendation module, the cognitive training module, and the cognitive assessment module.

[0032] Figure 2 : Schematic diagram of the personalized recommendation module architecture.

[0033] Figure 3 : Schematic diagram of the cognitive training module architecture. Detailed Implementation

[0034] The system of the present invention will be further described below with reference to the accompanying drawings:

[0035] Please see Figure 1 The image-text association cognitive training system of the present invention includes the following modules:

[0036] 1. Image memorability database module: Based on deep neural network models (such as ResMem) or manual scoring methods, images are pre-analyzed for memorability. The scoring results are stored together with the images, and personalized updates based on user performance are supported in the future.

[0037] 2. Automatic matching module: It uses image and text feature encoders such as CLIP model to semantically embed the descriptive labels of the target memory object and the image, and combines semantic similarity and memorability score to comprehensively evaluate the training value of each image-text combination.

[0038] 3. The personalized recommendation module collects multi-dimensional features such as users' past training results to build user profiles, compares them with data in the image memory database module, and dynamically optimizes image-text matching recommendations;

[0039] 4. Cognitive training module, supports multiple task modes, such as picture-text association memory task, delayed recall test, interference memory test, etc. All tasks are presented under standardized conditions and collect multi-dimensional behavioral data such as user clicks, recall, and evaluation;

[0040] 5. The cognitive assessment module analyzes the data collected during the training process and outputs indicators such as user memory accuracy, recall reaction time, and memory encoding time, providing quantitative support for cognitive training intervention.

[0041] Unless otherwise specified, the various technical features of this invention can be arbitrarily combined to form a variety of embodiments that are easily implemented by those skilled in the art, and all such embodiments should fall within the protection scope of this invention.

[0042] Example 1

[0043] This embodiment demonstrates the practical application of the image-text association cognitive training system of the present invention in a complete cognitive training process. The cognitive training module includes a memory encoding unit, a semantic relevance scoring unit, a memory interference unit, and a recall unit. The training is made intelligent and personalized through the collaborative work of an image automatic matching module, a personalized recommendation module, and a cognitive evaluation module.

[0044] 1. Automated selection of training materials

[0045] Before training begins, the system first calls the image memorability database module to filter images with memorability scores higher than a preset threshold, constructing a candidate image set. Then, the system activates the automatic image matching module, automatically calculating the semantic similarity between each candidate image and the candidate words, based on the "low semantic relevance between image and word" condition set for the current training task. Semantic similarity calculation uses the CLIP model's image and text feature encoder to obtain vectorized embedding codes for the image and text, further calculating the cosine distance of the embedding vectors, and combining this with the image memorability score to select 20 image-text pairs with low semantic similarity but high memorability. Subsequently, the personalized recommendation module re-ranks the initially selected image-text pairs. Based on the user's historical training data, including the accuracy of recalling previously presented images and personalized scores for image-text semantic relevance, the system further optimizes the image-word pair combinations, ensuring that each stimulus both meets the task setting and aligns with the user's personalized cognitive needs.

[0046] 2. Formal training (encoding units and semantic relevance scoring units)

[0047] In formal training, the system presents 20 optimized image-word pairs in random order, with each pair presented for 4 seconds. The images and words are displayed side-by-side (images on the left, words on the right). To enhance attention control, after each pair is presented, the system displays a semantic relevance assessment prompt. Users are required to rate the semantic relevance of the image and text on a Likert scale ranging from 1 to 9, with 1 representing the least relevant and 9 representing the most relevant.

[0048] 3. Formal Training (Memory Disruption Unit)

[0049] After completing the encoding phase, the system automatically enters the memory interference unit. The memory interference task consists of multiple randomly generated three-digit addition problems (e.g., 7+3+5=?, 2+5+4=?, 8+6+7=?). The user must answer each problem for one minute. This part is independent of the core cognitive training task, but the system synchronously records the answering time and accuracy for each problem as indirect reference indicators of attention and executive function.

[0050] 4. Formal Training (Recall Unit)

[0051] Upon entering the recall test, the system presents the 20 images from the encoding phase in a shuffled order. Users can choose one of two response modes: (1) type in the matching words they remember; (2) select one from four candidate words automatically generated by the system (including the correct answer and three distractors, which are generated based on high semantic similarity to the target word).

[0052] 5. Cognitive Assessment Module

[0053] After the training task is completed, the cognitive assessment module automatically analyzes the user's behavioral data at each stage, including: input accuracy and multiple-choice correctness during the recall stage; response time distribution; and error types (input errors or incorrect selection of distractors). The system combines this data to generate a cognitive assessment report for the user's current training task, covering their image-text matching memory ability and attention stability indicators. The assessment results are automatically fed back to the personalized recommendation module, providing a basis for parameter adjustments in the next training task, such as adjusting image memorability strength, distractor word generation strategies, or improving semantic matching relevance, to achieve continuous optimization.

[0054] As can be seen from this embodiment, the system of the present invention can automatically match high-value training materials, dynamically adapt to individual differences, and achieve closed-loop adjustment between training tasks and individual abilities through cognitive assessment, thereby significantly improving the personalization and scientific nature of cognitive training.

Claims

1. A cognitive training system for image-text association, characterized in that, include: 1) Image memorability database module, used to store a set of images with pre-calculated or manually labeled image memorability scores, and supports dynamic updates based on user feedback; This module is responsible for building and maintaining an image memory feature database, in which all images have a memoryability score, which can be automatically generated by a deep convolutional neural network or obtained through behavioral experiment annotation; Memorability score M i ∈[0,1] is the image I i The probability of being remembered under the group average is denoted by the scoring function as: in σ is the visual feature vector of the image, W is the activation function of the deep neural network, b is the bias of the deep convolutional neural network, and the superscript Τ is the transpose of the matrix. 2) Automatic image matching module, used to match images with target memory objects, including text, words, phrases, and images. Semantic relevance is used for pairing calculations; in language learning scenarios, images with high semantic relevance to the target memory object are selected, while in other scenarios, images with low semantic relevance to the target memory object are selected. This module implements a semantic relevance matching mechanism between text targets and images. Its core task is to match the semantic relevance of a given target memory object T. j Under the condition of selecting an image I i Calculate the semantic relevance of image-text pairs: in It is the semantic vector of the text. It is the semantic vector of the image; Image modeling for matching target memory objects is a multi-objective optimization problem. Where α and β are weighting factors, when the application scenario is language learning, the optimization problem becomes 3) A personalized recommendation module, used to make personalized recommendations for image-text pairings based on the user's individual characteristic information and historical training data. The personalized recommendation module includes: The user cognitive feature acquisition unit is used to acquire individual variables including the user's age, cognitive ability assessment results, and training records; The image personalization and memorability assessment unit is used to evaluate a user's prior training on an image I. i The recall accuracy of the paired text is used to assess the user's personalized memorability rating of the image; when the number of times the image is presented exceeds a threshold θ. N At that time, calculate the user's personalized memorability score for the image: in It is the recall stage that the user has a memory of image I i Paired text T j The number of times you can correctly recall Image I i The number of times it is presented during the recall phase; The image's memorability score M in the image memorability database module. i For comparison, when M i and The difference is lower than the threshold θ M When, then image I i Images that are marked as low memorability for the user will have their recommendation priority reduced. The image-text semantic relevance personalized correction unit is used to adjust the image-text semantic relevance based on the user's score during the encoding phase. The semantic relevance score S calculated in the image automatic matching module ij Differences between them: Determine if there is any personalized semantic bias in the image and text pair; when the bias exists... Exceeding the preset threshold θ S At that time, This score serves as the semantic relevance score between the user's text and images, and is used for personalized recommendations. 4) A cognitive training module, used to generate training tasks, push image-text matching stimuli to users in a specific presentation format, and record user behavioral data, including memory accuracy, encoding reaction time, recall reaction time, and image-text semantic relevance score; the cognitive training module includes: The memory encoding unit is used to present paired image-text pairs to the user; the number of image-text pairs presented, the presentation time, the interval time, and the presentation order are all set by the user. The semantic relevance scoring unit is used to obtain a personalized semantic relevance score from the user for the image-text pairs presented in the memory encoding unit. And used in the personalized recommendation module's image-text semantic relevance personalized correction unit; The memory interference unit; the presentation method and presentation time of the memory interference unit are set by the user. The recall unit is used to present images to users and record their recall data of text; the presentation method of the recall unit is set by the user. 5) Cognitive assessment module, used to statistically analyze user behavior data during training, assess users' memory ability and other cognitive indicators, and use the assessment results as input for personalized recommendation module to achieve closed-loop regulation.

2. The method using the system as described in claim 1, characterized in that... The steps are as follows: a) Automated selection of training materials Before training begins, the image memorability database module is first invoked to filter images with memorability scores higher than a preset threshold, constructing a candidate image set. Then, the system activates the automatic image matching module, which automatically calculates the semantic similarity between each candidate image and the candidate words based on the "low semantic relevance between image and word" condition set for the current training task. The semantic similarity calculation is based on the vectorized embedding encoding of the image and text obtained from the image and text feature encoders, calculating the cosine distance of the embedding vectors, and combining the image memorability score to select multiple sets of image-text pairs with low semantic similarity but high memorability. Subsequently, the personalized recommendation module re-sorts the initially selected image-text pairs based on the user's historical training data, including the accuracy of remembering previously presented images and the personalized score of image-text semantic relevance. b) Formal Training In formal training, the system will present multiple optimized image-word pairs in random order, with each pair presented for 4 seconds. The images and words are displayed side by side. To enhance attention control, after each pair is presented, the system will display a semantic relevance assessment prompt. Users need to evaluate the semantic relevance of the image and the text on a Likert scale of 1-9, where 1 represents the least relevant and 9 represents the most relevant. c) Formal Training After the coding phase is completed, the system automatically enters the memory interference unit. The memory interference task consists of multiple three-digit addition questions randomly generated by the system. Users need to answer each question for one minute. This part is independent of the core cognitive training task, but the system records the answering time and accuracy rate simultaneously as indirect reference indicators of attention and executive function. d) Formal training Upon entering the recall test, the system presents multiple images from the coding phase in a shuffled order; the user selects one of two answer modes: (1) typing in the matching words in their memory; (2) selecting one from four candidate words automatically generated by the system, including the correct answer and three distractors, which are generated based on a high semantic similarity to the target word. e) Cognitive Assessment Module After the training task is completed, the cognitive assessment module automatically calls up the user's behavioral data at each stage for analysis, including: input accuracy and multiple-choice correctness in the recall stage; response time distribution; error type; generates a cognitive assessment report for the user's current training task, covering their image-text matching memory ability and attention stability indicators; the assessment results are automatically fed back to the personalized recommendation module.

Images