Hand-eye combined cognitive evaluation training method and system and storage medium

Abstract

The invention relates to the technical field of cognitive training, in particular to a hand-eye combined cognitive evaluation training method and system and a storage medium. The method comprises the following steps: acquiring spatial position information of a hand of a user and fixation point information of eyes of the user to form a hand coordinate sequence and an eye coordinate sequence; judging whether the hand coordinate sequence and the eye coordinate sequence stay at a target position for a preset time length or not; if yes, respectively calculating a first proportion that a hand coordinate point falls within an allowable error range of the target object and a second proportion that an eye coordinate point falls within the allowable error range within the preset duration; and if both the first proportion and the second proportion reach respective preset thresholds, judging that the user successfully completes the hand-eye combination cognitive task. According to the scheme of the invention, the accuracy of cognitive evaluation and training is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cognitive training. More particularly, the present application relates to a hand-eye combined cognitive assessment training method, system and storage medium. BACKGROUND

[0002] The cognitive training system is a computer-aided diagnosis and treatment tool constructed by comprehensively using the theories of neuropsychology and rehabilitation medicine. The system is applied to the clinical treatment of cognitive impairment after brain injury in adults and children through cognitive function assessment, diagnosis and rehabilitation training modules.

[0003] Traditional cognitive training applies digital, animal, transportation tools, daily life goods picture cards, building blocks, and commonly used daily life goods to train cognitive functions such as attention, memory, and understanding. On this basis, specific activities are carried out to improve executive function, gradually combine basic cognitive ability and cognitive function skills, and carry out complex high-level cognitive behavior activities to improve cognitive function of patients.

[0004] In addition to traditional cognitive training, there is computer-aided cognitive training. This training mode can be operated online or offline through a computer, breaking through the time and space limitations of traditional cognitive training. At present, there are various computer-aided cognitive training systems. Generally, they are efficient, convenient, systematic, and comprehensive brain cognitive ability training systems developed based on the research results of cutting-edge cognitive neuroscience, combined with theory, paradigm, and game thinking. For example, cognitive assessment and training systems based on virtual reality (VR) or augmented reality (AR).

[0005] However, in the current cognitive assessment and training method, the assessment and training are mainly carried out by the reaction of a certain part to the stimulus, such as eye movement or hand movement. This way may cause a large deviation in the assessment result, and cannot accurately reflect the cognitive ability.

[0006] Therefore, the current urgent problem to be solved is that the prior art cannot accurately assess and train cognitive ability. SUMMARY

[0007] To solve the problem that the prior art cannot accurately assess and train cognitive ability, the present application provides solutions in the following aspects.

[0008] In a first aspect, the present application provides a hand-eye combined cognitive assessment training method, a target object is set on a display interface, and an allowable error range is set in combination with the position coordinates of the target object, the cognitive assessment training method comprising: acquiring spatial position information of a user's hand and gaze point information of the user's eye, forming a hand coordinate sequence and an eye coordinate sequence; determining whether the hand coordinate sequence and the eye coordinate sequence stay at a target position for a preset time length; if so, calculating a first proportion of hand coordinate points falling within the allowable error range of the target object and a second proportion of eye coordinate points falling within the allowable error range within the preset time length, respectively; and if both the first proportion and the second proportion reach respective preset threshold values, determining that the user successfully completes a hand-eye combined cognitive task.

[0009] In an embodiment, whether staying at the target position for the preset time length comprises: calculating the distance between the coordinate point of the hand or the eye and the center coordinate of the target object in real time; and if the distance between the continuous coordinate points and the target center is all less than a preset stay determination radius, determining effective stay.

[0010] In an embodiment, the threshold values of the first proportion and the second proportion are both in a range of 70% to 100%.

[0011] In an embodiment, before performing the assessment or training, a mode selection step is further included: receiving a selection instruction input by a user to select one of a pure hand mode, a pure eye mode or a hand-eye combined mode; if the pure hand mode is selected, determining the success or failure of the task only according to whether the first proportion reaches its threshold value; if the pure eye mode is selected, determining the success or failure of the task only according to whether the second proportion reaches its threshold value; and if the hand-eye combined mode is selected, determining the success or failure of the task according to whether both the first proportion and the second proportion reach respective threshold values.

[0012] In an embodiment, the shape, size, allowable error range and position on the interface of the coordinate range of the target object are adaptively adjusted according to training difficulty.

[0013] In an embodiment, after determining the success or failure of the task, a task result data recording step is further included: recording task completion time, hand-eye coordinate trajectory and success rate; and a cognitive assessment report generation step is further included: generating a cognitive assessment report based on historical task result data or dynamically adjusting difficulty parameters of subsequent training tasks.

[0014] In an embodiment, a plurality of target objects are further set on the display interface, and a continuous grasping and gazing task of the plurality of target objects is further set; and working memory and task switching ability are further assessed according to the number of successful continuous grasping and gazing tasks.

[0015] In one embodiment, the display interface is a virtual scene in a virtual reality or augmented reality display device environment, and the target object is a virtual object placed in the virtual scene.

[0016] In a second aspect, the present application also provides a hand-eye coordination cognitive assessment training system, comprising: a processor; a memory storing computer program instructions, which, when executed by the processor, implement a hand-eye coordination cognitive assessment training method according to one or more of the preceding embodiments.

[0017] In a third aspect, the present application also provides a computer readable storage medium storing a computer program, wherein the program, when executed by a processor, implements a hand-eye coordination cognitive assessment training method according to one or more of the preceding embodiments.

[0018] The present application has the beneficial effect that, according to the scheme of the present application, cognitive assessment and training are performed using multi-modal data acquisition and fusion, and the dynamic dwell judgment mechanism according to the time of user hand or eye dwell in the target region effectively improves the accuracy of cognitive assessment and training.

[0019] Further, by introducing a filtering threshold mechanism, such as 70% to 100% of points being in the target region, the fault tolerance and adaptability of the judgment are improved.

[0020] Further, the filtering ratio and allowable error range are dynamically adjusted according to the user's historical performance, individualized difficulty adjustment is achieved, and the flexibility of cognitive ability assessment and training is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which several embodiments of the present application are shown by way of example, and wherein like reference numerals refer to like elements throughout. In the drawings:

[0022] Figure 1 is a flowchart showing a hand-eye coordination cognitive assessment training method according to an embodiment of the present application;

[0023] Figure 2 is a schematic diagram showing the setting position of a target object according to an embodiment of the present application;

[0024] Figure 3 is a flowchart showing a cognitive assessment training process according to an embodiment of the present application;

[0025] Figure 4 is a composition diagram of a hand-eye coordination cognitive assessment training system according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0027] The specific embodiments of the present application will be described in detail below with reference to the drawings.

[0028] Figure 1 is a flow chart showing a hand-eye coordination cognitive assessment training method 100 according to an embodiment of the present application. A target object is set on a display interface, and an allowable error range is set in combination with the position coordinates of the target object. The shape, size, allowable error range and position on the interface of the coordinate range of the target object are adaptively adjusted according to the training difficulty. The display interface is a virtual scene in a virtual reality or augmented reality display device environment, and the target object is a virtual object placed in the virtual scene.

[0029] As shown in Figure 1 At step S101, the spatial position information of the user's hand and the gaze point information of the eye are obtained to form a hand coordinate sequence and an eye coordinate sequence.

[0030] At step S102, it is determined whether the hand coordinate sequence and the eye coordinate sequence stay at the target position for a preset time length. In some embodiments, whether to stay at the target position for a preset time length can be determined in the following manner. The distance between the coordinate point of the hand or eye and the center coordinate of the target object is calculated in real time. If the distance between the target center and the continuous multiple coordinate points is less than the preset stay judgment radius, it is determined that the stay is effective.

[0031] At step S103, if yes, the first proportion of the hand coordinate points falling within the allowable error range of the target object and the second proportion of the eye coordinate points falling within the allowable error range are calculated respectively within the preset time length. In some embodiments, the threshold values of the first proportion and the second proportion are both in the range of 70% to 100%.

[0032] At step S104, if the first proportion and the second proportion both reach the respective preset threshold values, it is determined that the user successfully completes a hand-eye coordination cognitive task.

[0033] In some embodiments, machine learning models can be used to extract and analyze features from the collected hand and eye coordinate sequences to improve the accuracy of fixation judgment and range comparison. Specifically, a series of fixation point coordinates over a period of time are generated to form a fixation point sequence and plotted as a scan path map. A T×J×3 tensor is constructed from the coordinate sequences of key hand points (such as the 21 hand bone joints) (T is the time step, J is the number of joints, and 3 represents the x, y, z coordinates or x, y, confidence). The aforementioned temporal sequence of eye movement or hand movement is input into a convolutional neural network model, which uses a self-attention mechanism to calculate the relationship between each time point in the sequence and all other time points. The feature sequences extracted from the two modalities are fused and input into the classification head of the convolutional neural network. The model can output specific reasons for classification failure, such as "only hand not in position," "only eye not fixating," "incorrect hand-eye sequence" (hand moves before eye looks), "excessive hand tremor," "inattentive inattention," etc.

[0034] Furthermore, a mode selection step is included before performing the assessment or training. Specifically, a selection instruction input by the user is received to select one of a pure hand mode, a pure eye mode, or a hand-eye combined mode. If a pure hand mode is selected, the success or failure of the task is determined solely based on whether the first ratio reaches its threshold; if a pure eye mode is selected, the success or failure of the task is determined solely based on whether the second ratio reaches its threshold; if a hand-eye combined mode is selected, the success or failure of the task is determined based on whether both the first ratio and the second ratio reach their respective thresholds.

[0035] After determining the success or failure of the task, it also includes recording task result data, including task completion time, hand-eye coordinate trajectory, and success rate; based on historical task result data, it generates cognitive assessment reports or dynamically adjusts the difficulty parameters of subsequent training tasks.

[0036] In another application scenario, to achieve a more comprehensive evaluation, multiple target objects can be set on the display interface, along with continuous grasping and fixation tasks for these objects. The working memory and task switching capabilities are then assessed based on the number of successful grasping and fixation tasks.

[0037] This invention can record and analyze eye and hand tracking trajectories in real time through eye and hand tracking processes. By extracting the time that hand and eye coordinates stay at the target position, it can comprehensively and accurately assess an individual's cognitive and attentional state.

[0038] The solution of the present invention will be described in detail below with reference to specific implementation methods.

[0039] like Figure 2As shown, the target object is illustrated using a rectangle as an example. The coordinates of the four points A, B, C, and D of the rectangle are fixed. The coordinates of the four points A', B', C', and D' are within ±1mm of the coordinate axes, allowing for an error range. It should be noted that the shape of the target object is not specifically limited in this invention; the target object can also be a circle with a fixed center at the interface, a radius r, and an allowable error range of 1mm. The coordinate system range of the target object is stored in a database.

[0040] Hand detection: Key point coordinates of the arm (five fingers and the center of the arm), pause for 3 seconds, and check if the key point coordinates are within the allowed range of the rectangle. Specifically, the hand position information is collected in real time through a gesture camera, and the hand pauses at the target position for 3 seconds to complete a grasping action, recording the coordinate range of the hand position at this time.

[0041] Eye detection: The system checks whether the coordinates of key points around the eyes remain within the allowed range of a rectangle during a 3-second pause. Specifically, the eye camera collects real-time eye position information, pauses at the destination for 3 seconds, and records the coordinate range of the eye position.

[0042] Filtering: Points collected during a 3-second hand / eye pause are considered within a certain percentage range and thus within the acceptable range; otherwise, they are considered outside the acceptable range. This percentage can be configured as needed, such as 70%, 80%, 85%, 90%, 95%, 98%, or 100%. Specifically, the combined coordinate ranges of the hand and eye are calculated. The coordinates of the hand and eye during the 3-second pause are compared to see if they are within the allowed range. If both the hand and eye are within the allowed range, it means that both hand and eye movements were within the acceptable range during this evaluation training, and the task is completed; otherwise, they are outside the acceptable range. Points collected during a 3-second hand or eye pause are considered within the acceptable range if 70%, 80%, 85%, 90%, 95%, 98%, or 100% are within the acceptable range; otherwise, they are considered outside the acceptable range.

[0043] The specific implementation process is as follows: Figure 3 As shown, the camera that collects hand information or the eye tracker that collects eye information first needs to be calibrated. The user selects the appropriate evaluation method, such as the pure hand mode, pure eye mode, or hand-eye combined mode mentioned above.

[0044] This embodiment uses the user's selection of a hand-eye combined mode for evaluation as an example. After the user selects the hand-eye combined mode, the evaluation begins. During the evaluation initialization process, the original coordinates of the target object and the coordinates within the allowable error range are determined.

[0045] Next, the patient uses their hands and eyes to locate the target object on the display screen. After determining the target object's location, the hands and eyes remain in this position for 3 seconds to collect data. If the collected point is within the allowed coordinate range, the count is incremented by 1. After the data collection is completed, the statistical count is determined. If the percentage of the count is within the corresponding tolerance range, i.e., the percentage of the count reaches the preset threshold, then one cognitive assessment training process is considered complete.

[0046] The present invention will now be described in detail with reference to a specific embodiment.

[0047] Take the "grabbing an apple" hand-eye coordination training task as an example. The user sits in front of the device, with a display screen in front of them, an eye camera integrated above, and gesture cameras located to the side or in front.

[0048] Users can select "Training Mode" -> "Hand-Eye Coordination" -> "Grab the Apple" task from the main interface via touch or mouse operation. The system loads the preset information of the target object "apple" from the database: Shape: Circular. Center coordinates (X0, Y0) = (500, 300) (unit: pixels). Tolerance radius (r) = 50 pixels (meaning the user's gaze or grasping position is within a circle with (500, 300) as the center and a radius of 50 pixels). Dwell time (T) = 3 seconds. Threshold (P) = 80% (meaning that 80% of the points collected within 3 seconds fall within the circle, which is considered a success).

[0049] Correspondingly, the system displays an image of an "apple" at the (500, 300) coordinate position on the display interface. The eye camera begins to capture images of the user's eyes at a frequency of 30 frames per second, and calculates the user's gaze point coordinates (X, Y) in real time using a built-in algorithm. e Y e ), and form an eye coordinate sequence.

[0050] The gesture camera simultaneously begins working, recognizing the spatial coordinates (X) of the user's hand (especially the tip of the index finger or the center of the palm). h Y h ), and mapped onto the screen coordinate system to form a hand coordinate sequence.

[0051] The system starts a timer and a counter separately for the eyes and hands, respectively. For the eyes: the system calculates the current gaze point (X) in real time. e Y e The distance D between the center of the apple (500, 300) and the center of the apple. e =sqrt((X e -500) 2 +(Y e -300) 2 ).

[0052] If De If D is less than or equal to 50 (i.e., within the tolerance range), both the effective eye point counter Count_eye_valid and the total collection point counter Count_eye_total are incremented by 1. e If the value is >50, only Count_eye_total is incremented by 1. This judgment process lasts for 3 seconds (approximately 90 frames of data).

[0053] For the hand: the logic is exactly the same. Calculate the distance D. h =sqrt((X h -500) 2 +(Y h -300) 2 It determines whether the hand is within the range and updates the valid hand count counter Count_hand_valid and the total count counter Count_hand_total.

[0054] After 3 seconds, the system performs the calculation:

[0055] Effective eye proportion: P eye =Count_eye_valid / Count_eye_total;

[0056] Effective hand proportion: P hand =Count_hand_valid / Count_hand_total.

[0057] Judgment rule: If (P) eye >=80% and (P hand If the success rate is >= 80%, the system determines that the "hand-eye coordination grasping" task was successful. To enhance the user experience, an Apple logo on the screen will display a smiley face animation accompanied by a positive sound effect: "Great!" The system will record this success in the background.

[0058] If any of the above conditions are not met (for example, the user is looking at the apple but their hand is not in place, or their hand is in place but they are not looking at the apple), the attempt is considered a failure. The apple displays a disappointed emoji and provides a sound effect saying, "Try again." The system records this failure.

[0059] After the user completes a training set (e.g., 10 attempts), the system generates a report including: Task Name: Apple Grabbing (Hand-Eye Coordination), Total Attempts: 10, Success Rate: 70%, Average Eye Fixation Accuracy: 85% (P out of 10 attempts) eye Average (of the average values), average hand grasping accuracy: 78% (P in 10 tasks) handThe average value of the results is calculated as follows: Hand-eye coordination consistency analysis: Number of successful attempts / (Number of successful attempts + (Number of successful attempts with hand failures) + (Number of successful attempts with eye failures)) = 7 / (7 + 1 + 2) = 70%. This report can intuitively help therapists assess a user's performance in attention (eye movement), motor execution (hand), and the coordination between the two.

[0060] To further enhance training effectiveness, multiple apples can be displayed on the real-world interface for the user to grab. For example, apples can be displayed in different positions on the screen in chronological order for the user to grab. The training result is determined by the number of apples successfully grabbed. This process allows for dynamic monitoring of user activity, enabling a comprehensive assessment of reaction speed, hand-eye coordination, and other factors, effectively improving the accuracy of cognitive judgments.

[0061] It should be noted that the above-mentioned method of using "grabbing apples" for evaluation and training is merely exemplary and not restrictive. Those skilled in the art can choose other game task methods, such as "whack-a-mole," according to actual needs.

[0062] Based on the solution of this invention, the problem of large bias in single-modal assessment is solved, providing more comprehensive and accurate cognitive function assessment results. This allows for more targeted training and improvement of users' hand-eye coordination and overall cognitive function. It can be widely applied to the assessment and training of various cognitive impairment groups. The eye-tracking cognition and attention assessment and training system of this invention is a cognitive rehabilitation training product integrating assessment and training. It can record and analyze an individual's eye movement trajectory in real time, extracting data such as fixation point, fixation time, and fixation frequency, thereby assessing the individual's cognitive and attentional state. It is suitable for various indications such as Attention Deficit Hyperactivity Disorder (ADHD), learning disabilities, autism, cognitive impairment, intellectual disability, speech disorders, cerebral disorders, cerebral dysplasia, and developmental delay. Through engaging animated games, it improves children's concentration and cognitive abilities from multiple aspects. Furthermore, the system also has a cognitive assessment function with multiple built-in scales, helping doctors improve work efficiency and providing a scientific basis and personalized training plan for patient rehabilitation.

[0063] Furthermore, in addition to the aforementioned applications such as screening and intervention for developmental disorders in children and monitoring cognitive decline in the elderly, the solution of this invention can also be applied to the content of vocational ability assessment and training, such as the assessment and training of vocational abilities that require a high degree of hand-eye coordination, such as pilots and surgeons.

[0064] The present invention also provides a hand-eye cognitive assessment and training system, including a processor and a memory, wherein the memory stores computer program instructions, and when the processor executes the computer program instructions, it implements the hand-eye cognitive assessment and training method described above.

[0065] The system also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface, the settings and functions of which are known in the art and will not be described in detail here. In addition, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a hand-eye coordination cognitive assessment and training method as described above.

[0066] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0067] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A cognitive assessment and training method that combines hand-eye coordination, characterized in that, The cognitive assessment training method includes setting a target object on the display interface and setting an allowable error range based on the target object's position coordinates. Obtain the spatial position information of the user's hands and the gaze point information of the eyes to form a hand coordinate sequence and an eye coordinate sequence; Determine whether the hand coordinate sequence and eye coordinate sequence remain at the target position for a preset duration; If so, then calculate the first proportion of the hand coordinates falling within the allowable error range of the target object within the preset time period, and the second proportion of the eye coordinates falling within the allowable error range. If both the first ratio and the second ratio reach their respective preset thresholds, the user is deemed to have successfully completed a hand-eye coordination cognitive task.

2. The cognitive assessment and training method for hand-eye coordination according to claim 1, characterized in that, Whether the target location is stayed for a preset time includes: Calculate in real time the distance between the coordinates of the hand or eye and the center coordinates of the target object; If the distance between multiple consecutive coordinate points and the target center is less than the preset dwell radius, then the dwell is determined to be valid.

3. The cognitive assessment and training method for hand-eye coordination according to claim 1 or 2, characterized in that, The thresholds for both the first and second ratios are between 70% and 100%.

4. The cognitive assessment and training method for hand-eye coordination according to claim 1, characterized in that, Before performing the assessment or training, a mode selection step is also included: Receive user input selection instructions to select one of pure hand mode, pure eye mode, or hand-eye combined mode; If the pure hand mode is selected, the success or failure of the task will be determined solely based on whether the first ratio reaches its threshold. If the pure eye mode is selected, the success or failure of the task will be determined solely based on whether the second ratio reaches its threshold. If the hand-eye coordination mode is selected, the success or failure of the task is determined by whether both the first ratio and the second ratio reach their respective thresholds.

5. The cognitive assessment and training method for hand-eye coordination according to claim 1, characterized in that, The shape, size, allowable error range, and position on the interface of the target object's coordinate range are adaptively adjusted according to the training difficulty.

6. The cognitive assessment and training method for hand-eye coordination according to claim 1, characterized in that, After determining the success or failure of the task, the following is also included: Record task result data, including task completion time, hand-eye coordinate trajectory, and success rate; Based on historical task results data, generate cognitive assessment reports or dynamically adjust the difficulty parameters of subsequent training tasks.

7. The cognitive assessment and training method for hand-eye coordination according to claim 1, characterized in that, Also includes: Set multiple target objects on the display interface, and set up continuous grasping and gaze tasks for multiple target objects; Assessment of working memory and task switching ability is based on the number of successful consecutive grasping and fixation tasks.

8. The cognitive assessment and training method for hand-eye coordination according to claim 1, characterized in that, The display interface is a virtual scene in a virtual reality or augmented reality display device environment, and the target object is a virtual object placed in the virtual scene.

9. A cognitive assessment and training system that combines hand and eye movements, characterized in that, include: processor; A memory that stores computer program instructions, which, when executed by the processor, implement a hand-eye coordination cognitive assessment and training method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a hand-eye cognitive assessment training method as described in any one of claims 1-8.

Images