Augmented reality-based visual space function quantification analysis method and system

Abstract

This disclosure provides a method and system for quantitative analysis of visuospatial function based on augmented reality, relating to the fields of medical and mixed reality technologies. The method includes: acquiring the depth data of the real space where the user is located and converting it into a spatial triangular mesh in a 3D virtual engine; obtaining world coordinates in the 3D virtual engine based on the real physical plane; generating corresponding spatial anchor points; responding to the user's adjustment operations on the PC control terminal for the first and second spaces, achieving precise anchoring of the 3D virtual object on the real horizontal physical desktop and the real vertical physical wall; capturing the user's corresponding touch interaction operations and calculating the physical offset distance generated by the interaction in real time. Based on the user's touch and ray interaction operations, this disclosure quantitatively tests the 3D spatial positioning ability at different depths and orientations, effectively improving the ability to detect and accurately assess visuospatial function abnormalities or early deterioration states.

Description

Technical Field

[0001] This disclosure relates to the fields of medical and mixed reality technologies, specifically to a method and system for quantitative analysis of visuospatial functions based on augmented reality. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Visuospatial dysfunction is a common and core symptom of cognitive impairment in neurodegenerative diseases such as Alzheimer's disease (AD). Traditional methods for assessing visuospatial function often rely on paper-and-pencil tests (such as the clock drawing test and geometric figure copying). However, these methods are mostly limited to a two-dimensional plane and cannot accurately reflect a patient's distance perception and spatial positioning abilities in three-dimensional space. Furthermore, traditional methods struggle to obtain high-precision quantitative data, such as the patient's reaction time and spatial positioning deviation. In addition, the implementation and interaction of paper-and-pencil tests present difficulties for some elderly patients with motor impairments or mobility issues.

[0004] In recent years, with the development of XR (Extended Reality) technology, especially the widespread adoption of Augmented Reality (AR) and Mixed Reality (MR) headsets (such as HoloLens 2), new opportunities for three-dimensional visualization and precise interaction have emerged in medical assessment. However, current AR applications still have the following limitations when used for assessing elderly patients: (1) In AR assessment tasks (e.g., having patients memorize and find virtual objects placed in a room, or stepping on virtual patterns), virtual objects may move or jitter unexpectedly relative to their real-world positions. This phenomenon of virtual object drift can lead to inaccurate test results and make it difficult for patients to operate independently. (2) It is impossible to quantitatively collect the three-dimensional spatial positioning ability at different depths and orientations, and it is impossible to accurately and quickly obtain the spatial perception performance of Alzheimer's disease (AD) patients in a three-dimensional real environment. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a quantitative analysis method and system for visuospatial function based on augmented reality. By utilizing an augmented reality device (HoloLens2) in conjunction with a PC control terminal, and constructing highly stable spatial anchor points and customized multi-dimensional interactive testing paradigms, the quantitative analysis of patients' visuospatial function can be achieved.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions: Augmented reality-based quantitative analysis methods for visuospatial functions include: Acquire the real-world spatial depth data of the test users and convert it into a spatial triangular mesh in a 3D virtual engine; The real physical plane is obtained based on the spatial triangular mesh, and the world coordinates in the 3D virtual engine are obtained by transforming the real physical plane. Based on world coordinates, corresponding spatial anchor points are generated in the 3D virtual engine. The spatial anchor points include horizontal spatial anchor points and vertical spatial anchor points. Responding to user adjustments to the first and second spaces on the PC control panel, it enables precise anchoring of 3D virtual objects on real horizontal physical desktops and real vertical physical walls; Once the testing phase begins, a preset number of virtual test targets are generated based on the test task and anchoring results. The corresponding touch interaction operations of the test users are captured, and the physical offset distance generated by the interaction is calculated in real time.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: An augmented reality-based quantitative analysis system for visuospatial functions includes: The three-dimensional spatial anchor point positioning module is used to acquire the depth data of the real space where the test user is located and convert it into a spatial triangular mesh in the three-dimensional virtual engine; obtain the real physical plane based on the spatial triangular mesh, and obtain the world coordinates in the three-dimensional virtual engine based on the real physical plane; generate corresponding spatial anchor points in the three-dimensional virtual engine based on the world coordinates, the spatial anchor points including horizontal spatial anchor points and vertical spatial anchor points; respond to the user's adjustment operations on the first space and the second space on the PC control terminal to realize the precise anchoring of the three-dimensional virtual object on the real horizontal physical desktop and the real vertical physical wall; The visual space quantitative testing module is used to generate a preset number of virtual test targets based on the test task and anchoring results after entering the testing phase, capture the corresponding touch interaction operations of the test user, and calculate the physical offset distance generated by the interaction in real time.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program that, when executed by a processor, implements the augmented reality-based visuospatial function quantitative analysis method.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the augmented reality-based visuospatial function quantitative analysis method.

[0010] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the augmented reality-based visuospatial function quantitative analysis method.

[0011] Compared with the prior art, the beneficial effects of this disclosure are as follows: This disclosed augmented reality-based quantitative analysis method for visuospatial function generates stable spatial anchor points on a physical environment plane using spatial mapping technology, along with multi-dimensional virtual test targets. Through augmented reality display and user-generated touch and ray interaction, it quantitatively collects 3D spatial positioning capabilities at different depths and orientations. By combining augmented reality with data such as offset distance and response time generated by interaction, the testing process of visuospatial function is intuitively presented, breaking the limitations of traditional two-dimensional paper-and-pen tests.

[0012] This disclosed augmented reality-based quantitative analysis method for visuospatial function transforms real-time hand interaction behavior into structured clinical assessment data, providing quantitative and traceable data support for the assessment system to subsequently execute grading diagnostic rules based on average error values. Medical assessors can accurately acquire the spatial perception performance of Alzheimer's disease (AD) patients in a three-dimensional real-world environment. By calculating and grading the average error data of various test tasks, the method can effectively improve the quantitative detection and accurate assessment of visuospatial function abnormalities or early deterioration states. Attached Figure Description

[0013] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0014] Figure 1 This is a system architecture diagram of a visuospatial function quantitative analysis system based on augmented reality, according to an embodiment of this disclosure. Figure 2 This is a flowchart illustrating the quantitative testing process for augmented reality-based visuospatial functions according to an embodiment of this disclosure. Detailed Implementation

[0015] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0016] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0017] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0018] Example 1 One embodiment of this disclosure provides a method for quantitative analysis of visuospatial functions based on augmented reality, comprising the following steps: Step 1: Obtain the depth data of the real space where the test user is located, and convert it into a spatial triangular mesh in the 3D virtual engine; Step 2: Obtain the real physical plane based on the spatial triangular mesh, and transform the real physical plane to obtain the world coordinates in the 3D virtual engine; Step 3: Generate corresponding spatial anchor points in the 3D virtual engine based on world coordinates. The spatial anchor points include horizontal spatial anchor points and vertical spatial anchor points. Step 4: Respond to the user's adjustment operations on the first and second spaces on the PC control terminal to achieve precise anchoring of the three-dimensional virtual objects on the real horizontal physical desktop and the real vertical physical wall. Step 5: After entering the testing phase, generate a preset number of virtual test targets based on the test task and anchoring results, capture the corresponding touch interaction operations of the test users, and calculate the physical offset distance generated by the interaction in real time.

[0019] As one embodiment, the augmented reality-based spectral spatial function quantitative analysis method disclosed herein utilizes the augmented reality capabilities and spatial mapping technology of HoloLens 2, in conjunction with a PC-based control backend, to construct a highly stable holographic interactive field at the evaluation site using spatial 3D technology. In traditional augmented reality applications, virtual objects are prone to drifting, jittering, or position loss, severely affecting the accuracy of quantitative test data and the test subject's experience. To achieve precise alignment between virtual and real spaces, this embodiment develops environmental mesh construction technology and a precise spatial anchor point positioning algorithm to ensure that the test target can be absolutely and stably fixed in the real world. The specific implementation process is as follows: Step 1: Obtain the depth data of the real space where the test user is located, and convert it into a spatial triangular mesh in the 3D virtual engine; In this embodiment, the spatial mapping function provided by HoloLens 2 is first employed. This function utilizes the device's built-in TOF depth sensor and camera to scan the surrounding test environment of the user in real time. The mesh recognition observer provided by MRTK2 (Mixed Reality Toolkit 2) is used to acquire triangular mesh data of the real space. Then, through a coordinate transformation algorithm, the identified local coordinate system of the mesh is converted into the world coordinate system in the Unity 3D virtual engine, thereby accurately extracting the world coordinates of the actual horizontal physical desktop and vertical physical walls. A mesh composed of triangles as basic units is then attached to the surface of surrounding objects, providing the system with mesh information of the surrounding environment.

[0020] Step 2: Obtain the real physical plane based on the spatial triangular mesh, and transform the real physical plane to obtain the world coordinates in the 3D virtual engine; Specifically, planar feature recognition is performed on the triangular mesh in the test space to obtain the real physical plane, and the coordinate data of the identified real physical plane is extracted and converted into world coordinates in the 3D virtual engine. In the triangular mesh generated by spatial mapping, physical plane identification is achieved using normal vector clustering combined with the Random Sample Consensus Algorithm (RANSAC). The detailed steps are as follows: (1) Extract the set of vertex coordinates V and the set of faces F of the triangular mesh in the test space. Traverse all triangular faces and calculate the normal vector n and the coordinates of the center point of each triangular face using the three vertices of the face. Divide the continuous triangular faces whose normal vector angle is within the allowable error range into the same candidate region. For the extracted candidate regions, use the RANSAC algorithm to fit the optimal plane: Randomly select three non-collinear vertices within the candidate region and calculate an initial plane equation: Ax + By + Cz + D = 0; calculate the equations for all other vertices within the candidate region. , , The orthogonal distance d to the initial plane: Set a distance threshold ε (0.02 meters). If the orthogonal distance d < ε, then mark the corresponding vertex as an "interior point". After multiple iterations of sampling, retain the plane model containing the most interior points as the identified real physical plane.

[0021] Furthermore, local feature data is extracted from the real physical plane. The local feature data includes the local coordinates of the center point, the local normal vector, and the local coordinates of the vertices of the two-dimensional boundary polygon data in the local mapping coordinate system of the underlying hardware device.

[0022] (2) In order to avoid abnormal offset of coordinate values ​​due to the mismatch between the hardware sensor and the scale of the Unity 3D virtual engine, the local feature data is scaled according to a preset ratio (millimeters-level data are multiplied by 0.001 and scaled to meters-level data) to match the standard unit of the 3D virtual engine; at the same time, according to the coordinate system rules of the Unity 3D virtual engine, the scaled coordinate data is chiralized (the Z-axis coordinate is reversed to realize the conversion from the right-hand coordinate system to the left-hand coordinate system).

[0023] (3) Construction of homogeneous transformation matrix. Obtain the pose data of the hardware device relative to the origin of the 3D virtual engine world, and construct a 4x4 homogeneous transformation matrix. The matrix R contains a rotation matrix R and a 3x1 translation vector T, and its matrix structure is as follows: .

[0024] (4) World coordinate system transformation. The local feature data after the magnitude calibration and chirality transformation are converted into homogeneous coordinate form and multiplied by the homogeneous transformation matrix. Obtain the world coordinates of the local feature data in the 3D virtual engine. .

[0025] Step 3: Generate corresponding spatial anchor points in the 3D virtual engine based on world coordinates; Specifically, based on the world coordinate characteristics of the real physical plane, corresponding spatial anchor points are generated in the 3D virtual engine. These spatial anchor points include horizontal spatial anchor points to be aligned with the horizontal physical desktop and vertical spatial anchor points to be aligned with the vertical physical wall. Based on the calculated world coordinates Pworld and the transformed normal vector direction, a corresponding virtual plane object is instantiated in the 3D virtual engine; and an initial spatial anchor point is generated and bound at that world coordinate position.

[0026] Step 4: Respond to the user's adjustment operations on the first and second spaces on the PC control terminal to achieve precise anchoring of the three-dimensional virtual object on the real horizontal physical desktop and the real vertical physical wall; Specifically, in response to user adjustments made to the first space via the PC control terminal, the system acquires horizontal movement step parameters, translates and fixes the horizontal spatial anchor point within the virtual space, thereby achieving precise anchoring of the 3D virtual object on the real horizontal physical desktop. In response to user adjustments made to the second space via the PC control terminal, the system acquires vertical movement and rotation step parameters, translates and rotates the vertical spatial anchor point within the virtual space to adapt to the wall's tilt angle, thus achieving precise anchoring of the 3D virtual object on the real vertical physical wall. The virtual space refers to the space within the HoloLens, automatically generated by the HoloLens sensors scanning the environment.

[0027] As one implementation example, the positioning and fine-tuning of spatial anchor points are achieved by maintaining real-time wireless communication between HoloLens2 and the PC control terminal via the TCP / IP protocol, responding to debugging commands issued by the user on the PC control terminal. Specifically, this involves adjustment verification of the first space (horizontal desktop) and the second space (vertical wall): (1) The adjustment of the first space is mainly used to adapt to the subsequent circular center point test. The PC control terminal provides an input interface for the horizontal movement "step size" parameter (default 0.01 meters) and translation control buttons for six dimensions: up, down, left, right, forward, and backward. It responds in real time to the first space adjustment operation generated by the professional user clicking the translation button on the PC terminal, and sends the command to the HoloLens2 terminal to drive the horizontal space anchor point to move instantly in the three-dimensional virtual space according to the corresponding step size until the center of the virtual anchor point is completely attached to and fixed with the real horizontal physical desktop.

[0028] (2) The adjustment of the second space is mainly used to adapt to the center point tests of squares and strips. Since the wall surface may be uneven in the actual test environment, or the test subject may have a tilt angle when facing the wall, a rotation control was specially set up on the basis of translation control: The system responds in real-time to the "rotation step size" parameter (default 5 degrees) input by professional users on the PC control terminal, as well as the secondary spatial adjustment operations generated by the left and right rotation buttons, driving the vertical spatial anchor point to rotate around its own normal axis in virtual space. This adjustment mechanism, which combines translation and rotation, allows the virtual plane to perfectly adapt to the tilt angle of the wall, greatly enhancing the system's robustness in various complex room environments.

[0029] As one implementation, the spatial anchor point setting on the HoloLens 2 terminal is achieved in conjunction with the PC control terminal. In a fixed medical testing environment, before the first test, the user only needs to perform the above steps to complete the spatial plane anchor point calibration and setting once. After the setting is completed, the spatial coordinate data will be persistently saved, enabling permanent access by different test subjects within the same testing environment. This avoids the tediousness of repeated calibration and provides the most crucial underlying spatial stability guarantee for subsequent acquisition of millimeter-level precision offset distance data.

[0030] Step 5: After the anchor point is located, the testing phase begins. Based on the test task and the anchoring results, a preset number of virtual test targets are generated to capture the corresponding touch interaction operations of the test user and calculate the physical offset distance generated by the interaction in real time.

[0031] As one example, this step performs an augmented reality-based spectral quantification test, the specific test process of which is as follows: Step 51: Perform device communication and environment deployment, obtain the test number, name, gender, age and education level information of the test subjects, and complete the data interaction and synchronization between the augmented reality device and the PC control backend; Specifically, before the initial test after determining the test environment, the user sets up the spatial anchor points on the PC control terminal. This PC backend uses the TCP / IP protocol to achieve real-time wireless communication with the HoloLens2. The HoloLens2 and PC need to be on the same local area network. The user enters the HoloLens2's IPv4 address and port (default 8080) on the PC and clicks "Connect Test" to control the HoloLens2 in real time. Within the same test environment, the user only needs to set the horizontal and vertical spatial anchor points once to enable permanent use by different test subjects.

[0032] Next, the participant information entry stage begins. On the HoloLens 2 main interface, participants can move their fingers so that when the index finger's ray is within the input box, they pinch their index finger and thumb to bring up the virtual keyboard. They can then use pinching or clicking the virtual keyboard to complete the input of test number, name, age, and education level. By moving the index finger's ray to the gender and test selection boxes and pinching, participants can select their gender and test content. If a participant has mobility issues, a professional can fill in all the above information on their behalf in the information management module on the PC. Then, clicking the "One-Click Synchronize All Data" button will synchronize the data, including test number, name, gender, age, education level, and test type, to the HoloLens 2 in one go.

[0033] Step 52: In response to the test type selection instruction from the test subject or the PC control backend, load the corresponding visual space function test task in the virtual space. The test task includes circular center point test, square center point test and bar center point test. After information entry is complete, the user selects a test task on the HoloLens 2 device. The user then moves the ray to the "Start Test" button on the HoloLens 2 device and pinches and clicks, or a professional user clicks the "Start Test" button on a PC. The system then enters the initial tutorial phase for the selected test. After the initial tutorial is completed, clicking the "Continue" button or clicking the "Formal Test" button on the PC initiates the formal test phase. This system provides three dimensions of interactive testing: (1) Circular Center Point Test: In the initial teaching, three circular targets appeared sequentially in the space. The subjects used their right index finger to click on them in sequence to complete the teaching. After entering the formal test, the subjects interacted by directly clicking on the desktop with their right index finger, clicking on the circular center point that appeared 15 times in sequence. Schematic diagram of the circular center point teaching test interface and the formal test interface. The holographic panel prompts "Please use your right index finger to click on the center point of the circle generated in front of you in sequence." The user completes the direct touch click on a horizontal virtual plane that fits the physical desktop.

[0034] (2) Square Center Point Test: In the initial teaching, three square targets appear in sequence in the space for practice; after entering the formal test, the subject is required to use the right index finger to directly click on the wall to complete the interaction, clicking the center point of the rectangle that appears 15 times in sequence. The holographic panel of the square center point teaching test interface and the formal test interface prompts to use the right index finger to click on the center point of the square. The target is generated on the vertical wall that has been calibrated with anchor points.

[0035] (3) Bar center point test: In the initial teaching, the subjects were required to use the right-hand ray pinching method to click on three targets in space in sequence; in the formal test, the subjects were required to use the right-hand ray pinching method to click on the bar targets generated on the wall, and complete 15 interactions in sequence. The holographic panel of the bar center point teaching test interface and the formal test interface prompted "Please use the right index finger ray, pinch the index finger and thumb, and click on the center of the bar generated in front of you in sequence", which demonstrated the ray-based air-tapping interaction mode of the system for mid-to-long-distance visual-spatial function assessment.

[0036] Step 53: Before the formal test, the initial teaching targets are presented in sequence in the virtual space to guide the test users to complete the preset hand interaction actions to verify the feasibility of the device operation. Specifically, AR devices include corresponding voice prompts and questions to guide participants in performing actions such as clicking objects with their hands or using a ray to click on them. If participants have limited cognitive abilities, experts can also provide guidance and instruction.

[0037] Step 54: In the formal testing phase, based on the selected test task, a preset number of virtual test targets are cyclically generated on the fixed spatial anchor point surface. The corresponding touch interaction operations of the test user are captured, and the spatial offset distance from the target center point and response time generated by each interaction are collected in real time. The virtual test targets are controlled by code, and the spatial range of the generated virtual test targets is consistent with the spatial range of the plane after adjustment in Step 4. Only one virtual test target is generated at a time; the next target is generated only after a finger click or ray click is completed.

[0038] In this embodiment, the interactive operations of the subject user are captured in real time and the spatial offset distance is calculated. Specifically, the following calculation and conversion steps are performed: (1) Obtain the target reference position: After each virtual test target is generated, the transformation component data of the current target is obtained through the 3D virtual engine (Unity), and the 3D coordinates (X_target, Y_target, Z_target) of its geometric center point in the augmented reality world coordinate system are extracted as the spatial reference center point for this interaction.

[0039] (2) Obtain the 3D coordinates of the hand interaction points: Based on the input data providers of the MRTK2 (Mixed Reality Toolkit 2) framework, the coordinates of key hand nodes are tracked in real time according to the type of test task. For the circular center point test and the rectangular center point test (near field interaction): the MRTK2's Articulated Hand Tracking algorithm is called to capture the three-dimensional coordinates (X_touch, Y_touch, Z_touch) of the index fingertip of the subject's right hand in the world coordinate system in real time.

[0040] For the bar center point test (far-field interaction): the HandRay component of MRTK2 is called to obtain the intersection of the far-field ray emitted by the right index finger of the subject with the vertical wall space anchor point grid plane in real time through physical ray detection (Raycast), and the three-dimensional coordinates (X_ray, Y_ray, Z_ray) of the intersection point in the world coordinate system are recorded.

[0041] (3) Calculate the three-dimensional spatial offset distance: An event listener is constructed to respond in real time to the user's triggered actions (index finger directly touching the target or hand ray completing a pinching motion). At the precise moment (frame) of the triggered event, the coordinates of the current hand interaction point are recorded. Subsequently, the offset is calculated using the Euclidean distance formula in three-dimensional space.

[0042] Specifically, for near-field interaction, the spatial straight-line distance between the index fingertip coordinates and the reference center point coordinates is calculated; for far-field interaction, the spatial straight-line distance between the ray projection intersection point coordinates and the reference center point coordinates is calculated. First, obtain the ray projection point object of the right-hand controller: rr = GameObject.Find("MRTK XR Rig / Camera Offset / MRTK RightHandController / Far Ray / RayReticle"); The right index finger touch point object of the right-hand controller: rf = GameObject.Find("MRTK XR Rig / Camera Offset / MRTK RightHandController / IndexTip PokeInteractor / PokeReticle"); Then, obtain the position of the virtual target in virtual space: Vector3 targetWorldPos = button.transform.position; Get the position of the ray projection point of the right-hand controller in virtual space: Vector3 rrloc = rr.transform.position; The position of the right index finger touch point of the right-hand controller in virtual space: Vector3 rfloc = rf.transform.position; Finally, calculate the distance: float distance = Vector3.Distance(activeClickWorldPos, targetWorldPos); activeClickWorldPos is the valid click position. When the test mode is circular or square, activeClickWorldPos=rfloc; when the test mode is bar, activeClickWorldPos=rrloc.

[0043] (4) Standardization of physical dimensions: Obtain the calculated spatial straight-line distance value. Since the default physical unit of measurement in the Unity 3D virtual engine's world coordinate system is the meter (m), this value needs to be converted to a different dimension.

[0044] The spatial straight-line distance value is converted from the system's meter-level units to a millimeter-level physical offset distance that conforms to medical quantitative analysis standards by multiplying it by a scaling factor (e.g., multiplying by 100 to convert to centimeters, or multiplying by 1000 to convert to millimeters). This final value is then written into a structured text record along with the timestamp of the current test.

[0045] Step 55: After the current test task is completed, the collected subject information, spatial offset distance and response time are packaged into a structured text record and stored. In response to the reset command of the augmented reality device or the PC control backend, the current test state is cleared and the system returns to the initial standby interface.

[0046] During the test, the PC-side operation log module centrally displays system feedback in the bottom area. All user operations (such as connection testing, data synchronization, sending test commands, etc.) will call the unified AddLog function to append a timestamped log record in the format of "[HH:MM:SS] message content". The log record is automatically assigned green (success), red (failure / abnormality), or cyan (normal information) according to the operation result to achieve visual differentiation. The container automatically scrolls to the bottom to ensure that the latest messages are visible.

[0047] After the test is completed, the distance of each interaction from the center point offset and the reaction time are automatically extracted and merged with the input subject information. This data is then recorded as text in the same directory where the application is installed on the HoloLens2 device, with the file naming strictly following the format "Number-Test Type-Test Date". Finally, the test subject clicks "Back" on the settlement page on the HoloLens2 device, or a professional user clicks the "Game Reset / Return to Main Interface" button on the PC to clear the current test on the HoloLens2 device and return to the initial interface.

[0048] The text file records in detail the subject's basic personal information and the 15 interaction data generated during the test. Each data item precisely corresponds to the spatial offset distance (unit: mm) and reaction time (unit: seconds) generated by a single click. It demonstrates the system's method of storing real-time hand interaction behavior as structured clinical assessment data, providing quantitative and traceable data support for the assessment system to subsequently execute grading diagnosis rules based on the average error value.

[0049] As one embodiment, the hardware required for this disclosure is a PC host and a HoloLens2 device.

[0050] HoloLens 2, developed by Microsoft, is a mixed reality headset based on augmented reality technology, providing users with a sensory experience that transcends reality. HoloLens 2 utilizes computer graphics and visualization techniques to generate virtual objects that do not exist in the real environment, and then uses sensing technology to accurately "place" these virtual objects within the real environment. In this embodiment, HoloLens 2 uses a new Time-of-Flight (TOF) depth sensor to acquire the grid of the real environment in real time through spatial mapping technology, providing underlying support for setting spatial anchor points. Furthermore, this system integrates the MRTK2 hand tracking and interaction framework, allowing users to manipulate the holographic target through natural gestures such as direct clicking (near-field interaction) or ray pinching (far-field interaction). Leveraging the deep integration of the HoloLens 2 articulated hand algorithm into MRTK2, quantitative acquisition of visuospatial motion data is achieved, significantly improving the interactive accuracy and data reliability of the evaluation process.

[0051] In this embodiment, the PC host is a Lenovo Legion Y7000 9th generation Intel Core i7 laptop. Other models can also be selected for the augmented reality device and PC host, depending on the requirements.

[0052] HoloLens 2 supports wireless wear; users simply wear it on their head and adjust the lens position to easily view the mixed reality test scenarios constructed by the system. When users place their hands within the camera's field of view, HoloLens 2 automatically recognizes their shape and provides interactive support. The HoloLens 2's main interface includes input boxes for ID, name, gender, age, and education level, as well as a dropdown selection box for test type and flow control buttons at the bottom, designed to meet the visual recognition needs of elderly patients.

[0053] The PC-based control interface is an experimental control backend for HoloLens, integrating device connection, test subject information synchronization, circular and square bar space anchor point settings, test process control buttons, and an operation log monitoring area at the bottom, providing professional users with an intuitive and convenient means of remote operation.

[0054] Example 2 One embodiment of this disclosure provides a quantitative analysis system for visuospatial functions based on augmented reality, comprising: The three-dimensional spatial anchor point positioning module is used to acquire the depth data of the real space where the test user is located and convert it into a spatial triangular mesh in the three-dimensional virtual engine; obtain the real physical plane based on the spatial triangular mesh, and obtain the world coordinates in the three-dimensional virtual engine based on the real physical plane; generate corresponding spatial anchor points in the three-dimensional virtual engine based on the world coordinates, the spatial anchor points including horizontal spatial anchor points and vertical spatial anchor points; respond to the user's adjustment operations on the first space and the second space on the PC control terminal to realize the precise anchoring of the three-dimensional virtual object on the real horizontal physical desktop and the real vertical physical wall; The visual space quantitative testing module is used to generate a preset number of virtual test targets based on the test task and anchoring results after entering the testing phase, capture the corresponding touch interaction operations of the test user, and calculate the physical offset distance generated by the interaction in real time.

[0055] As one embodiment, the visual spatial quantitative testing module includes an information input and synchronization module, a test task distribution module, a guidance and teaching module, an interactive acquisition and process execution module, and a process reset control module.

[0056] Among them, the information entry and synchronization module is used to obtain the test number, name, gender, age and education level of the test subjects, and to complete the data interaction and synchronization between the augmented reality device and the PC control backend. The test task distribution module is used to respond to the test type selection instructions from the test subject or the PC control backend, and load the corresponding visual space function test tasks in the virtual space. The test tasks include circular center point test, circular center point test and bar center point test. The guidance and instruction module is used to present the initial instructional targets in a virtual space before the formal test, guiding the test subjects to complete preset hand interaction actions to verify the feasibility of device operation. The interactive acquisition and process execution module is used to generate a preset number of virtual test targets on the surface of a fixed spatial anchor point according to the selected test task during the formal testing phase, capture the corresponding touch interaction operations of the test user, and collect the spatial offset distance from the center point of the target and the response time of each interaction in real time. The process reset control module is used to package and store the collected subject information, spatial offset distance and response time into a structured text record after the current test task is completed, and to respond to the reset command on the augmented reality device or PC control backend to clear the current test state and return to the initial standby interface.

[0057] Example 3 One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the augmented reality-based visuospatial function quantitative analysis method.

[0058] Example 4 One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the augmented reality-based visuospatial function quantitative analysis method.

[0059] Example 5 One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the augmented reality-based visuospatial function quantitative analysis method.

[0060] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A quantitative analysis method for visuospatial functions based on augmented reality, characterized in that, include: Acquire the real-world spatial depth data of the test users and convert it into a spatial triangular mesh in a 3D virtual engine; The real physical plane is obtained based on the spatial triangular mesh, and the world coordinates in the 3D virtual engine are obtained by transforming the real physical plane. Based on world coordinates, corresponding spatial anchor points are generated in the 3D virtual engine. The spatial anchor points include horizontal spatial anchor points and vertical spatial anchor points. Responding to user adjustments to the first and second spaces on the PC control panel, it enables precise anchoring of 3D virtual objects on real horizontal physical desktops and real vertical physical walls; After entering the testing phase, a preset number of virtual test targets are generated based on the test task and anchoring results, capturing the corresponding touch interaction operations of the test users and calculating the physical offset distance generated by the interaction in real time. The process of generating a preset number of virtual test targets based on the test task and anchoring results, capturing the corresponding touch interaction operations of the test user, and calculating the physical offset distance generated by the interaction in real time includes: After each virtual test target is generated, the transformation component data of the current target is obtained through the 3D virtual engine, and the 3D coordinates of its geometric center point in the augmented reality world coordinate system are extracted as the spatial reference center point for this interaction. The articulated hand tracking algorithm of MRTK2 is called to capture the three-dimensional coordinates of the fingertip joint of the right index finger of the test user in the world coordinate system in real time; or the hand ray projection component of MRTK2 is called to obtain the three-dimensional coordinates of the intersection of the far-field ray emitted by the right index finger of the test user and the projection of the vertical wall spatial anchor point grid plane in the world coordinate system in real time through physical ray detection. At the precise moment the event is triggered, record the coordinates of the current hand interaction point and use the Euclidean distance formula in three-dimensional space to calculate the offset; Multiply the spatial straight-line distance by a scaling factor to convert it from the system's meter-level units to a physical offset distance with millimeter-level precision.

2. The method for quantitative analysis of visuospatial functions based on augmented reality as described in claim 1, characterized in that, The process of acquiring the real-world spatial depth data of the test user and converting it into a spatial triangular mesh in a 3D virtual engine includes: Using the TOF depth sensor and camera built into the augmented reality device, the surrounding test environment of the test subject is scanned in real time, and a grid composed of triangles as basic units is attached to the surface of the surrounding objects.

3. The method for quantitative analysis of visuospatial functions based on augmented reality as described in claim 1, characterized in that, The process of obtaining the real physical plane based on a spatial triangular mesh and transforming it to obtain world coordinates in the 3D virtual engine includes: The mesh recognition viewer provided by MRTK2 is used to obtain triangular mesh data in real space, and the local coordinate system of the recognized mesh is converted into the world coordinate system in the Unity 3D virtual engine through a coordinate transformation algorithm.

4. The method for quantitative analysis of visuospatial functions based on augmented reality as described in claim 1, characterized in that, In response to user adjustments to the first space on the PC control terminal, the system acquires horizontal movement step parameters, translates and fixes the horizontal spatial anchor point within the virtual space, achieving precise anchoring of the 3D virtual object on the real horizontal physical desktop; in response to user adjustments to the second space on the PC control terminal, the system acquires vertical movement step and rotation step parameters, translates and rotates the vertical spatial anchor point within the virtual space to adapt to the tilt angle of the wall, achieving precise anchoring of the 3D virtual object on the real vertical physical wall.

5. The method for quantitative analysis of visuospatial functions based on augmented reality as described in claim 1, characterized in that, After entering the testing phase, the test number, name, gender, age, and education level of the test subjects are obtained, and data interaction and synchronization are completed between the augmented reality device and the PC control backend. The corresponding visual space function test tasks are loaded in the virtual space. The test tasks include circular center point test, square center point test, and bar center point test. The adjustment operation of the first space is used to adapt to the circular center point test process in the testing phase, and the adjustment of the second space is used to adapt to the square center point and bar center point test processes in the testing phase.

6. A quantitative analysis system for visuospatial functions based on augmented reality, characterized in that, Specifically, the method for quantitative analysis of visuospatial functions based on augmented reality as described in any one of claims 1-5 includes: The three-dimensional spatial anchor point positioning module is used to acquire the depth data of the real space where the test user is located and convert it into a spatial triangular mesh in the three-dimensional virtual engine; obtain the real physical plane based on the spatial triangular mesh, and obtain the world coordinates in the three-dimensional virtual engine based on the real physical plane; generate corresponding spatial anchor points in the three-dimensional virtual engine based on the world coordinates, the spatial anchor points including horizontal spatial anchor points and vertical spatial anchor points; respond to the user's adjustment operations on the first space and the second space on the PC control terminal to realize the precise anchoring of the three-dimensional virtual object on the real horizontal physical desktop and the real vertical physical wall; The visual space quantitative testing module is used to generate a preset number of virtual test targets based on the test task and anchoring results after entering the testing phase, capture the corresponding touch interaction operations of the test user, and calculate the physical offset distance generated by the interaction in real time.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the augmented reality-based visuospatial function quantitative analysis method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the augmented reality-based visuospatial function quantitative analysis method as described in any one of claims 1-5.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the augmented reality-based visuospatial function quantitative analysis method as described in any one of claims 1-5.

Images