Postoperative cognitive impairment risk prediction method and system based on augmented reality
By using augmented reality technology and the XGBoost model to determine target AR scenarios in multiple AR test scenarios, and combining data feature extraction from image acquisition devices and AR devices, the problem of accuracy in predicting postoperative cognitive impairment risk was solved, and adaptive assessment of patients' activity limitations was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU YUHANG DISTRICT NO 5 PEOPLES HOSPITAL
- Filing Date
- 2026-02-07
- Publication Date
- 2026-05-15
AI Technical Summary
Existing methods for predicting the risk of cognitive impairment cannot accurately adapt to the complex situation of patients with limited mobility after surgery, and traditional medical data cannot effectively predict higher-order cognitive dysfunction.
By employing augmented reality technology combined with the XGBoost model and multimodal feature fusion, the target AR scene is determined in multiple test scenarios through image acquisition equipment and AR devices. The test data of patients is acquired, and feature extraction and decision generation are performed to output the risk prediction results of cognitive dysfunction.
It can adapt to the complex situation of limited activity in postoperative patients, improve the accuracy and adaptability of cognitive impairment risk prediction, and reduce the error of human experience judgment.
Smart Images
Figure CN122050830A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of smart healthcare technology, and in particular to an augmented reality method and system for predicting postoperative cognitive impairment risk. Background Technology
[0002] Postanesthesia cognitive impairment (POCD) is associated with a variety of factors. For example, anesthetic drugs inhibit the central nervous system via GABA receptors, which may trigger a neuroinflammatory cascade (such as factors like IL-6 and TNF-α crossing the blood-brain barrier). Individual physiological factors, such as genetic polymorphisms (like CYP450 variants), decreased brain reserve function in the elderly, and comorbidities (such as diabetes and hypertension), can simultaneously increase the risk of neuronal damage, leading to impaired cognitive functions such as memory and attention after surgery.
[0003] In related technologies, methods for predicting the risk of cognitive impairment have emerged, such as patent number CN 120148867 A, entitled "An Intelligent Early Warning System for Post-Anesthesia Complications in Surgical Patients." This system predicts whether patients will experience complications, including cognitive impairment, by collecting data such as preoperative medical history, intraoperative physical indicators, and postoperative physical indicators and consciousness assessment scores. However, this approach uses traditional medical data and aims to monitor the impact of anesthesia on basic vital signs and overall clinical status, making it unable to accurately predict higher-order cognitive functions. Augmented Reality (AR) technology is increasingly being applied to the field of cognitive impairment prediction. Patent number CN120148867 A, entitled "Screening and Assessment System for Cognitive Impairment in the Elderly," proposes to predict cognitive impairment by collecting EEG signals, VR / AR interactive behavior data, gait parameters, and neurotransmitter level data combined with relevant algorithms. However, this method is suitable for long-term observation of the elderly in nursing homes and cannot adapt to the complex and ever-changing cognitive impairment prediction of postoperative patients. Summary of the Invention
[0004] This application provides an augmented reality method and system for predicting the risk of postoperative cognitive impairment, which breaks through the limitations of traditional single medical data and can adapt to the complex situation of patients' limited activity after surgery.
[0005] The first aspect of this application provides an augmented reality method for predicting the risk of postoperative cognitive impairment, the prediction method including: When determining the permissible assessment of cognitive functions based on feasibility labels, medical monitoring data and AR interaction behavior types corresponding to feasibility labels are input into a pre-trained XGboost model to identify the target AR scenario in multiple AR test scenarios. Acquire test data of patients in target AR scenes. The test data includes the device location information of the image acquisition device, test images of the patient test, and AR behavioral interaction data. The device location information is different in each AR test scene. The image recognition module identifies the image features of the test image, maps the device position information through a linear transformation layer, and activates it through a rectified linear unit to generate a view weight vector that matches the dimension of the image feature channel. The view weight vector and the image features are then fused to obtain the calibrated image features. AR behavioral interaction data and medical monitoring data are processed by corresponding feature extraction modules to extract features. The resulting AR behavioral features, medical monitoring features, and calibrated image features are then jointly analyzed and used to generate decisions, outputting risk prediction results for cognitive impairment.
[0006] In some embodiments, the multiple AR test scenarios include a first AR test scenario, a second AR test scenario, and a third AR test scenario. The image acquisition device corresponding to the first AR test scenario is integrated into the AR device, the image acquisition device corresponding to the second AR test scenario is installed on the wall directly in front of the hospital bed, and the image acquisition device corresponding to the third AR test scenario is installed on the intravenous infusion stand or the ceiling.
[0007] In some embodiments, AR behavioral interaction data and medical monitoring data are respectively processed by corresponding feature extraction modules to extract features. The resulting AR behavioral features, medical monitoring features, and calibrated image features are jointly analyzed and used for decision generation to output risk prediction results for cognitive impairment, including: The calibrated image features, AR behavioral features, and time-extended medical monitoring features are concatenated to construct a multimodal time-series feature sequence; The multimodal temporal feature sequence is input into the spatiotemporal Transformer encoder for joint encoding, and the encoded high-level feature sequence is output. The encoded high-level feature sequence is then subjected to global average pooling along the time dimension to obtain a global feature vector that characterizes the entire evaluation process. The global feature vector is input into the classifier to calculate the probability distribution of each preset cognitive impairment risk level, and the level with the highest probability is taken as the final risk prediction result.
[0008] In some embodiments, medical monitoring data and AR interaction behavior types corresponding to feasibility labels are input into a pre-trained XGboost model to determine the target AR scene in multiple AR test scenarios, including: The patient's baseline data, surgical trauma data, postoperative status data, and feasibility tags are received and constructed into a structured feature vector, which includes subtle interactions, local interactions, or systemic interactions. The feature vector is input into the pre-trained XGBoost multi-class model. The model performs non-linear mapping and decision-making on the input features through the gradient boosting tree structure, and outputs the probability distribution of sub-scenes corresponding to multiple cognitive task difficulties in multiple AR test scenarios. The sub-scene with the highest probability value in the probability distribution is identified as the target AR scene.
[0009] In some embodiments, patient baseline data include age, years of education, and APOE genotype; surgical trauma data includes operation duration, blood loss, and cumulative duration of hypotension; and postoperative status data includes inflammatory marker levels.
[0010] In some embodiments, the method further includes: Acquire medical diagnostic data and match it with preset security screening rules, outputting a feasibility label for whether cognitive function assessment is allowed, as well as the AR interaction behavior type corresponding to the feasibility label.
[0011] In some embodiments, medical diagnostic data includes surgical type and medical order restrictions. The medical diagnostic data is matched using preset safety screening rules to output a feasibility label indicating whether cognitive function assessment is permitted, and the corresponding AR interaction behavior type for the feasibility label, including: If the patient's surgical procedure is determined to be a pre-defined high-risk procedure, a prohibition label for assessing cognitive function will be output. If the patient's surgical procedure is determined to be a non-preset high-risk procedure, the AR interaction behavior type is determined based on the doctor's orders, including subtle part interaction, local part interaction, or whole-body interaction. The output is a permission label that allows assessment of cognitive function and the corresponding AR interaction behavior type.
[0012] In some embodiments, the device location information is mapped through a linear transformation layer and activated by a rectified linear unit to generate a viewpoint weight vector that matches the dimension of the image feature channels. The mathematical expression for this is: in, Represents the viewpoint weight vector, L represents the weight matrix, and L represents the device location information. Represents the bias vector, Represents a rectifier linear unit. Dimensions representing image features This represents a linear transformation layer.
[0013] In some embodiments, the mathematical expression for obtaining the calibrated image features by fusing the viewpoint weight vector and image features is: in, Indicates the features of the calibrated image and Indicates image features.
[0014] The second aspect of this application provides an augmented reality postoperative cognitive impairment risk prediction system, characterized in that the system includes a server, an image acquisition device, and an AR device; The server is used to obtain medical monitoring data and AR interaction behavior types corresponding to feasibility labels when determining whether cognitive function can be assessed based on feasibility labels. The input of these data into the pre-trained XGboost model determines the target AR scene in multiple AR test scenarios. And to acquire patient test data in target AR scene, the test data includes the original test image of patient test and the original device location information sent by the image acquisition device, as well as the device location information, patient test image and AR behavior interaction data obtained by the original AR behavior interaction data sent by the AR device and after feature preprocessing. Among them, the device location information is different in each AR test scene. The image features of the test image are identified by the image recognition module, the device position information is mapped through the linear transformation layer and activated by the rectified linear unit to generate a view weight vector that matches the dimension of the image feature channel. The view weight vector and the image features are fused to obtain the calibrated image features. The system extracts features from AR behavioral interaction data and medical monitoring data through corresponding feature extraction modules, and then performs joint analysis and decision generation on the obtained AR behavioral features, medical monitoring features, and calibrated image features to output risk prediction results for cognitive impairment.
[0015] Understandably, the augmented reality postoperative cognitive impairment risk prediction method and system provided in this application, when determining whether cognitive function can be assessed based on feasibility labels, obtains medical monitoring data and AR interactive behavior types corresponding to feasibility labels and inputs them into a pre-trained XGboost model. The target AR scene is determined among multiple AR test scenarios, and the target AR test scene is automatically matched by the XGBoost model, which can adapt to complex situations such as limited activity of postoperative patients (e.g., only eye rotation or local movement).
[0016] Next, test data is acquired, including the device location information of the image acquisition device, test images of the patient, and AR behavioral interaction data, for the patient in the target AR scene. To address the differences in device location across different AR scenes, a rectified linear unit is used to generate a viewpoint weight vector to eliminate viewpoint bias, ensuring the consistency of image features across different scenes. This allows patients to directly conduct tests in different AR scenarios within their wards, meeting the needs of postoperative clinical settings. Finally, based on joint analysis and decision generation using AR behavioral features, medical monitoring features, and calibrated image features, a risk prediction result for cognitive impairment is output. This approach overcomes the limitations of traditional single medical data and can adapt to complex situations such as postoperative activity restrictions for patients. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] Figure 1 A schematic diagram of the structure of the augmented reality postoperative cognitive impairment risk prediction system provided in the embodiments of this application; Figure 2 A flowchart illustrating an augmented reality method for predicting postoperative cognitive impairment risk, as provided in an embodiment of this application. Figure 3 A system framework diagram of the prediction model in the augmented reality postoperative cognitive impairment risk prediction method provided in the embodiments of this application; Figure 4 This is another schematic flowchart illustrating the augmented reality method for predicting postoperative cognitive impairment risk provided in this application. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation
[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.
[0020] The terms “first”, “second”, etc. used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.
[0021] Please see Figure 1 , Figure 1This application provides an augmented reality postoperative cognitive impairment risk prediction system 100, which includes a server 10, an image acquisition device 20, and an AR device 30. The augmented reality postoperative cognitive impairment risk prediction method provided in this application is applied to the prediction system 100.
[0022] The technical solution of this application and how the technical solution of this application solves the technical problem are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0023] Please see Figures 1 to 3 , Figure 2 A flowchart illustrating the augmented reality-based postoperative cognitive impairment risk prediction method provided in this application is shown. The predictive system is the executing entity for this method, which can specifically be a server, such as... Figure 2 As shown, the prediction method may include the following steps: Step S210: When determining whether cognitive function can be assessed based on feasibility labels, obtain medical monitoring data and AR interaction behavior types corresponding to feasibility labels and input them into the pre-trained XGboost model to determine the target AR scene among multiple AR test scenarios.
[0024] Specifically, medical monitoring data can include patient baseline data consisting of data such as age, years of education, and APOE genotype; surgical trauma data consisting of data such as operation duration, blood loss, and cumulative duration of hypotension; and postoperative status data consisting of data such as inflammatory marker levels. AR interaction behavior types include subtle body interactions completed through localized fine movements such as the eyes and fingers; localized body interactions involving activities in localized areas such as the upper limbs and head; or whole-body interactions requiring coordinated movements of the trunk, lower limbs, and other parts of the body. Multiple AR test scenarios include a first AR test scenario (i.e., subtle body interaction scenario), a second AR test scenario (i.e., localized interaction scenario), and a third AR test scenario (i.e., whole-body interaction scenario), with each AR test scenario containing multiple sub-scenarios of varying cognitive task difficulty. A target AR scenario refers to a specific sub-scenario within a certain type of AR test scenario.
[0025] The training process of the XGBoost model is based on historical medical data and AR interaction behavior records. The specific steps are as follows: First, collect postoperative patient characteristic data: patient baseline data (age, years of education, APOE genotype, etc.), surgical trauma data (operation duration, blood loss, cumulative duration of hypotension, etc.), postoperative status data (inflammatory marker levels, etc.), and corresponding AR interaction behavior type labels (detailed area interaction, local interaction, or whole-body interaction). Label data: the AR test scenarios actually applicable to each patient during the assessment (sub-scenes of the detailed area interaction scenario, sub-scenes of the local interaction scenario, or sub-scenes of the whole-body interaction scenario).
[0026] Next, the data is preprocessed: the feature data is transformed into numerical features, and structured feature vectors are constructed. The AR test scene is encoded into multi-class labels, and training and validation sets are created.
[0027] Then, the XGBoost multi-class classification algorithm is used for model training: model parameters are initialized, and hyperparameters such as the number of trees, depth, and learning rate are set. The decision tree is iteratively built using a gradient boosting framework. In each iteration, the model calculates the gradient of the loss function between the current prediction and the real-world label, and generates a new tree based on the gradient information to correct the residuals. During tree generation, the optimal feature split is selected through feature splitting gain, and a regularization term is introduced to control model complexity and prevent overfitting. Finally, the hyperparameters are adjusted based on the validation set performance to optimize the model's generalization ability, resulting in a pre-trained XGBoost model.
[0028] In the prediction phase of the pre-trained XGBoost model, the patient's baseline data, surgical trauma data, postoperative status data, and feasibility labels are first received to construct a structured feature vector, which is then constructed from the subtle interactions, local interactions, or whole-body interactions corresponding to the feasibility labels. Next, the feature vector is input into the pre-trained XGBoost multi-class model. The model performs non-linear mapping and decision-making on the input features through a gradient boosting tree structure, and outputs the probability distribution of sub-scenes corresponding to multiple cognitive task difficulties in multiple AR test scenarios. Finally, the sub-scene with the highest probability value in the probability distribution is determined as the target AR scene.
[0029] For example, the system receives patient baseline data, surgical trauma data, postoperative status data, and structured feature vectors generated from eye-only interactions (i.e., subtle part interactions) corresponding to feasibility labels. These feature vectors are then input into a pre-trained XGBoost multi-classification model, which outputs an eye gaze tracking and recognition scenario (i.e., a sub-scenario of moderate difficulty within the subtle part interaction scenario). The AR interface of the eye gaze tracking and recognition scenario displays randomly moving virtual targets (such as colored balls). The patient must follow the target's trajectory by gazing at it. The AR device records AR interaction behavior data such as the accuracy of the user's gaze point, reaction time, and trajectory smoothness.
[0030] Understandably, by integrating patient baseline data (age, years of education, etc.), surgical trauma data (operation duration, blood loss, etc.), and postoperative status data (inflammatory marker levels, etc.), and combining this with the AR interaction behavior type (minor / local / full-body interaction) corresponding to the feasibility label, it is possible to match the most suitable AR test scenario for different patients who can only perform minor movements such as eye and finger movements, local movements such as only upper or lower body movements, or full-body movements, thereby improving the targeting of the assessment. Using a pre-trained XGBoost multi-classification model, a gradient boosting tree structure is used to perform non-linear mapping and decision-making on the input features, outputting the probability distribution of multiple sub-scenes with different cognitive task difficulties in various AR test scenarios. Finally, the sub-scene with the highest probability is selected, reducing errors from human experience-based judgment and improving the scientific rigor and objectivity of scenario selection.
[0031] Step S220: Obtain the patient's test data in the target AR scene. The test data includes the device location information of the image acquisition device, the test images of the patient, and AR behavioral interaction data.
[0032] Specifically, the device location information differs in each AR testing scenario. For example, in scenarios involving subtle interactions, the image acquisition device is integrated into the AR device. In scenarios involving partial interactions, the image acquisition device is mounted on the wall directly in front of the bed. In scenarios involving full-body interactions, the image acquisition device is mounted on an IV stand or the ceiling. This allows patients to directly conduct tests in different AR scenarios within their ward, meeting the needs of postoperative clinical settings by strategically positioning the image acquisition device.
[0033] The image acquisition device sends its original location information and raw test images of the patient to the server, while the raw AR behavioral interaction data is sent from the AR device to the server. For example... Figure 3 As shown, the server performs feature preprocessing on the received raw data to obtain test data that can be input into the prediction model built based on Transformer, namely device location information, test images of patients, and AR behavioral interaction data.
[0034] Understandably, test images from patient testing can synergize with AR behavioral interaction data to accurately pinpoint the root cause of the problem. Relying solely on AR behavioral interaction data (such as a high error rate in the order of dragging virtual objects) might lead to vague reasons such as poor memory, insufficient command comprehension, lack of concentration, or motor impairment. However, by combining test images from patient testing (videos, facial expressions, and posture analysis), specific manifestations such as shifty eyes (attention deficit), frequent confused expressions (difficulty in command comprehension), and smooth but incorrect hand movements (excluding motor impairment, pointing to cognitive problems) can pinpoint the problem to "attention deficit combined with difficulty in command comprehension," rather than the general "task failure," thus making the risk assessment more accurate.
[0035] Step S230: The image features of the test image are identified by the image recognition module, the device position information is mapped through the linear transformation layer and activated by the rectified linear unit to generate a view weight vector that matches the dimension of the image feature channel. The view weight vector and the image features are fused to obtain the calibrated image features.
[0036] Specifically, such as Figure 3 As shown, the image recognition module can be a lightweight Vision Transformer model (the model expression is...). ,in, Indicates image features, This represents the test image, where H represents the image height, W represents the image width, and 3 represents the number of RGB channels. This represents a lightweight Vision Transformer model. Indicates the number of time steps. Preset, (Represents the dimension of image features). The image features output by the image recognition module are input into a calibration fusion layer that includes the following two mathematical expressions, and the output is the calibrated image features.
[0037] ReLU (Rectified Linear Unit) is a commonly used activation function in neural networks. Combined with a linear transformation layer, the mathematical expression for generating the viewpoint weight vector can be obtained as follows: in, Represents the viewpoint weight vector, Represents the weight matrix, This indicates the device location information, specifically the position and orientation of the image acquisition device. This represents the bias vector. Represents a rectifier linear unit. This represents a linear transformation layer.
[0038] The mathematical expression for obtaining the calibrated image features by fusing the viewpoint weight vector and image features is as follows: in, This indicates the features of the calibrated image.
[0039] Understandably, by rectifying the device position information using a Rectified Linear Unit (ReLU), a viewpoint weight vector matching the image feature dimensions is generated, effectively eliminating viewpoint bias caused by differences in the position of image acquisition devices in different AR testing scenarios. For example, when image acquisition devices are integrated into AR devices (for subtle interaction scenarios), mounted on walls (for local interaction scenarios), or mounted on ceilings (for full-body interaction scenarios), this mechanism can dynamically adjust feature weights to ensure consistency of image features under different viewpoints. Addressing potential activity limitations for postoperative patients (such as only being able to make subtle or localized movements), the calibrated image features can be adapted to different AR interaction scenarios (subtle / local / full-body), ensuring accurate capture of patients' cognitive and behavioral data even when device position is limited.
[0040] Step S240: Extract features from AR behavioral interaction data and medical monitoring data through corresponding feature extraction modules. Perform joint analysis and decision generation on the obtained AR behavioral features, medical monitoring features, and calibrated image features to output risk prediction results for cognitive impairment.
[0041] Specifically, AR behavioral interaction data is input into the first feature extraction module to extract features and obtain AR behavioral features, and medical monitoring data is input into the second feature extraction module to extract features and obtain medical monitoring features.
[0042] The expression for the first feature extraction module is: ,in, Indicates AR behavioral characteristics, Indicates AR behavioral interaction data, This represents the weight matrix of the module. This represents the bias vector of the module. Representing the dimensions of the data This represents the hidden dimension of the prediction model.
[0043] The expression for the second feature extraction module is: ,in, Indicates medical monitoring characteristics, Indicates fully connected layer, This indicates medical monitoring data.
[0044] Understandably, in the above technical solution, when determining the permissible assessment of cognitive function based on feasibility labels, medical monitoring data and AR interaction behavior types corresponding to feasibility labels are input into the pre-trained XGboost model. The target AR scene is determined in multiple AR test scenarios. The target AR test scene is automatically matched by the XGBoost model, which can adapt to the complex situation of postoperative patients with limited activity (such as only being able to rotate the eyes or perform local movements).
[0045] Next, test data is acquired, including the device location information of the image acquisition device, test images of the patient, and AR behavioral interaction data, within the target AR scene. To address the differences in device location across different AR scenes, a rectified linear unit is used to generate a viewpoint weight vector to eliminate viewpoint bias, ensuring consistency of image features across different scenes. This allows patients to directly conduct tests in different AR scenarios within their wards, meeting the needs of postoperative clinical settings. Finally, features are extracted from the AR behavioral interaction data and medical monitoring data using corresponding feature extraction modules. The resulting AR behavioral features, medical monitoring features, and calibrated image features are then jointly analyzed and used for decision generation, outputting a risk prediction result for cognitive impairment. This approach overcomes the limitations of traditional single medical data while adapting to the complex situation of patients with limited postoperative activity.
[0046] In some embodiments, such as Figure 3 As shown, step S240: Features are extracted from the AR behavioral interaction data and medical monitoring data using their respective feature extraction modules. The resulting AR behavioral features, medical monitoring features, and calibrated image features are then jointly analyzed and used for decision generation to output a risk prediction result for cognitive impairment. This includes the following steps: Step S241: Concatenate the calibrated image features, AR behavioral features, and time-extended medical monitoring features to construct a multimodal time-series feature sequence.
[0047] Specifically, through the linear projection layer ( , This refers to aligning and calibrating image features, and then performing temporal extension on medical monitoring features (i.e.,...). , (This represents the time-series extended medical monitoring features), and then, they are concatenated through a feature concatenation layer. The expression for the feature concatenation layer is: ,in, Represents a multimodal temporal feature sequence. Represents the position encoding matrix, Sine-cosine coding is used as the splicing feature. Provide location information, location coding matrix The calculation method is as follows: For position and dimensional index , when When it is even: , when When it is an odd number: In this way, all modal features share the same positional encoding at the same time step, ensuring temporal alignment.
[0048] Step S242: Input the multimodal temporal feature sequence into the spatiotemporal Transformer encoder for joint encoding, output the encoded high-level feature sequence, and perform global average pooling on the encoded high-level feature sequence along the time dimension to obtain a global feature vector used to characterize the entire evaluation process.
[0049] Specifically, the mathematical expression for the spatiotemporal Transformer encoder is as follows: in, Representation layer index ( hour, ), . Presentation layer normalization operation, Indicates feedforward network ( It consists of two fully connected layers with a rectified linear unit in between. (Composition). When When, output This is the encoded high-level feature sequence.
[0050] Next, the encoded high-level feature sequence (High-level feature sequences) Features of splicing multiple time steps t Composition, that is Global average pooling is performed on the concatenated features that make up the Lth layer at time step t. After that, the global feature vector is obtained. .
[0051] Step S243: Input the global feature vector into the classifier, calculate the probability distribution of each preset cognitive impairment risk level, and take the level with the highest probability as the final risk prediction result.
[0052] Specifically, applying the standard Softmax classifier, the expression for the classifier can be obtained as follows: , Where s represents the probability distribution of each preset risk level of cognitive impairment, including the probability of a normal cognitive state. The probability of mild cognitive impairment The probability of dementia . Represents the global feature vector. Represents the classification weight matrix, This represents the classification bias vector.
[0053] Understandable, such as Figure 3The training process of the prediction model shown may include the following steps: preparing training samples containing multimodal data, each sample including device location information, test images of the patient, AR behavioral interaction data, and medical monitoring data, and labeling them with corresponding cognitive risk category labels (normal, MCI, or dementia). The forward computation of the training process is the same as the aforementioned prediction process (i.e., steps S230 to S240), and will not be repeated here. The training objective of the model is to minimize the difference between the predicted probability distribution and the true label, that is, to use the cross-entropy loss function to calculate the loss value (quantified value of the difference) between the category label and the predicted probability of the prediction model: in, This represents the loss value, and c represents the category index. c=1 indicates the category is normal, c=2 indicates the category is MCI, and c=3 indicates the category is dementia. Indicates category label, This represents the probability predicted by the prediction model that a class belongs to the c-th category.
[0054] Next, starting with the loss value, backpropagation is used to calculate the loss value relative to each trainable parameter in the prediction model (trainable parameters include the bias vector). , Weight matrix , The gradient of the model is calculated, and then the optimizer (Adam) updates the trainable parameters of the prediction model based on the gradient. This process is repeated iteratively until the model converges or reaches the predetermined number of training rounds, thus completing the training.
[0055] Please see Figure 4 In some embodiments, before determining whether cognitive function assessment is allowed based on the feasibility label in step S210, the prediction method further includes step S110: acquiring medical diagnostic data and matching the medical diagnostic data through preset security screening rules, outputting a feasibility label for whether cognitive function assessment is allowed, and the AR interaction behavior type corresponding to the feasibility label.
[0056] Specifically, the medical diagnostic data includes the type of surgery and medical order restrictions. After acquiring the medical diagnostic data, if the patient's surgery is determined to be a pre-set high-risk surgery, a prohibition label for assessing cognitive function is output. If the patient's surgery is not determined to be a pre-set high-risk surgery, the AR interaction behavior type (detailed area interaction, local area interaction, or whole-body interaction) is determined based on the medical order restrictions, and a permission label for assessing cognitive function and the corresponding AR interaction behavior type are output.
[0057] For example, if the surgery type is "intracranial tumor resection" (pre-set high-risk surgery) and the doctor's order restricts it to "absolute bed rest after surgery," then a prohibition label for assessing cognitive function will be output. This is understandable because the surgery type falls into the high-risk category, disallowing AR interactive assessment.
[0058] Understandably, by setting up safety screening rules, a prohibition label can be directly output for high-risk surgical patients (such as "intracranial tumor resection") to avoid secondary risks that AR interactive assessment may cause to patients.
[0059] For example, if the surgery type is "phacoemulsification cataract surgery" (non-high-risk surgery) and the doctor's order restricts the procedure to "no vigorous head movements within 24 hours after surgery, only slight eye movements are allowed", then the output will be an "allowed" label for evaluation, and the AR interaction behavior type will be "micro-part interaction" (such as eye gaze tracking test).
[0060] Understandably, for patients undergoing non-high-risk surgeries, the type of AR interactive behavior (minor / local / systemic interaction) is dynamically matched according to the doctor's orders. For example, when a patient undergoing "phacoemulsification cataract surgery" is only allowed to move their eyes slightly, "minor part interaction" is matched to ensure that the assessment process is compatible with the patient's postoperative condition.
[0061] Understandably, by using a dual screening process based on surgical type and medical order restrictions, we can ensure that the AR interaction scenario matches the patient's physical condition, reduce assessment errors caused by activity limitations, and provide reliable data input for subsequent risk prediction.
[0062] Understandably, in conjunction with the solution provided in this embodiment, this application first matches medical diagnostic data (including surgical type and medical order restrictions) through preset safety screening rules, outputting a feasibility label for whether cognitive function assessment is allowed and the corresponding AR interaction behavior type (detailed part interaction, local part interaction, or whole-body interaction). This eliminates inappropriate testing for high-risk patients from the source and limits the feasible interaction range for subsequent steps. Next, the pre-trained XGboost model receives the feasibility label and the corresponding AR interaction behavior type, and, combined with the patient's medical monitoring data (including patient baseline data, surgical trauma data, and postoperative status data), intelligently determines the target AR scene from multiple AR test scenarios (such as the first AR test scenario, the second AR test scenario, and the third AR test scenario). This solves the problem of how to conduct assessments for specific patients, ensures the feasibility and relevance of the test, and determines the device location information of the image acquisition device. Then, the viewpoint calibration module uses the device location information corresponding to the selected target AR scene, rectifies the device location information through a rectified linear unit, generates a viewpoint weight vector, and then fuses it with the image features extracted by the image recognition module to obtain calibrated image features. This solves the problem of "inconsistent visual features" caused by differences in device deployment across different scenarios, providing accurate visual input for subsequent multimodal feature fusion. Finally, the calibrated image features, AR behavioral features, and medical monitoring features are deeply fused, and a classifier calculates the final output of a risk prediction result for cognitive impairment, which can adapt to complex situations such as postoperative activity limitations in patients.
[0063] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. An augmented reality method for predicting the risk of postoperative cognitive impairment, characterized in that, Prediction methods include: When determining the permissible assessment of cognitive functions based on feasibility labels, medical monitoring data and AR interaction behavior types corresponding to feasibility labels are input into a pre-trained XGboost model to identify the target AR scenario in multiple AR test scenarios. Acquire test data of patients in target AR scenes. The test data includes device location information of image acquisition devices, test images of patients, and AR behavioral interaction data. The device location information is different in each AR test scene. The image recognition module identifies the image features of the test image, maps the device position information through a linear transformation layer, and activates it through a rectified linear unit to generate a view weight vector that matches the dimension of the image feature channel. The view weight vector and the image features are then fused to obtain the calibrated image features. AR behavioral interaction data and medical monitoring data are processed by corresponding feature extraction modules to extract features. The resulting AR behavioral features, medical monitoring features, and calibrated image features are then jointly analyzed and used to generate decisions, outputting risk prediction results for cognitive impairment.
2. The method according to claim 1, characterized in that, The multiple AR test scenarios include a first AR test scenario, a second AR test scenario, and a third AR test scenario. The image acquisition device corresponding to the first AR test scenario is integrated into the AR device. The image acquisition device corresponding to the second AR test scenario is installed on the wall directly in front of the hospital bed. The image acquisition device corresponding to the third AR test scenario is installed on an IV stand or on the ceiling.
3. The method according to claim 1, characterized in that, The AR behavioral interaction data and medical monitoring data are processed by corresponding feature extraction modules to extract features. The resulting AR behavioral features, medical monitoring features, and calibrated image features are then jointly analyzed and used for decision generation to output a risk prediction result for cognitive impairment, including: The calibrated image features, AR behavioral features, and time-extended medical monitoring features are concatenated to construct a multimodal time-series feature sequence; The multimodal temporal feature sequence is input into a spatiotemporal Transformer encoder for joint encoding, and the encoded high-level feature sequence is output. The encoded high-level feature sequence is then subjected to global average pooling along the time dimension to obtain a global feature vector that characterizes the entire evaluation process. The global feature vector is input into the classifier to calculate the probability distribution of each preset cognitive impairment risk level, and the level with the highest probability is taken as the final risk prediction result.
4. The method according to claim 1, characterized in that, The acquisition of medical monitoring data and AR interaction behavior types corresponding to feasibility labels are input into a pre-trained XGboost model to determine the target AR scene in multiple AR test scenarios, including: The patient's baseline data, surgical trauma data, postoperative status data, and feasibility tags are received and constructed into a structured feature vector, which includes subtle interactions, local interactions, or systemic interactions. The feature vector is input into a pre-trained XGBoost multi-class model. The model performs non-linear mapping and decision-making on the input features through a gradient boosting tree structure, and outputs the probability distribution of sub-scenes corresponding to multiple cognitive task difficulties in multiple AR test scenarios. The sub-scene with the highest probability value in the probability distribution is identified as the target AR scene.
5. The method according to claim 4, characterized in that, The patient baseline data includes age, years of education, and APOE genotype; the surgical trauma data includes operation duration, blood loss, and cumulative duration of hypotension; and the postoperative status data includes inflammatory marker levels.
6. The method according to claim 1, characterized in that, The method further includes: Acquire medical diagnostic data and match it with preset security screening rules, outputting a feasibility label for whether cognitive function assessment is allowed, and the AR interaction behavior type corresponding to the feasibility label.
7. The method according to claim 6, characterized in that, The medical diagnostic data includes surgical types and medical order restrictions. The data is matched using preset safety screening rules to output a feasibility label for assessing cognitive function, and the corresponding AR interaction behavior type, including: If the patient's surgical procedure is determined to be a pre-defined high-risk procedure, a prohibition label for assessing cognitive function will be output. If the patient's surgical procedure is determined to be a non-preset high-risk procedure, the AR interaction behavior type is determined based on the doctor's orders, including subtle part interaction, local part interaction, or whole-body interaction. The output is a permission label that allows assessment of cognitive function and the corresponding AR interaction behavior type.
8. The method according to claim 1, characterized in that, The mathematical expression for mapping device location information through a linear transformation layer and activating it via a rectified linear unit to generate a viewpoint weight vector matching the image feature channel dimension is as follows: in, Represents the viewpoint weight vector, L represents the weight matrix, and L represents the device location information. Represents the bias vector, Represents a rectifier linear unit. Dimensions representing image features This represents a linear transformation layer.
9. The method according to claim 8, characterized in that, The mathematical expression for the calibrated image features obtained by fusing the viewpoint weight vector and image features is: in, Indicates the features of the calibrated image and Indicates image features.
10. An augmented reality-based system for predicting the risk of postoperative cognitive impairment, characterized in that, The system includes a server, image acquisition equipment, and AR equipment; The server is used to obtain medical monitoring data and AR interaction behavior types corresponding to feasibility labels when determining whether cognitive function assessment is allowed based on feasibility labels, and input them into a pre-trained XGboost model to determine the target AR scene in multiple AR test scenarios. And to acquire patient test data in target AR scene, the test data includes the original test image and original device location information of the patient test sent by the image acquisition device, and the device location information, patient test image and AR behavior interaction data obtained by the original AR behavior interaction data sent by the AR device and after feature preprocessing, wherein the device location information is different in each AR test scene; The system identifies image features of test images through an image recognition module, maps device location information through a linear transformation layer, activates it via a rectified linear unit, generates a viewpoint weight vector matching the dimension of the image feature channel, and fuses the viewpoint weight vector and image features to obtain calibrated image features. Furthermore, it extracts features from AR behavioral interaction data and medical monitoring data through corresponding feature extraction modules, and performs joint analysis and decision generation on the obtained AR behavioral features, medical monitoring features, and calibrated image features to output risk prediction results for cognitive impairment.