A ward intelligent evaluation system and method based on cognitive evaluation robot

Abstract

This invention discloses an intelligent assessment system and method for hospital wards based on a cognitive assessment robot. The intelligent assessment system includes a doctor controller and a robot body. The doctor controller provides a multimodal command input channel, supporting multimodal command input. The robot body is communicatively connected to the doctor controller and integrates a mobile chassis module, a vision module, a speech recognition and synthesis module, a cognitive assessment module, and a large-scale task parsing agent. This large-scale task parsing agent is configured to: receive and parse multimodal commands from the doctor controller or the speech recognition and synthesis module to generate multi-task control signals; drive the mobile chassis module to perform patient positioning and navigation based on the multi-task control signals; control the vision module and the speech recognition and synthesis module to collaboratively complete patient identity verification; and perform cognitive function assessment operations through the cognitive assessment module, ultimately generating a cognitive assessment report containing patient information.

Description

Technical Field

[0001] This invention relates to an intelligent assessment system for wards based on a cognitive assessment robot, and also to a corresponding intelligent assessment method for wards, belonging to the field of healthcare informatics technology. Background Technology

[0002] Currently, the aging population is becoming increasingly serious. Among people aged 65 and above, the prevalence of Alzheimer's disease (AD) is as high as 10% to 30%; the proportion is even higher in this population during the prodromal stage of AD—mild cognitive impairment (MCI). Early screening and timely intervention are the most effective means to delay the progression of MCI to AD.

[0003] However, current MCI screening in China mainly relies on paper-and-pencil tests using a set of neuropsychological scales. This method is time-consuming, requires standardized scoring by professionals, and its results are easily influenced by the participants' educational level. Furthermore, the management of assessment materials is inconvenient, making it difficult to promote as a large-scale, rapid detection tool. In recent years, although molecular imaging and electrophysiological techniques have been introduced into MCI screening, the equipment is expensive, the entry standards are strict, the personnel and technical requirements are high, and the economic costs are significant. Therefore, these methods are also unsuitable for large-scale early risk screening in communities and outpatient clinics.

[0004] Therefore, there is an urgent need to develop a fully automated screening system that can move autonomously, is based on objective physiological indicators, and can quickly complete cognitive function assessments. This would reduce time and labor costs, enable patients to conduct self-screening, significantly improve clinical diagnostic efficiency, and meet the needs of widespread adoption. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide an intelligent assessment system for wards based on a cognitive assessment robot.

[0006] Another technical problem to be solved by the present invention is to provide a ward intelligent assessment method based on a cognitive assessment robot.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, a ward intelligent assessment system based on a cognitive assessment robot is provided, comprising:

[0009] A doctor controller is used to provide a multimodal command input channel; wherein the multimodal command input channel includes at least one of a computer terminal, a mobile terminal device, and a direct voice interaction interface;

[0010] The robot body is communicatively connected to the doctor controller and integrates a mobile chassis module, a vision module, a speech recognition and synthesis module, a cognitive assessment module, and a large-scale model task parsing intelligent agent;

[0011] The large-scale task parsing agent is configured to: receive and parse multimodal instructions from the doctor controller and the speech recognition and synthesis module to generate multi-task control signals; and, based on the multi-task control signals, drive the mobile chassis module to perform patient positioning and navigation, control the vision module and the speech recognition and synthesis module to collaboratively complete patient identity information confirmation, and perform cognitive function assessment operations through the cognitive assessment module, and finally generate a cognitive assessment report containing patient information locally on the robot body.

[0012] Preferably, the large-model task parsing agent receives and parses multimodal instructions from the doctor controller and the speech recognition and synthesis module, and generates multi-task control signals including:

[0013] Receive multimodal instructions from the doctor controller and the speech recognition and synthesis module, and convert them into a structured task description;

[0014] Task planning is performed based on the structured task description to output a task planning result in the form of a four-tuple, which includes the subtask sequence, the modules to be called by each subtask, the target information of each subtask, and the execution status of each subtask.

[0015] Based on the task planning results, subtask control signals are generated sequentially according to the order of the subtask sequence to control the modules corresponding to each subtask to perform closed-loop operations.

[0016] The subtask control signals corresponding to all subtasks together constitute the multitask control signal.

[0017] Preferably, the task planning result in the quadruplet format is generated through the following steps:

[0018] Based on the structured task description, the extracted operation steps are decomposed into multiple sub-tasks in logical order through a temporal reasoning mechanism, and each sub-task is assigned a unique sequence number to form an ordered task sequence T, T = {T1, T2, ..., T...}. n};

[0019] Construct a mapping function f to map each subtask T i Mapped to the corresponding robot execution module M j f:T→M,f(T) i ) = M j Where M = {M1, M2, ..., M} k} represents the set of available robot functional modules, i∈(1,n), j∈(1,k);

[0020] For each subtask Ti Set task status S i The task state S i The status values ​​are divided into pending execution, executing, successful, failed, and skipped, with the initial value set to pending execution;

[0021] For each subtask T i Configuration target information G i The target information includes at least: patient bed availability and patient identification information;

[0022] Based on the task sequence T, the mapping function f, and the task state S set for each subtask. i And the configuration target information G for each subtask i The combined four-tuple task planning result is as follows:

[0023] T′=[(T1,M1,G1,S1),(T2,M2,G2,S2),…,(T n M n G n ,S n )).

[0024] Preferably, the multimodal instruction corresponds to multiple evaluation schemes, and each evaluation scheme is decomposed into multiple sub-tasks;

[0025] Semantic modeling of the decomposed subtasks is performed based on graph neural networks, so as to transform each subtask T i Consider it as a task node v i Edge e is constructed using semantic similarity, resource dependencies, or temporal logic. ij This forms a task association graph F = (V, E); where v i ∈V, e ij ∈E, v i Let i represent the i-th task node, e ij This represents the edge formed between the i-th task node and the j-th task node;

[0026] The task paths in the task association graph are optimized and sorted using dynamic programming, and a state transition function DP is defined. C(v j ,v i ) indicates from task node v j To task node v i The execution cost;

[0027] Based on the state transition function DP, the optimal multi-scheme execution order and path are planned for the robot body, and the execution status of each task state is fed back during execution.

[0028] Preferably, the current subtask sequence, subtask state, subtask target information, and historical execution data are defined as the current state S. t ;

[0029] The adjustment of the subtask sequence and resource allocation strategy is defined as action a. t ;

[0030] The reward function rt is defined based on the task success rate, task execution time, and task failure rate, where rt = α·task success rate + β·task execution time + γ·task failure rate; and α, β, and γ are weight coefficients, and α + β + γ = 1.

[0031] Reinforcement learning is performed using a deep deterministic policy gradient algorithm, and the policy network and Q-network are initialized.

[0032] At each time step t, according to the current state S t Different actions a are selected through the policy network. t The action is then executed, and the corresponding reward function rt and the next state S are output through the Q network. t+1 ;

[0033] Continue reinforcement learning until the preset learning rate is reached, and then select the action 'a' that maximizes the reward function rt. t As the next execution sequence.

[0034] Preferably, the vision module includes a first depth camera and a second depth camera;

[0035] The first depth camera is located in the head region of the robot body and is used to locate the patient by combining the head movement of the robot body, so that the robot body faces the patient and identifies the patient.

[0036] The second depth camera is located in the waist region of the robot body to identify pedestrian gait, distance, and scene-based loopback localization or obstacle avoidance.

[0037] Preferably, during the cognitive assessment module's execution of the cognitive function assessment operation, it further includes:

[0038] Based on the target information parsed by the task parsing agent, the ROI of the target face information is extracted using a face recognition algorithm;

[0039] Based on the target face information ROI, extract the face point cloud set from the corresponding point cloud data;

[0040] Based on the face point cloud set, obtain the face orientation and robot orientation, as well as the coordinates of the face center point and the robot center point;

[0041] Based on the face orientation and robot orientation, calculate the angle between the face and the robot; and based on the center point coordinates of the face and the center point coordinates of the robot, calculate the target distance between the face and the robot body.

[0042] Based on the angle, the eye tracker of the robot body is adjusted to face the pupil of the human face, and the actual distance between the human face and the robot body in the current state is obtained.

[0043] The actual distance is compared with the target distance, and the actual distance is adjusted according to a preset distance tolerance threshold so that the patient's pupil is always within the optimal measurement range of the eye tracker.

[0044] Preferably, during the cognitive assessment module's execution of the cognitive function assessment operation, it further includes:

[0045] By tracking skeletal key points, information on skeletal key points of the patient in sitting or lying position can be obtained in real time.

[0046] Based on the skeletal key point information, the spatial coordinates of the patient's pupil are calculated;

[0047] The robot's pose is dynamically adjusted based on the spatial coordinates of the patient's pupil so that the patient's pupil is always within the optimal measurement range of the eye tracker.

[0048] Preferably, the robot body further includes:

[0049] The facial expression interaction module is used to display emotional expressions and, in conjunction with voice, to conduct human-computer interaction with the patient;

[0050] A display is located in the head area of ​​the robot body and connected to the facial expression interaction module for displaying the human-computer interaction interface;

[0051] An emergency stop module, connected to the mobile chassis module, is used to control the power supply to the mobile chassis module, thereby forcibly stopping the movement of the robot body.

[0052] According to a second aspect of the present invention, a ward intelligent assessment method based on a cognitive assessment robot is provided, comprising the following steps:

[0053] Acquire multimodal instructions input by the doctor through the doctor controller and convert them into task text;

[0054] The multimodal instructions are parsed by a large model task parsing agent to decompose the task text into navigation, verification and evaluation sub-tasks;

[0055] For the navigation, verification, and evaluation subtasks, respectively, navigation control signals, verification control signals, and evaluation control signals are generated;

[0056] The mobile chassis module is driven to perform patient positioning and navigation based on the navigation control signal;

[0057] Based on the verification control signal, the vision module and the speech recognition and synthesis module are controlled to work together to confirm the patient's identity information.

[0058] Based on the assessment control signal, the cognitive assessment module is controlled to perform cognitive function assessment operations, and a cognitive assessment report containing patient information is generated locally on the robot body.

[0059] Compared with the prior art, the present invention has the following technical effects:

[0060] (1) This intelligent ward assessment system uses speech recognition and synthesis modules to parse medical staff's instructions into text, or receives text instructions directly input by the doctor's controller. It then uses a large-scale task parsing agent to complete task parsing, breaking it down into sub-tasks such as navigation, identity verification, and cognitive assessment. Each module is then controlled to execute its corresponding closed-loop operation sequentially, thereby achieving automated cognitive assessment of patients. Thus, this intelligent ward assessment system partially replaces the work of medical staff, significantly saving medical human resources.

[0061] (2) By using the large model task parsing agent, not only can the automatic parsing of tasks, sub-task decomposition and sequential execution of tasks in specific scenarios be automated, but also the natural language interaction channel of speech recognition and synthesis modules can be combined to simplify the operation process, reduce the professional requirements of the test taker, solve the problems of high professional requirements of test takers and difficulty in ensuring the consistency of screening quality in traditional testing methods, and improve screening efficiency.

[0062] (3) By integrating a high-precision eye tracker into the robot body and designing eye tracking tasks, a rapid MCI screening based on eye tracking was realized. With the help of objective eye tracking data that is not under conscious control, the subjective method of traditional scale screening was replaced. Combined with subsequent cognitive assessment norms, an efficient and automatic MCI screening scheme was realized, which is suitable for the needs of large-scale general screening. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a ward intelligent assessment system based on a cognitive assessment robot, provided in the first embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the doctor's control interface in the first embodiment of the present invention;

[0065] Figure 3This is a schematic diagram of the front view structure of the robot body in the first embodiment of the present invention;

[0066] Figure 4 This is a rear view structural diagram of the robot body in the first embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of pre-experimental data for the cognitive assessment module in the first embodiment of the present invention;

[0068] Figure 6 The flowchart illustrates a ward intelligent assessment method based on a cognitive assessment robot, as provided in the second embodiment of the present invention. Detailed Implementation

[0069] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0070] First Embodiment

[0071] The first embodiment of the present invention provides a ward intelligent assessment system based on a cognitive assessment robot, comprising: a doctor controller 1 and a robot body 2. The doctor controller 1 provides a multimodal command input channel, supporting multimodal command input. The robot body 2 is communicatively connected to the doctor controller 1 and integrates multiple functional modules, enabling the robot body 2 to automatically perform cognitive assessments on elderly patients through the multimodal channel.

[0072] like Figure 1 As shown, the doctor controller 1 includes a computer controller 11 and a mobile control terminal 12. Both the computer controller 11 and the mobile control terminal 12 have built-in computer control software, allowing the doctor to directly input text commands to issue task instructions. These task instructions are then transmitted to the robot body 2 in text form for task parsing. In this embodiment, the computer controller 11 and the mobile control terminal 12 issue tasks using a combination of task and location. Alternatively, the doctor can also issue task instructions directly via voice, which are then transmitted to the robot body 2, where the voice information is converted into text instructions for task parsing.

[0073] Figure 2 The image shows the doctor control interface in the first embodiment of the present invention. This interface mainly implements three functions: task selection, bed confirmation, and dialogue monitoring. Doctors can assign tasks by selecting beds and tasks, and by arranging task sequences. Simultaneously, they can monitor the dialogue between the robot and the patient on the screen as a diagnostic reference.

[0074] like Figure 3 and Figure 4As shown, the robot body 2 integrates a mobile chassis module 21, a vision module 22, a speech recognition and synthesis module 23, a cognitive assessment module 24, a large-scale task parsing agent 25, an facial expression interaction module 26, an emergency stop module 27, and power and network support for each module, as well as a shell structure. The mobile chassis module 21 drives the robot body 2 to move. This module includes a wireless network module 211, a lidar module 212, an ultrasound module 213, a map navigation algorithm module, a wheeled structure supporting movement, and a power supply module. The vision module 22 is used for patient localization and works with the speech recognition and synthesis module 23 to confirm patient information. The cognitive assessment module 24 performs cognitive function assessments on the patient to generate a cognitive assessment report containing patient information on the robot body. The large-scale task parsing agent 25 is used for task parsing to generate multi-task control signals, thereby controlling each module to perform corresponding operations. The facial expression interaction module 26 is used to display emotional expressions and interact with patients via the voice unit 261. Furthermore, the human-computer interaction is displayed on the screen 262 located on the head of the robot body. The emergency stop module 27 is used to control the power supply to the chassis wheel hubs to ensure that the movement of the robot body is forcibly stopped in case of an accident.

[0075] In one embodiment of the present invention, the large-scale task parsing agent 25 is configured to: receive and parse multimodal instructions from the doctor controller 1 or the speech recognition and synthesis module 23 to generate multi-task control signals. Furthermore, based on the multi-task control signals, the mobile chassis module 21 is driven to perform patient positioning and navigation, the vision module 22 and the speech recognition and synthesis module 23 are controlled to collaboratively complete patient identity verification, and the cognitive function assessment operation is performed through the cognitive assessment module 24. Finally, a cognitive assessment report containing patient information is generated locally on the robot body.

[0076] In one embodiment of the present invention, the large model task parsing agent 25 specifically performs multimodal instruction parsing through the following steps:

[0077] S1: Receives multimodal instructions from the doctor controller and the speech recognition and synthesis module, and converts them into a structured task description.

[0078] Specifically, in the first embodiment of the present invention, the large-model task parsing agent 25 receives natural language task instructions input by medical staff based on a general large language model, such as: "Please have the robot assist in completing the admission presentation, cognitive assessment, and examination reminders." The large-model task parsing agent 25 first parses the input text through a semantic understanding module, identifying key verbs (such as "presentation," "cognitive assessment," and "examination reminders") and operation objects (such as "face" and "eyes"), and extracting target information (such as "patient bed and identity information"). This process uses a combination of a task verb classifier based on the Transformer model and syntactic dependency analysis to ensure accurate extraction of key actions.

[0079] S2: Obtain the task planning results.

[0080] Based on the structured task description in S1 above, task planning is performed to output a task planning result in the form of a four-tuple, which includes the subtask sequence, the modules that each subtask needs to call, the target information of each subtask, and the execution status of each subtask.

[0081] Specifically, it includes the following steps:

[0082] S21: Subtask decomposition to form a task sequence.

[0083] Based on structured task description, the extracted operation steps are decomposed into multiple sub-tasks in logical order through a temporal reasoning mechanism, and each sub-task is assigned a unique sequence number to form an ordered task sequence T, T = {T1, T2, ..., T...}. n In one embodiment of the present invention, the task text is specifically decomposed into three sub-tasks: navigation, verification, and evaluation.

[0084] S22: Construct the mapping function f.

[0085] Specifically, each subtask T i Mapped to the corresponding robot execution module M j f:T→M,f(T) i ) = M j Where M = {M1, M2, ..., M} k} represents the set of available robot functional modules, i∈(1,n), j∈(1,k).

[0086] It is understandable that this mapping function f can match the module that needs to be executed to each subtask.

[0087] S23: For each subtask T i Set task status S i .

[0088] Specifically, in one embodiment of the present invention, the task state S i The status values ​​are divided into pending execution, executing, successful, failed, and skipped, with the initial value set to pending execution.

[0089] S24: For each subtask T i Configuration target information G i .

[0090] In one embodiment of the present invention, the target information includes at least: patient bed and patient identity information.

[0091] S25: Based on the task sequence T, mapping function f, and task state S set for each subtask. i And the configuration target information G for each subtask i The combined four-tuple task planning result is as follows:

[0092] T′=[(T1,M1,G1,S1),(T2,M2,G2,S2),…,(T n M n G n ,S n )).

[0093] S3: Based on the task planning results, generate subtask control signals in the order of the subtask sequence to control the modules corresponding to each subtask to perform closed-loop operations.

[0094] In one embodiment of the present invention, navigation control signals, verification control signals, and assessment control signals need to be generated for the navigation, verification, and assessment sub-tasks, respectively, so that each module can perform closed-loop operations of navigation, patient information verification, and cognitive assessment. Specifically, the navigation control signal parsed by the large model task parsing agent 25 controls the mobile chassis module 21 to perform patient positioning navigation, thereby driving the robot body 2 to automatically find the patient who needs to undergo cognitive assessment. After finding the patient, the vision module 22 and the speech recognition and synthesis module 23 are controlled based on the verification control signal to collaboratively complete the patient's identity information confirmation. Finally, the cognitive assessment module 24 is controlled based on the assessment control signal to perform a cognitive assessment on the patient and generate a cognitive assessment report.

[0095] In the above embodiments, preferably, the multimodal instructions correspond to multiple evaluation schemes, and each evaluation scheme is decomposed into multiple sub-tasks. Based on this, firstly, semantic modeling is performed on the decomposed sub-tasks using a graph neural network, so as to express each sub-task T... i Consider it as a task node v i Edge e is constructed using semantic similarity, resource dependencies, or temporal logic. ij This forms a task association graph F = (V, E). Where vi ∈V, e ij ∈E, v i Let i represent the i-th task node, e ij This represents the edge formed between the i-th task node and the j-th task node.

[0096] Then, dynamic programming is used to optimize and sort the task paths in the task association graph, and a state transition function DP is defined. Wherein, C(v) j ,v i ) indicates from task node v j To task node v i The execution cost includes module switching costs and constraints such as navigation paths and task priorities. Finally, based on this state transition function DP, the robot can plan the optimal execution order and path for multiple solutions, and provide feedback on the execution status of each task state during execution.

[0097] It should be noted that the state transition function in this embodiment can be considered a multi-objective function, using a task cost function to determine which of several consecutive tasks should be executed first. For example, if task i is being executed and the next task is to conduct educational activities at bed x, but a new task, cognitive assessment at bed y, is suddenly inserted, then diagnostic tasks have higher priority than educational tasks. Therefore, the cognitive assessment task can be prioritized. Specifically, the execution cost C of tasks with higher priority is assigned a smaller value, and the state transition function can be used to achieve this ranking.

[0098] Specifically, in one embodiment of the present invention, the current subtask sequence, subtask state, subtask target information, and historical execution data are defined as the current state S. t The adjustment of the subtask sequence and resource allocation strategy is defined as action a. t The reward function r is defined based on the task success rate, task execution time, and task failure rate. t r t = α·task success rate + β·task execution time + γ·task failure rate; where α, β, and γ are weighting coefficients, and α + β + γ = 1.

[0099] Furthermore, a deep deterministic policy gradient algorithm is used for reinforcement learning, and the policy network μ(s|θ) is initialized. μ ) and Q-network Q(s,a|θ Q Thus, at each time step t, according to the current state S t Different actions a are selected through the policy network. t The action is then executed, and the corresponding reward function r is output through the Q-network. t The value and the next state S t+1Therefore, through continuous reinforcement learning until a preset learning rate is reached, and a reward function r is selected. t The action with the largest value, a t As the next execution sequence.

[0100] In this context, the policy network μ(s|θμ) is used to select action a. t Refer to the formula:

[0101] a t =μ(st|θμ)+N t , where Nt is the exploration noise.

[0102] The network is optimized by updating the parameters of the Q network and the policy network. The calculation formula is as follows:

[0103]

[0104] Among them, y t Let η represent the target Q value and η represent the learning rate. It is the gradient operator, used to calculate the derivative of a function with respect to a variable. Ultimately, by softly updating the target network parameters, the stability of the network is ensured, thereby achieving dynamic optimization of the task planning mechanism.

[0105] In the above embodiments, preferably, the vision module 22 includes a first depth camera 221 and a second depth camera 222. The first depth camera 221 is located in the head region of the robot body and is used to locate the patient by combining the robot body's head movements, so that the robot body faces the patient and identifies the patient. The second depth camera 222 is located in the waist region of the robot body and is used to identify pedestrian gait, distance, and for scene-based loopback localization or obstacle avoidance.

[0106] In the above embodiments, preferably, the cognitive assessment module 24 includes an eye-tracking assessment system built on infrared camera hardware and a touchscreen display system. The touchscreen display system primarily selects specific task combinations based on pre-selected cognitive assessment tasks and presents the corresponding cognitive assessment tasks to the patient. The eye-tracking assessment system extracts images from facial images captured by the infrared camera, uses eye features as input to a neural network to predict eye fixation positions, monitors the patient's eye fixation timing and position during task execution using a 200Hz recognition frequency, and compares the duration of eye lingering at specific positions under a specific cognitive task with norms, thus achieving rapid and accurate cognitive screening in a non-contact manner.

[0107] Furthermore, to better ensure the accuracy of the cognitive assessment of the robot body 2, the cognitive assessment module 24 also includes a face distance estimation system, and combines it with the 3D identity recognition algorithm of the head camera to realize a closed-loop control strategy of identity recognition-large-range positioning-local adjustment, ensuring that the human eye appears within the optimal measurement range of the eye tracker.

[0108] Specifically, the face distance estimation process is as follows:

[0109] First, the robot body 2 uses the task parsing agent 25 to parse the navigation target information (including the target person's bed, name, and face ID). Then, using face recognition algorithms such as YOLO-Face, it achieves identity recognition, confirms the task target information in the scene, and extracts the target face information ROI.

[0110] Secondly, based on the ROI coordinates in the face image (i.e., the RGB image), in the corresponding point cloud data... Extract the point cloud data corresponding to the face region:

[0111] P face ={p i ∈P|project(p i ∈ROI face )}

[0112] Among them, project(p i ) represents the point cloud coordinates (x i ,y i ,z i Projected onto the image plane (based on the camera intrinsic parameter matrix); P face This represents a set of face point clouds.

[0113] Then, based on this face point cloud set, the face orientation is obtained. With the current direction of the robot Furthermore, obtain the coordinates of the center point of the face (x... face ,y face ,z face )) and robot center point coordinates (x robot ,y robot ,z robot ).

[0114] Then, based on the face orientation and the robot orientation, the angle between the face and the robot is calculated. The target distance D between the face and the robot is calculated based on the coordinates of the face center point and the robot center point.

[0115]

[0116] Finally, when the robot faces the human face directly and is near the target distance, start the infrared camera of the eye tracker, collect the face image, identify the pupil position or facial feature points, and further calculate the actual distance D between the current face and the lens camera . D camera > D + ε, prompt the patient "Please get closer to the screen"; D camera < D - ε, prompt the patient "Please move away from the screen", where ε is the distance tolerance threshold

[0117] In the above embodiments, preferably, during the cognitive assessment process, add the function of tracking skeletal key points. Through the skeletal key point tracking technology, the skeletal key point information of the patient's sitting or lying posture is obtained in real time. Then, based on the obtained skeletal key point information, the spatial coordinates of the patient's pupils are calculated in real time. Finally, according to the calculated pupil spatial coordinates, the pose of the robot is dynamically adjusted (for example: the robot's head rotates or the robot's head extends and retracts, etc.) to ensure that the patient's pupils are always within the best measurement range of the eye tracker. It can be understood that through this dynamic adjustment mechanism, the measurement accuracy and stability of the eye tracker can be further improved, ensuring accurate measurement results in different poses

[0118] Specifically, in an embodiment of the present invention, let the set of skeletal key points of the patient be K = {k1, k2,..., k n}, where k i represents the three-dimensional coordinates (x i , y i , z i ) of the i-th skeletal key point

[0119] Through the skeletal key point tracking technology, obtain the head key point k head and the eye key point k eye . Assume k eye = {k eye-left , k eye-right}, then the spatial coordinates P pupil of the pupil can be calculated from the relative positions of the head and eye key points: P pupil = k eye + ΔP. Here, ΔP is the position offset of the pupil relative to the center of the eye, usually a constant vector. Thus, after calculation, the spatial coordinates P pupil of the pupil are expressed as (x p , y p , z p )

[0120] Then, let the current angular pose of the robot be R robot = (x r , y r , z r , θr ), where (x r ,y r ,z r ) represents the robot's position coordinates, θ r Let x represent the robot's orientation angle. Furthermore, it's understandable that once the robot's pose is determined, since the center of the robot's eye tracker is a fixed point on the robot, the center of the robot's eye tracker (x...) e ,y e ,z e The result will also be determined accordingly.

[0121] Finally, the target was adjusted to the center of the robot eye tracker (x) e ,y e ,z e The distance between the pupil center and the center of the pupil is within the optimal distance: d=√{(x p -x e ) 2 +(y p -y e ) 2 +(z p -z e ) 2}

[0122] Figure 5 The image shows preliminary experimental data for the cognitive assessment module 24 in the first embodiment of this invention. First, 100 participants aged 60-70 years were recruited from the community. Through two tasks—smooth tracking and visuospatial memory—the elderly were able to complete a non-invasive cognitive assessment within 5 minutes without the participation of an examiner. Simultaneously, all participants were assessed using the MoCA scale, the gold standard for MCI (Mean Clinical Infection). Experimental results show that, in screening for MCI-positive patients, the ward-based intelligent assessment system provided in this embodiment achieves 88.2% consistency with the MoCA gold standard. The ward-based intelligent assessment system provided in this embodiment achieves 88.2% consistency with the clinical gold standard without direct examiner participation, and its efficiency is 87.5% higher than the MoCA (40 minutes). These results indicate that the ward-based intelligent assessment system provided in this embodiment has the potential to become a large-scale MCI screening tool for the elderly.

[0123] Second Embodiment

[0124] like Figure 6 As shown, based on the first embodiment described above, the second embodiment of the present invention provides a ward intelligent assessment method based on a cognitive assessment robot, specifically including the following steps:

[0125] S10: Obtain multimodal instructions.

[0126] Specifically, doctors can choose to issue text commands directly via computer or mobile terminal, or they can issue voice commands and use the speech recognition and synthesis module 23 to convert them into text commands.

[0127] S20: Multimodal instruction parsing, generating multi-task control signals.

[0128] Specifically, the large model task parsing agent 25 parses multimodal instructions to decompose the task text into navigation, verification, and evaluation sub-tasks. Then, navigation control signals, verification control signals, and evaluation control signals are generated for the navigation, verification, and evaluation sub-tasks respectively, to form a multi-task control signal.

[0129] The specific parsing process of the large model task parsing agent 25 can be referred to the description in the first embodiment above, and will not be repeated here.

[0130] S30: Controls each module to perform corresponding closed-loop operations.

[0131] Specifically, the mobile chassis module 21 is driven by navigation control signals to perform patient positioning and navigation. The vision module 22 and the speech recognition and synthesis module 23 are controlled by verification control signals to collaboratively verify the patient's identity information. The cognitive assessment module 24 is controlled by assessment control signals to perform cognitive function assessment operations and generate a cognitive assessment report containing patient information locally on the robot.

[0132] It should be noted that the above embodiments are merely illustrative examples, and the technical solutions of each embodiment can be combined, all of which are within the protection scope of this invention.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0134] The above provides a detailed description of the intelligent ward assessment system and method based on a cognitive assessment robot provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essential content of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A ward intelligent assessment system based on a cognitive assessment robot, characterized in that... include: A doctor controller is used to provide a multimodal command input channel; wherein the multimodal command input channel includes at least one of a computer terminal, a mobile terminal device, and a direct voice interaction interface; The robot body is communicatively connected to the doctor controller and integrates a mobile chassis module, a vision module, a speech recognition and synthesis module, a cognitive assessment module, and a large-scale model task parsing intelligent agent; The large-scale task parsing agent is configured to: receive and parse multimodal instructions from the doctor controller and the speech recognition and synthesis module to generate multi-task control signals; and, based on the multi-task control signals, drive the mobile chassis module to perform patient positioning and navigation, control the vision module and the speech recognition and synthesis module to collaboratively complete patient identity information confirmation, and perform cognitive function assessment operations through the cognitive assessment module, and finally generate a cognitive assessment report containing patient information locally on the robot body.

2. The intelligent ward assessment system as described in claim 1, characterized in that: The large-scale model task parsing agent receives and parses multimodal instructions from the doctor controller and the speech recognition and synthesis module, and generates multi-task control signals, specifically including: Receive multimodal instructions from the doctor controller and the speech recognition and synthesis module, and convert them into a structured task description; Task planning is performed based on the structured task description to output a task planning result in the form of a four-tuple, which includes the subtask sequence, the modules to be called by each subtask, the target information of each subtask, and the execution status of each subtask. Based on the task planning results, subtask control signals are generated sequentially according to the order of the subtask sequence to control the modules corresponding to each subtask to perform closed-loop operations. The subtask control signals corresponding to all subtasks together constitute the multitask control signal.

3. The intelligent ward assessment system as described in claim 2, characterized in that... The task planning results in the quadruplet format are generated through the following steps: Based on the structured task description, the extracted operation steps are decomposed into multiple sub-tasks in logical order through a temporal reasoning mechanism, and each sub-task is assigned a unique sequence number to form an ordered task sequence T, T = {T1, T2, ..., T...}. n }; Construct a mapping function f to map each subtask T i Mapped to the corresponding robot execution module M j f:T→M,f(T) i ) = M j ; Where M = {M1, M2, ..., M} k } represents the set of available robot functional modules, i∈(1,n), j∈(1,k); For each subtask T i Set task status S i The task state S i The status values ​​are divided into pending execution, executing, successful, failed, and skipped, with the initial value set to pending execution; For each subtask T i Configuration target information G i The target information includes at least: patient bed availability and patient identification information; Based on the task sequence T, the mapping function f, and the task state S set for each subtask. i And the configuration target information G for each subtask i The combined four-tuple task planning result is as follows: T′=[(T1,M1,G1,S1),(T2,M2,G2,S2),…,(T n ,M n ,G n ,S n )]。 4. The intelligent ward assessment system as described in claim 2, characterized in that: The multimodal instructions correspond to multiple evaluation schemes, and each evaluation scheme is decomposed into multiple sub-tasks; Semantic modeling of the decomposed subtasks is performed based on graph neural networks, so as to transform each subtask T i Consider it as a task node v i Edge e is constructed using semantic similarity, resource dependencies, or temporal logic. ij This forms a task association graph F = (V, E); Among them, v i ∈V, e ij ∈E, v i Let i represent the i-th task node, e ij This represents the edge formed between the i-th task node and the j-th task node; The task paths in the task association graph are optimized and sorted using dynamic programming, and a state transition function DP is defined. C(v j ,v i ) indicates from task node v j To task node v i The execution cost; Based on the state transition function DP, the optimal multi-scheme execution order and path are planned for the robot body, and the execution status of each task state is fed back during execution.

5. The intelligent ward assessment system as described in claim 4, characterized in that: Define the current subtask sequence, subtask status, subtask target information, and historical execution data as the current state S. t ; The adjustment of the subtask sequence and resource allocation strategy is defined as action a. t ; A reward function r is defined based on task success rate, task execution time, and task failure rate. t r t = α·task success rate + β·task execution time + γ·task failure rate; where α, β, and γ are weighting coefficients, and α + β + γ = 1; Reinforcement learning is performed using a deep deterministic policy gradient algorithm, and the policy network and Q-network are initialized. At each time step t, according to the current state S t Different actions a are selected through the policy network. t The action is then executed, and the corresponding reward function r is output through the Q-network. t The value and the next state S t+1 ; Continue reinforcement learning until the preset learning rate is reached, and then select the reward function r. t The action with the largest value, a t As the next execution sequence.

6. The intelligent ward assessment system as described in claim 4, characterized in that: The vision module includes a first depth camera and a second depth camera; The first depth camera is located in the head region of the robot body and is used to locate the patient by combining the head movement of the robot body, so that the robot body faces the patient and identifies the patient. The second depth camera is located in the waist region of the robot body to identify pedestrian gait, distance, and scene-based loopback localization or obstacle avoidance.

7. The intelligent ward assessment system as described in claim 1, characterized in that... The cognitive assessment module, during the cognitive function assessment process, also includes: Based on the target information parsed by the task parsing agent, the ROI of the target face information is extracted using a face recognition algorithm; Based on the target face information ROI, extract the face point cloud set from the corresponding point cloud data; Based on the face point cloud set, obtain the face orientation and robot orientation, as well as the coordinates of the face center point and the robot center point; Based on the face orientation and robot orientation, calculate the angle between the face and the robot; and based on the center point coordinates of the face and the center point coordinates of the robot, calculate the target distance between the face and the robot body. Based on the angle, the eye tracker of the robot body is adjusted to face the pupil of the human face, and the actual distance between the human face and the robot body in the current state is obtained. The actual distance is compared with the target distance, and the actual distance is adjusted according to a preset distance tolerance threshold so that the patient's pupil is always within the optimal measurement range of the eye tracker.

8. The intelligent ward assessment system as described in claim 7, characterized in that... The cognitive assessment module, during the cognitive function assessment process, also includes: By tracking skeletal key points, information on skeletal key points of the patient in sitting or lying position can be obtained in real time. Based on the skeletal key point information, the spatial coordinates of the patient's pupil are calculated; The robot's pose is dynamically adjusted based on the spatial coordinates of the patient's pupil so that the patient's pupil is always within the optimal measurement range of the eye tracker.

9. The intelligent assessment system for wards as described in claim 1, characterized in that... The robot body also includes: The facial expression interaction module is used to display emotional expressions and, in conjunction with voice, to conduct human-computer interaction with the patient; A display is located in the head area of ​​the robot body and connected to the facial expression interaction module for displaying the human-computer interaction interface; An emergency stop module, connected to the mobile chassis module, is used to control the power supply to the mobile chassis module, thereby forcibly stopping the movement of the robot body.

10. A ward intelligent assessment method based on a cognitive assessment robot, implemented based on the ward intelligent assessment system described in any one of claims 1 to 9, characterized in that... include: Acquire multimodal instructions input by the doctor through the doctor controller and convert them into task text; The multimodal instructions are parsed by a large model task parsing agent to decompose the task text into navigation, verification and evaluation sub-tasks; For the navigation, verification, and evaluation subtasks, respectively, navigation control signals, verification control signals, and evaluation control signals are generated; The mobile chassis module is driven to perform patient positioning and navigation based on the navigation control signal; Based on the verification control signal, the vision module and the speech recognition and synthesis module are controlled to work together to confirm the patient's identity information. Based on the assessment control signal, the cognitive assessment module is controlled to perform cognitive function assessment operations, and a cognitive assessment report containing patient information is generated locally on the robot body.

Images