Nursing data processing method and device

Abstract

The invention provides a nursing data processing method and equipment, relates to a data processing technology, and aims at effectively making up cognitive and memory defects of a patient through visual guidance of VR equipment, helping the patient clearly understand action steps and smoothly complete daily activities, such as guiding the patient to go to a toilet according to a correct route. In the guiding process, different guiding modes can be adopted according to the illness conditions of the patients, the autonomous guiding mode can guide the patients to complete corresponding activities by themselves, and the cooperative guiding mode can guide the patients to complete corresponding activities in a mode that nursing personnel cooperate with the patients. Therefore, the nursing accuracy and individuation can be improved, and the requirements of different patients are met.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a nursing data processing method and device. Background Art

[0002] With the aging population and the rising incidence of various neurological diseases, the number of patients suffering from cognitive and behavioral impairments is gradually increasing, with Alzheimer's disease being a particularly prominent example. Alzheimer's patients often experience memory loss, spatial disorientation, and decreased autonomy, resulting in the inability to independently complete basic activities of life, such as toileting and taking medication. In this context, effectively helping these patients complete daily activities has become a pressing social and medical issue.

[0003] Currently, nursing assistance options for patients with cognitive and motor impairments are relatively limited. Traditional care methods rely primarily on real-time companionship and verbal guidance from caregivers, but this approach is both labor-intensive and inefficient. While some existing electronic assistance devices offer guidance, their guidance methods are often overly simplistic and lack personalization. For example, some devices can only provide simple voice prompts and cannot be flexibly adjusted based on the patient's specific condition and actual operating conditions, making it difficult to meet the diverse needs of different patients.

[0004] Therefore, how to provide patients with customized guidance and assist them in completing daily activities has become an urgent problem that needs to be solved. Summary of the Invention

[0005] The present invention provides a nursing data processing method and device, which can provide customized guidance for users and assist users in completing daily activities.

[0006] A first aspect of the present invention provides a nursing data processing method, comprising:

[0007] Recording a user's customized demonstration video, and selecting an autonomous guidance mode or a collaborative guidance mode based on the user's nursing assessment results, wherein the customized demonstration video corresponds to multiple positioning nodes;

[0008] When the autonomous guidance mode is executed, the interactive device is triggered to display the customized demonstration video, and the playback timing of the customized demonstration video is adjusted in real time according to the item status of each positioning node;

[0009] When the collaborative guidance mode is executed, the interactive devices of the user and the nursing end are synchronized, and the playback timing of the customized demonstration video is controlled according to the adjustment instructions of the nursing end.

[0010] Optionally, in a possible implementation of the first aspect, a user's customized demonstration video is recorded, and the autonomous guidance mode or the collaborative guidance mode is selected based on the user's nursing assessment result, wherein the customized demonstration video corresponds to multiple positioning nodes, including:

[0011] Generating multiple positioning nodes according to the multiple location markers added to the customized demonstration video;

[0012] Obtaining the average duration and completion of the user's historical activities based on the nursing assessment results, and selecting the autonomous guidance mode when the average duration is less than a benchmark duration and the completion is greater than a benchmark completion;

[0013] When the average duration is greater than or equal to the benchmark duration and / or the completion degree is less than or equal to the benchmark completion degree, the collaborative guidance mode is selected.

[0014] Optionally, in a possible implementation of the first aspect, when the autonomous guidance mode is executed, triggering the interactive device to display the customized demonstration video and adjusting the playback timing of the customized demonstration video in real time according to the item status of each positioning node includes:

[0015] triggering a startup instruction of the interactive device to retrieve a customized demonstration video to generate a virtual guidance interface including a video playback area and a guidance mark, wherein the customized demonstration video includes video sub-segments of each positioning node and is configured with a playback order;

[0016] Comparing the similarity between the video frame set of each positioning node and the virtual guidance interface, and selecting a positioning node with a similarity greater than a threshold;

[0017] The standard item status of the positioning node is compared with the current item status of the virtual guide interface. If the statuses are the same, the next video sub-segment is played; otherwise, the current video sub-segment is played in a loop.

[0018] Optionally, in a possible implementation of the first aspect, calling a customized demonstration video to generate a virtual guidance interface including a video playback area and a guidance mark includes:

[0019] Displaying corresponding video sub-segments based on the video playback area;

[0020] Identifying a first target object corresponding to the positioning node in the virtual guidance interface, and determining guidance attributes of the first target object, wherein the guidance attributes include movement attributes and action attributes;

[0021] determining, according to the movement attribute, a direction from a preset point to a reference point of the first target object as an indication direction, and generating a guidance mark based on the indication direction;

[0022] A preset gesture is retrieved based on the action attribute, and the preset gesture is positioned at a reference position of the first target object for display.

[0023] Optionally, in a possible implementation of the first aspect, the method further includes:

[0024] If the first target object corresponding to the positioning node is not identified in the virtual guidance interface, retrieving a spatial map including a plurality of preset feature points;

[0025] Matching the feature points in the virtual guidance interface with a plurality of preset feature points to determine the current position and current orientation of the user;

[0026] The orientation from the current position to the position of the first target object is determined as a target orientation, a rotation direction from the current orientation to the target orientation is obtained, and reminder information corresponding to the rotation direction is generated.

[0027] Optionally, in a possible implementation of the first aspect, comparing the standard item status of the positioning node with the current item status of the virtual guidance interface, and playing the next video sub-segment if the statuses are the same, and looping the current video sub-segment if they are different, includes:

[0028] Obtaining a standard video frame selected from the video frame set corresponding to the positioning node, and comparing the similarity between the first target object corresponding to the positioning node in the standard video frame and the corresponding object in the virtual guidance interface;

[0029] When the similarity is greater than or equal to the similarity threshold, it is determined that the item status of the positioning node and the virtual guidance interface is the same, and the next video sub-segment is played;

[0030] When the similarity is less than the similarity threshold, it is determined that the item states of the positioning node and the virtual guidance interface are different, and the current video sub-segment is played in a loop.

[0031] Optionally, in a possible implementation of the first aspect, when the autonomous boot mode is executed, the following steps are further included:

[0032] When the first target object in the virtual guidance interface corresponds to a moving attribute, the playback speed of the customized demonstration video is updated according to an adjustment speed corresponding to an area change rate of the first target object, and the area change difference is positively correlated with the adjustment speed;

[0033] When the first target item in the virtual guidance interface corresponds to an action attribute, the number of cycles of the customized demonstration video is obtained, and the playback speed of the customized demonstration video is updated according to the adjustment speed corresponding to the number of cycles.

[0034] Optionally, in a possible implementation of the first aspect, when the collaborative guidance mode is executed, synchronizing the interactive devices of the user and the nursing end, and controlling the playback timing of the customized demonstration video according to the adjustment instructions of the nursing end, includes:

[0035] After synchronizing the interactive device of the user and the nursing end, the multiple video sub-segments contained in the customized demonstration video are played in sequence, and the voice data of the nursing end is obtained in real time;

[0036] Identifying keywords of the voice data, wherein the keywords include instruction keywords and operation keywords;

[0037] determining an adjustment instruction corresponding to the instruction keyword, and playing the video sub-segments sequentially or in a loop according to the adjustment instruction;

[0038] A second target object corresponding to the current video sub-segment in the virtual guidance interface is identified, a dynamic display mark corresponding to the operation keyword is generated, and the dynamic display mark is displayed at a reference position of the second target object.

[0039] Optionally, in a possible implementation of the first aspect, generating a dynamic display mark corresponding to the operation keyword and displaying the dynamic display mark at a reference position of the second target item includes:

[0040] Determining the operation attributes of the operation keyword, wherein the operation attributes include movement attributes and action attributes;

[0041] determining an operation direction corresponding to the operation keyword according to the movement attribute, displaying a reference mark corresponding to the movement attribute at a reference position of the second target object, and adjusting it based on the operation direction;

[0042] An operation action corresponding to the operation keyword is determined based on an action attribute, a reference mark corresponding to the action attribute is displayed at a reference position of the second target item, and the second target item is adjusted based on the operation action.

[0043] Optionally, in a possible implementation manner of the first aspect, when adjusting the reference marker based on the operation direction, the method further includes:

[0044] Receive feedback instructions from the nursing end, identify direction information and angle information in the voice data, and adjust the reference mark according to the direction information and the angle information.

[0045] According to a second aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor runs the computer program to execute the first aspect of the present invention and various methods that may be involved in the first aspect.

[0046] The beneficial effects of the present invention are as follows:

[0047] 1. The present invention significantly improves the accuracy and personalization of nursing care by recording the user's customized demonstration video and selecting the autonomous guidance mode or the collaborative guidance mode based on the nursing assessment results. For users who have a good acceptance of guidance and strong autonomy in historical activities, the autonomous guidance mode can give full play to their autonomy. The system can automatically adjust the playback timing of the customized demonstration video according to the user's completion of the item status of each positioning node, so as to achieve precise synchronization between the video and the user's actions. For example, when guiding the user to go to the toilet, when the user walks to the toilet door, the VR device identifies the state of the door. If the state of the door matches the state when the node is completed, it will automatically play the next video clip to guide the user to the next activity smoothly; otherwise, the current clip will be repeated to ensure that the user can keep up with the guidance rhythm. For users with poor autonomous action ability and high dependence on guidance, in the collaborative guidance mode, the caregiver and the user will complete the activity collaboratively through synchronized interactive devices. Nursing staff can obtain the user's operation status in real time. After the system recognizes the keywords of their voice commands, they can accurately control the playback timing of customized demonstration videos. For example, video sub-segments can be played sequentially or in a loop according to command keywords such as "continue" and "repeat". At the same time, dynamic display logos are generated according to the operation keywords to guide users to perform correct operations on the second target item, greatly meeting the needs of different users.

[0048] 2. The present invention utilizes technical means such as similarity comparison between positioning nodes, video frame sets and virtual guidance interfaces, as well as identification of the attributes of the first target item and adjustment of the reference mark, to effectively improve the success rate of user operations. In autonomous guidance mode, the system displays the corresponding video sub-segment based on the video playback area, identifies the first target item corresponding to the positioning node in the virtual guidance interface, and determines its movement attributes and action attributes. The guidance mark is generated based on the movement attribute to indicate the direction, allowing the user to clearly know the starting direction of the operation. For example, at the door of a toilet, the user can quickly find the location of the door handle; based on the action attribute, the preset gesture display is called to help the user accurately master the key points of the operation, such as the correct hand posture when opening the toilet door. If the first target item is not identified, the system determines the user's position and orientation by calling the spatial map and matching the feature points, generates a rotation direction reminder message, and guides the user to find the target item. At the same time, by comparing the standard item status of the positioning node with the current item status of the virtual guidance interface, it decides to play the next video sub-segment or loop the current video sub-segment to ensure that the user's operation is correct. In the collaborative guidance mode, the operation direction and action are determined based on the operation attributes of the operation keywords, and the benchmark signs are displayed and adjusted to provide users with clear and accurate operation guidance, greatly improving the success rate of users in completing daily activities.

[0049] 3. In the collaborative guidance mode of the present invention, the nursing staff can send feedback instructions through the nursing terminal according to the user's performance. After receiving the feedback instructions, the system recognizes the direction information and angle information in the voice data and adjusts the reference mark to make the guidance more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present invention;

[0051] Figure 2 This is a flow chart of a nursing data processing method provided by an embodiment of the present invention;

[0052] Figure 3 The figure is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0054] See also Figure 1 , is a schematic diagram of an application scenario provided by an embodiment of the present invention. It is understandable that patients may have cognitive and behavioral disorders. For example, as the disease progresses, Alzheimer's patients' brain functions gradually decline, their cognitive abilities continue to decline, and it becomes difficult for them to understand and execute daily activities. They often forget the process of daily affairs, such as forgetting when to go to the toilet. In response to this situation, the present invention will effectively make up for the patient's cognitive and memory defects through the visual guidance of VR equipment, help patients clearly understand the action steps, and smoothly complete daily activities, such as guiding patients to go to the toilet in the correct route. During the guidance process, different guidance modes can be adopted according to the patient's condition. The autonomous guidance mode can guide the patient to complete the corresponding activities on their own, and the collaborative guidance mode can guide the patient to complete the corresponding activities through the collaboration of the caregiver and the patient. Thereby, the accuracy and personalization of nursing care can be improved to meet the needs of different patients. Among them, the interactive device is a device that interacts with the patient, such as a VR device.

[0055] Schematic diagram, Figure 2The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, the user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDAs) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S101 to S103, as follows:

[0056] S101, recording a user's customized demonstration video, and selecting an autonomous guidance mode or a collaborative guidance mode based on the user's nursing assessment result, wherein the customized demonstration video corresponds to multiple positioning nodes.

[0057] Customized demonstration videos are recorded for patients to guide them through activities. For example, a video could show them walking to the toilet, including getting up from a seat, finding their way to the toilet, opening the door, and using the toilet. These videos can be accompanied by concise voice narration and large font prompts. The nursing assessment results are a comprehensive assessment of factors such as the patient's past nursing guidance experience, including their completion status and duration of each activity. The autonomous guidance mode is designed for patients who have historically responded well to guidance and can relatively independently complete nursing activities based on their own understanding and system prompts. The collaborative guidance mode is designed for patients who have historically relied heavily on guidance, have poor autonomy, and struggle to independently complete nursing activities. Location nodes are nodes corresponding to key locations marked in the customized demonstration video, corresponding to operational steps in the patient's nursing process. For example, when guiding a patient to the toilet, location nodes can be set for different locations, such as the living room, the entrance connecting the living room and toilet, the toilet door, and the interior of the toilet.

[0058] Understandably, pre-recording video data is intended to facilitate patient understanding and compliance. By analyzing the effectiveness of guidance based on the patient's historical activities and generating a nursing assessment, the guidance model is selected to best align with the patient's actual abilities, improving the relevance and effectiveness of care. Positioning nodes help the system accurately determine the patient's position and progress during the activity, providing a basis for timely adjustments to the guidance process.

[0059] When recording a customized demonstration video, the caregiver can simulate the patient's perspective, starting with the starting action and demonstrating the movements slowly and clearly, following the patient's daily routine. Appropriate pauses are provided between each action to give the patient time to understand and react. For example, when recording a trip to the bathroom, the caregiver can slowly rise from their seat, turn toward the bathroom, and then walk slowly, pausing after every few steps. While demonstrating the movements, the caregiver can provide detailed verbal instructions. For example, when opening the bathroom door, the caregiver can say, "Walk to the door, reach out and grasp the door handle, turn it to the left, and push the door open."

[0060] Based on the above embodiment, the specific implementation of step S101 may be:

[0061] Based on the multiple location markers added to the customized demonstration video, multiple positioning nodes are generated; based on the nursing assessment results, the average duration and completion of the user's historical activities are obtained, and when the average duration is less than the benchmark duration and the completion is greater than the benchmark completion, the autonomous guidance mode is selected; when the average duration is greater than or equal to the benchmark duration and / or the completion is less than or equal to the benchmark completion, the collaborative guidance mode is selected.

[0062] It's understandable that the addition of location markers is intended to closely align with the patient's actual activity scenarios. Positioning nodes act like "navigation points" along the patient's path, enabling the system to accurately identify the patient's specific location and operational phase during their activity. This facilitates seamless switching of guidance content based on the patient's location during the guidance process. It also allows patients to enhance their understanding and identification of the guidance information by seeing familiar perspectives and corresponding location prompts while watching the video guidance, thereby improving the effectiveness and practicality of the guidance.

[0063] Using the average duration and completion of patients' historical activities as the basis for selecting a guidance mode can achieve personalized care. Different patients have different cognitive abilities, physical functions, and proficiency in daily activities. By comparing patients' average duration and completion with baseline values, it is possible to judge the patient's ability level in various activities. For patients with a shorter average duration and a higher degree of completion, it indicates that they have strong autonomy and are suitable for the autonomous guidance mode to give full play to their autonomy; for patients with a longer average duration or a lower degree of completion, it indicates that they need more external support. The collaborative guidance mode allows caregivers to intervene in real time and provide necessary guidance to ensure that patients can complete activities safely and smoothly.

[0064] Location markers can be recorded during the recording process when a caregiver reaches a specific location, such as the bathroom entrance, that is closely related to the activity. This location information is clearly marked in the video by operating the VR device. The average duration of a historical activity can be calculated by statistically analyzing the time spent by the patient performing the same historical activity multiple times. The completion of a historical activity is a quantitative indicator that measures the accuracy and completeness of a patient's historical activity. The patient can be evaluated and scored based on whether all required actions were completed correctly, the degree of standardization of the actions, and whether the entire procedure was completed as required. The final score or percentage represents the completion of the patient's historical activity. The benchmark duration is a pre-set standard time value used as a reference for measuring the average duration of a patient's historical activity. The benchmark duration can be determined by statistical analysis of activity time data from a large number of similar patients, or by referring to the reasonable duration of the activity under normal circumstances. For example, if the average bathroom visit time for 100 patients with similar conditions is 6 minutes, the benchmark duration can be set to 6 minutes. The benchmark completion is a pre-set standard completion value used for comparison with the patient's completion. It represents the level of accuracy and completeness with which patients should ideally complete the activity. The benchmark completion level can be set based on factors such as the standard of care and the general requirements for similar patients. For example, the benchmark completion level could be set at 85% based on the patient's specific needs.

[0065] S102, when the autonomous guidance mode is executed, triggering the interactive device to display the customized demonstration video, and adjusting the playback timing of the customized demonstration video in real time according to the item status of each positioning node.

[0066] When the self-guided mode is activated, after the patient puts on the VR device, the VR device can display the patient's guidance video of each operation step. The reason for adjusting the playback timing of the customized demonstration video in combination with the object status is to synchronize the patient's current operation steps with the operation steps displayed in the video, so that the patient can perform corresponding operations based on the data displayed in the video.

[0067] Item status refers to the actual state of the item at the location node, including its position, posture, and whether it's being operated. For example, in the toilet door opening process, the door's open or closed state. Playback sequence refers to the order and timing of each segment of the customized demonstration video. Based on the patient's completion of the item status at the location node, the decision can be automatically made whether to play the next segment or repeat the current segment.

[0068] For example, when a patient reaches the restroom door, the VR device can use the camera to identify the door's status and determine whether it is closed or open. When the door's status matches the state at the completion of the node, it can be determined that the patient has completed the preparatory actions for the current node. The device then switches to and plays the next video clip, guiding the patient to smoothly enter the next activity, such as entering the restroom or walking towards the toilet. Otherwise, it means that the patient may not have completed the current operation step, and the current video clip can be played repeatedly.

[0069] Based on the above embodiment, the specific implementation of step S102 may be:

[0070] The interactive device startup instruction is triggered, and a customized demonstration video is retrieved to generate a virtual guidance interface including a video playback area and a guidance mark. The customized demonstration video includes a video sub-segment of each positioning node and is configured with a playback order.

[0071] It is understandable that the virtual guidance interface can present the guidance information in a way that is easy for patients to understand and operate. The video playback area is used to display specific guidance video content and can be located in a corner of the interface, such as the upper left corner. The guidance logo can help patients quickly understand the current guidance stage and the key information that needs attention. The customized demonstration video is divided into video sub-segments containing each positioning node and configured with a playback order. It can provide patients with guidance on each key node in an orderly manner according to the patient's activity flow, ensuring the continuity and logic of the guidance process.

[0072] Among them, the virtual guidance interface is an interface presented to the patient through a VR device, which includes a video playback area for playing customized demonstration videos, and guidance signs to assist patients in understanding guidance information and mastering the progress of activities. The video sub-segments are multiple independent segments that the customized demonstration video is divided into according to the positioning nodes. Each sub-segment corresponds to a positioning node, which shows in detail the operation process that the patient needs to complete at the node. The guidance signs are signs for guiding patients in operations, such as arrows, hand models for demonstrating actions, etc., which can be located in the designated positions of the corresponding items. During the display, AR perspective technology can be used, and the patient can see the real environment and virtual guidance at the same time when wearing the device.

[0073] In some embodiments, a virtual guide interface may be generated by the following steps:

[0074] Based on the video playback area, a corresponding video sub-segment is displayed; a first target item corresponding to the positioning node in the virtual guidance interface is identified, and guidance attributes of the first target item are determined, wherein the guidance attributes include movement attributes and action attributes; according to the movement attributes, a direction from a preset point to a reference point of the first target item is determined as an indication direction, and a guidance mark is generated based on the indication direction; based on the action attributes, a preset gesture is retrieved, and the preset gesture is positioned at the reference position of the first target item for display.

[0075] After determining the patient's location, the corresponding video sub-segment is displayed in the video playback area, providing visual operational guidance closely related to the patient's current location. For example, when the patient reaches the location node corresponding to the restroom door, the video sub-segment played can demonstrate how to correctly open the door, the movements for entering the restroom, and related precautions. This allows the patient to intuitively see the next steps, enhancing the intuitiveness and operability of the guidance and helping patients better understand and execute the task.

[0076] The first target item refers to the key item that the patient needs to operate or interact with at a specific positioning node. For example, the first target item is different at different positioning nodes when going to the toilet. At the positioning node corresponding to the toilet door, the toilet door is the first target item; at the positioning node corresponding to the toilet, the toilet is the first target item. Movement attributes refer to the characteristics of position movement, and action attributes refer to the specific action characteristics required to operate the first target item, such as the hand grip method, force direction, rotation angle, etc.

[0077] For items at different positioning nodes, corresponding attributes can be configured in advance, so that identification can be generated in combination with the identified attributes. Determining the indication direction and generating guidance identification based on the movement attributes can allow patients to clearly know the starting direction of the target object. For example, in the toilet scene, when the patient faces the toilet door, clearly indicating the direction from their current position to the key positions of the door, such as the door handle, can help patients quickly find the starting point of the operation and improve the accuracy and efficiency of the operation. Recalling and displaying preset gestures based on action attributes is to provide patients with more specific and intuitive action demonstrations, helping patients to accurately master the action essentials of operating target objects. In the toilet scene, for operations such as opening the toilet door, showing the correct hand posture and action position can allow patients to learn and imitate more intuitively, reduce the difficulty of operation, and increase the success rate of operation.

[0078] The preset point is a starting reference point pre-set in the virtual guidance interface and related to the patient's current position. For example, it can be located at the center of the lower boundary of the virtual guidance interface. The reference point is a representative key position point on the first target object, which is closely related to the operation. For example, for a toilet door, the center point of the door handle can be the reference point. The guidance mark generated according to the indicated direction can be an arrow. The preset gesture is a standard hand posture and action sequence related to the operation of the first target object pre-stored in the system. The reference position is the specific position on the first target object where the operation needs to be performed, which is closely related to the action attributes. For example, for a toilet door, the reference position can be the position of the door handle.

[0079] In addition, if the first target object is not recognized in the virtual guidance interface, the user may be reminded through the following embodiments to improve the flexibility and accuracy of guidance:

[0080] If the first target object corresponding to the positioning node is not identified in the virtual guidance interface, a spatial map containing multiple preset feature points is retrieved; the feature points in the virtual guidance interface are matched with multiple preset feature points to determine the user's current position and current orientation; the orientation from the current position to the location of the first target object is determined as the target orientation, the rotation direction from the current orientation to the target orientation is obtained, and a reminder message corresponding to the rotation direction is generated.

[0081] It is understandable that in the process of guiding the patient to perform activities, the first target object may not be identified due to various reasons such as changes in light, objects being blocked, etc. The spatial map is retrieved at this time to determine the relative position relationship between the patient and the target object. A spatial map refers to a digitally constructed map of the patient's environment. The map accurately marks the three-dimensional coordinate information of multiple preset feature points and their respective unique feature descriptors. These feature points are selected from locations with obvious and stable features in the environment, such as corner vertices, corners of large furniture, and centers of unique decorative patterns on walls. The preset feature points are key location points in the spatial map with specific coordinates and feature descriptions. The spatial map can be constructed by taking photos of the environment with a panoramic camera, which contains preset feature point information for areas such as the living room, corridor, and toilet door.

[0082] By matching real-time environmental feature points in the virtual guidance interface with pre-set feature points in the spatial map, the similarity between the two can be exploited to determine the patient's actual position and orientation in space. The VR device's camera continuously captures the patient's surroundings. Using the ORB algorithm, feature points in the virtual guidance interface, such as the edges of wall sockets and the corners of floor tiles, are extracted. These feature points are then matched against pre-set feature points in the spatial map. Assuming a Euclidean distance matching strategy, the Euclidean distance between each extracted feature point and the pre-set feature point descriptor is calculated, selecting pairs with the smallest distance (i.e., the highest matching score). Based on these matching point pairs, the PnP algorithm, combined with the VR device camera's internal and external parameters, is used to calculate the patient's three-dimensional coordinates in the spatial map, confirming the patient's specific location near the restroom door. Furthermore, by analyzing the relative position of the matching feature points in the image and map, trigonometric functions are used to calculate the patient's current orientation, such as whether the patient is facing a certain direction in the hallway.

[0083] Determining the target orientation and generating a rotation direction reminder can provide patients with clear directional guidance, allowing them to adjust their orientation and move toward the location of the target object. When determining the target orientation, the coordinate information of the target object's location is known in the spatial map. Based on the determined coordinates of the patient's current position, a vector pointing from the current position to the target object's location is calculated through vector operations. The projection direction of this vector on the horizontal plane is defined as the target orientation. When comparing the patient's current orientation and the target orientation and calculating the rotation direction required to rotate from the current orientation to the target orientation, the rotation direction can be determined by calculating the angle between the two direction vectors and the cross product. If the cross product result is positive, it indicates that clockwise rotation is required; if the cross product result is negative, counterclockwise rotation is required. The reminder information can be played to the patient through the voice broadcast module of the VR device, and can also be displayed in eye-catching text on the virtual guidance interface to help patients understand and adjust their orientation.

[0084] The similarity between the video frame set of each positioning node and the virtual guidance interface is compared, and a positioning node with a similarity greater than a threshold is selected.

[0085] During a patient's movements, their current location and stage need to be determined in real time to provide accurate guidance. By comparing the similarity between a set of video frames of a positioning node and the virtual guidance interface, image recognition technology can be used to quickly and accurately determine whether the patient has reached a specific positioning node. Selecting positioning nodes with a similarity greater than a threshold ensures accurate recognition, avoiding misjudgments that could lead to incorrect guidance, and providing a reliable basis for subsequent guidance decisions.

[0086] Each video subsegment of a location node consists of a series of continuous video frames, each containing various image information about that location node. For example, when a patient reaches the restroom door location node, the video frame collection may display the restroom door's appearance and closed state. If the calculated similarity between the restroom door image in the virtual guidance interface and the location node's video frame collection exceeds a pre-set threshold, such as 85%, the patient is determined to have reached the restroom door location node and that node is selected.

[0087] The standard item status of the positioning node is compared with the current item status of the virtual guide interface. If the statuses are the same, the next video sub-segment is played; otherwise, the current video sub-segment is played in a loop.

[0088] After confirming the patient's arrival at the corresponding positioning node, it is necessary to determine whether the patient has correctly completed the node's task. Comparing the status of the standard items at the positioning node with the current item status on the virtual guidance interface verifies compliance with the patient's operation. If the status is the same, the patient has completed the current node's task and can proceed to the next stage of guidance. If the status is different, the patient may have made an error or incomplete the task. Looping the current video subsegment and providing prompts helps the patient correct errors, ensuring they complete the restroom activity according to the correct procedure, and improving the effectiveness and accuracy of guidance.

[0089] For example, suppose the standard object state for the location node corresponding to the restroom door is open. Once the patient is confirmed to have reached the restroom door, the actual state of the restroom door in the virtual guidance interface is identified and analyzed. If the door is open, which aligns with the standard object state, the video is controlled to play the next video subsegment. If the door is closed, the current video subsegment is played in a loop.

[0090] In some embodiments, the step of "comparing the standard item status of the positioning node with the current item status of the virtual guidance interface, and if the statuses are the same, playing the next video sub-segment; otherwise, looping and playing the current video sub-segment" can be implemented by the following steps:

[0091] A standard video frame selected from a set of video frames corresponding to the positioning node is obtained, and a similarity is compared between a first target item corresponding to the positioning node in the standard video frame and a corresponding item in the virtual guidance interface; when the similarity is greater than or equal to a similarity threshold, it is determined that the item states of the positioning node and the virtual guidance interface are the same, and the next video sub-segment is played; when the similarity is less than the similarity threshold, it is determined that the item states of the positioning node and the virtual guidance interface are different, and the current video sub-segment is played in a loop.

[0092] The standard video frame can be a pre-selected image frame that reflects the standard operation and object status of the corresponding positioning node. After the standard video frame is determined, the first target object corresponding to the positioning node is compared with the corresponding object in the patient's current virtual guidance interface for similarity. This can determine the difference between the patient's operation at the current positioning node and the standard status. When comparing similarities, various features of the object in the standard video frame, including color, shape, texture, etc., can be accurately matched with the corresponding features of the object in the virtual guidance interface to obtain a numerical result.

[0093] The similarity threshold can be a pre-set numerical standard for judging whether the similarity between the object in the standard video frame and the object in the virtual guidance interface reaches an acceptable level. When the similarity is greater than or equal to this threshold, it indicates that the patient's operation at the current positioning node is very similar to the standard operation, and the status of the object is basically the same. At this time, the guidance of the next positioning node can be advanced and the next video sub-segment can be played. This can maintain the continuity and efficiency of the guidance process and allow the patient to smoothly enter the next operation link. On the contrary, when the similarity is less than the threshold, it means that there is a certain deviation in the patient's operation and the status of the object is inconsistent with the standard status. At this time, by looping the current video sub-segment, the patient has the opportunity to repeatedly watch the correct operation demonstration, helping the patient to continuously learn and correct his own operation until the standard state is reached, thereby ensuring that the entire guidance process can be completed smoothly and accurately, and improving the patient's operation success rate.

[0094] In addition, when executing in autonomous boot mode, the following embodiments are also included:

[0095] When the first target item in the virtual guidance interface corresponds to a movement attribute, the playback speed of the customized demonstration video is updated according to the adjustment speed corresponding to the area change rate of the first target item, and the area change difference is positively correlated with the adjustment speed; when the first target item in the virtual guidance interface corresponds to an action attribute, the number of cycles of the customized demonstration video is obtained, and the playback speed of the customized demonstration video is updated according to the adjustment speed corresponding to the number of cycles.

[0096] It can be understood that when the first target object has a mobile property, by monitoring its area change rate to adjust the playback speed of the customized demonstration video, the video guidance can be closely synchronized with the patient's actual movements. The speed of the patient's movement will result in different rates of change in the area of the first target object in the virtual guidance interface. A larger area change difference indicates faster patient movement. By making the adjustment speed positively correlated with the area change difference, the video playback speed can be adjusted in real time according to the patient's movements. This allows the video to provide timely and appropriate guidance regardless of the patient's speed during movement, avoiding situations where the video plays too fast or too slow, causing the patient to lose track of the pace or wait too long. This improves the effectiveness and accuracy of guidance, allowing patients to receive guidance information tailored to their specific circumstances. The area change rate refers to the change in the area occupied by the first target object in the virtual guidance interface per unit time. For example, if the area of a toilet door in the virtual guidance interface changes from 10% to 15% of the screen within 1 second, the area change rate is 5% per second. Different area change rates are preset with corresponding adjustment speeds.

[0097] When the first target item corresponds to an action attribute, the playback speed is adjusted by obtaining the number of loops of the customized demonstration video to adapt to the patient's learning and mastery of the specific action. If the video loops many times, it means that the patient may not be very proficient in the action and needs more time to understand and imitate. At this time, reducing the video playback speed can allow the patient to observe the details of the action more clearly, which helps the patient to better learn and correct the action. Conversely, if the number of loops is small, it means that the patient has a good grasp of the action. The playback speed can be appropriately increased to improve the guidance efficiency, so that the video guidance is more in line with the patient's individual action learning process and achieve personalized guidance.

[0098] S103, when the collaborative guidance mode is executed, the interactive device of the user and the nursing end is synchronized, and the playback timing of the customized demonstration video is controlled according to the adjustment instruction of the nursing end.

[0099] When collaborative guidance mode is activated, the caregiver can wear another VR device that is synchronized with the patient's VR device and has voice transmission capabilities. By observing the patient's real-time image, the caregiver can voice-command the patient to complete various actions. The adjustment commands are commands that control video playback based on the caregiver's voice data. In this mode, the caregiver can customize the playback order and timing of each demonstration video segment based on the patient's current condition. The caregiver operates the device, such as a mobile phone or computer.

[0100] Based on the above embodiment, the specific implementation of step S103 may be:

[0101] After synchronizing the interactive devices of the user and the nursing end, the multiple video sub-segments contained in the customized demonstration video are played in sequence, and the voice data of the nursing end is obtained in real time; the keywords of the voice data are identified, and the keywords include instruction keywords and operation keywords; the adjustment instructions corresponding to the instruction keywords are determined, and the video sub-segments are played sequentially or in a loop according to the adjustment instructions; the second target object corresponding to the current video sub-segment in the virtual guidance interface is identified, and a dynamic display logo corresponding to the operation keyword is generated, and it is displayed at the base position of the second target object.

[0102] After the patient and caregiver synchronize their devices, the caregiver can use the voice command information sent by the nursing-side interactive device to collect and transmit it to the patient's interactive device to guide patient activities. Keyword recognition of the voice data on the nursing side is to extract key content from a large amount of voice information so that the caregiver's intentions can be understood and responded accordingly. Command keywords are used to control the playback method of video sub-segments, such as deciding whether to play sequentially or in a loop, to help patients better master the operation process. Operation keywords are related to the items that the patient needs to operate, and are used to generate targeted dynamic display logos to guide patients to perform correct operations on specific items, improve the accuracy and effectiveness of guidance, and enable patients to quickly understand the caregiver's guidance and respond correctly.

[0103] Command keywords refer to words in the voice data that control the playback of video sub-segments, such as "continue," "repeat," and "pause." When a caregiver says "continue," the next video sub-segment is played sequentially; when they say "repeat," the current video sub-segment is played in a loop. Adjustment commands are specific operational instructions generated by mapping command keywords. For example, the command keyword "continue" corresponds to the adjustment command for playing the next video sub-segment sequentially; "repeat" corresponds to the adjustment command for playing the current video sub-segment in a loop.

[0104] For example, when the command keyword "repeat" is identified, the adjustment command is determined to loop the current video sub-segment. Assuming that the current video sub-segment shows the patient correctly grasping the door handle to open the toilet door, after receiving the "repeat" command, the system will restart the video sub-segment and allow the patient to watch and learn again. If the command keyword "continue" is identified, the next video sub-segment will be played sequentially, such as switching from the video sub-segment of opening the toilet door to the video sub-segment of walking towards the toilet after entering the toilet.

[0105] Operation keywords refer to the words in the voice data that correspond to the items that the patient needs to operate and the related operation actions, such as "open the door", "sit down", "flush", etc. in the toilet scene. These keywords are associated with the second target item in the virtual guidance interface and are used to generate dynamic display logos to guide the patient's operation. The second target item is the key item that the patient needs to operate or interact with in the current video sub-segment. The dynamic display logo is a visual logo generated based on the operation keyword, which is used to instruct the patient to operate the second target item, such as a flashing arrow, a dynamic operation step prompt animation, etc. The reference position of the second target item refers to the key position point on the second target item that is closely related to the operation. The dynamic display logo will be displayed at this position to guide the patient to perform accurate operation. For example, on a toilet, the position of the flush button is the reference position corresponding to the "flush" operation.

[0106] For example, after a patient enters the restroom, the current video subsegment shows the patient walking toward the toilet and sitting down. The system identifies the toilet as the second target object in the virtual guidance interface. If the caregiver issues the voice command "sit," the system recognizes "sit" as the action keyword and generates a flashing circle pointing to the toilet seat as a dynamic display indicator. This circle is then displayed precisely at the base position of the toilet seat, intuitively guiding the patient to sit down.

[0107] In some embodiments, the step of "generating a dynamic display logo corresponding to the operation keyword and displaying it at the reference position of the second target item" can be implemented by the following steps:

[0108] Determine the operation attributes of the operation keyword, wherein the operation attributes include movement attributes and action attributes; determine the operation direction corresponding to the operation keyword based on the movement attributes, display the reference identifier corresponding to the movement attribute at the reference position of the second target item, and adjust it based on the operation direction; determine the operation action corresponding to the operation keyword based on the action attributes, display the reference identifier corresponding to the action attribute at the reference position of the second target item, and adjust it based on the operation action.

[0109] It is understandable that movement attributes and action attributes can analyze operations from different perspectives. Movement attributes focus on the change of the position of an object in space, such as in which direction the object moves, along what trajectory, and how far it moves. Action attributes focus on the specific movement characteristics of human body parts when performing operations on objects, such as the way of grasping, the direction of force, the angle of rotation, and the order of actions. By accurately distinguishing these two attributes, guidance plans can be tailored for patients based on different types of operations. In this way, patients can understand the intention of the operation more clearly and perform the operation accurately, thereby ensuring the correct operation of related items during the activity and successfully completing various tasks.

[0110] Determining the operation direction based on the movement attributes and displaying the corresponding reference mark is intended to provide patients with intuitive and clear visual guidance, allowing them to know the direction and method of moving the second target object. In the virtual guidance interface, displaying the reference mark at the reference position of the second target object enables patients to quickly lock onto the operation target and perform operations according to the instructions of the mark. Adjusting the reference mark according to the operation direction ensures that the mark always points accurately to the operation direction, enhancing the intuitiveness and accuracy of the guidance, and helping patients successfully complete the movement operation of the second target object.

[0111] Among them, the operation direction is the direction in which the second target object needs to move, which is determined by the movement attribute. In the horizontal direction, there are left and right movement directions, such as moving an object on the table from the left to the right; vertically, there are up and down, like picking up an object on the ground and lifting it up. There are other specific direction angles, such as the rotation angle direction of the knob when turning it. Taking "opening the drawer" as an example, the operation direction is usually horizontally outward, that is, along the outward direction of the drawer track. The reference mark of the movement attribute is a visual mark generated to guide the patient's operation, such as an arrow. The direction of the arrow clearly indicates the operation direction, such as a rightward arrow indicating that the object is to move to the right. The reference position of the second target object is the key position point on the second target object that is closely connected with the operation, and is the position where the reference mark is displayed.

[0112] The purpose of determining the operation action based on the action attributes and displaying the corresponding reference mark is to show the patient in detail the specific action method when operating the second target object. By displaying the reference mark corresponding to the action attribute at the reference position of the second target object and adjusting it according to the operation action, the patient can clearly see the specific steps and action details of the operation, so as to accurately imitate and execute the operation. This helps patients master complex operation actions, improve the standardization and accuracy of operations, avoid damage to items or affecting the quality of activity completion due to improper operation, and improve the accuracy and effectiveness of patient operation guidance.

[0113] The manipulation action is the specific behavior of a human body part manipulating the second target object, as determined by the action attributes. This includes details such as gripping method, force direction, rotation angle, and action sequence. The baseline identifier of the action attribute is a visual element used to display the details of the manipulation action. This can be a dynamic animation that demonstrates how the fingers grasp the object and how the action is performed, such as a continuous animation showing the fingers gripping the key and the wrist rotating when turning the key.

[0114] In some embodiments, when adjusting the reference marker based on the operation direction, the following steps are further included:

[0115] Receive feedback instructions from the nursing end, identify direction information and angle information in the voice data, and adjust the reference mark according to the direction information and the angle information.

[0116] Feedback instructions refer to instructions sent by caregivers via the nursing device. While a user is wearing the VR device and performing an activity, the caregiver can observe the user's movements and performance in real time. If the caregiver notices a deviation in the user's performance, or if the user provides feedback through the device that they don't understand an instruction, the caregiver can send a feedback instruction and then determine the specific feedback information by recognizing the caregiver's voice data.

[0117] For example, suppose a caregiver sends a voice feedback instruction saying, "The user doesn't understand the specific direction of moving to the left front. They should specify moving 30 degrees to the left front." Upon receiving this voice data, speech recognition technology is used to analyze it. The speech is first converted into text, and then a natural language processing algorithm is used to identify the direction information, "left front," and the angle information, "30 degrees." This information is extracted and organized, and then used to adjust the reference markers, thereby improving guidance accuracy. Directional information describes the direction of movement of an object or user, such as left, right, forward, backward, up, down, etc., as well as more specific direction descriptions, such as southeast and northwest. When guiding user operations, this directional information indicates the direction the user should move. Angle information represents the angle data of the object or user's movement or operation, such as rotation angle and tilt angle.

[0118] Through the above methods, the feedback from caregivers can be converted into intuitive visual guidance to help users better understand and perform operations, thereby improving the success rate and efficiency of operations.

[0119] See also Figure 3 , is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention, wherein the electronic device 30 includes: a processor 31, a memory 32 and a computer program;

[0120] The memory 32 is used to store the computer program, which may also be a flash memory. The computer program is, for example, an application program or a functional module for implementing the above method.

[0121] The processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant description in the above method embodiment.

[0122] Optionally, the memory 32 may be independent or integrated with the processor 31 .

[0123] When the memory 32 is a device independent of the processor 31, the device may further include:

[0124] The bus 33 is used to connect the memory 32 and the processor 31 .

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nursing data processing method, characterized in that: include: Recording a user's customized demonstration video, and selecting an autonomous guidance mode or a collaborative guidance mode based on the user's nursing assessment results, wherein the customized demonstration video corresponds to multiple positioning nodes; When the autonomous guidance mode is executed, the interactive device is triggered to display the customized demonstration video, and the playback timing of the customized demonstration video is adjusted in real time according to the item status of each positioning node; When the collaborative guidance mode is executed, the interactive devices of the user and the nursing end are synchronized, and the playback timing of the customized demonstration video is controlled according to the adjustment instructions of the nursing end.

2. The method according to claim 1, characterized in that Record the user's customized demonstration video and select the autonomous guidance mode or collaborative guidance mode based on the user's nursing assessment results. The customized demonstration video corresponds to multiple positioning nodes, including: Generating multiple positioning nodes according to the multiple location markers added to the customized demonstration video; Obtaining the average duration and completion of the user's historical activities based on the nursing assessment results, and selecting the autonomous guidance mode when the average duration is less than a benchmark duration and the completion is greater than a benchmark completion; When the average duration is greater than or equal to the benchmark duration and / or the completion degree is less than or equal to the benchmark completion degree, the collaborative guidance mode is selected.

3. The method according to claim 1, characterized in that When the autonomous guidance mode is executed, the interactive device is triggered to display the customized demonstration video, and the playback timing of the customized demonstration video is adjusted in real time according to the item status of each positioning node, including: triggering a startup instruction of the interactive device to retrieve a customized demonstration video to generate a virtual guidance interface including a video playback area and a guidance mark, wherein the customized demonstration video includes video sub-segments of each positioning node and is configured with a playback order; Comparing the similarity between the video frame set of each positioning node and the virtual guidance interface, and selecting a positioning node with a similarity greater than a threshold; The standard item status of the positioning node is compared with the current item status of the virtual guide interface. If the statuses are the same, the next video sub-segment is played; otherwise, the current video sub-segment is played in a loop.

4. The method according to claim 3, characterized in that Retrieve a customized demonstration video to generate a virtual guide interface with a video playback area and guide logo, including: Displaying corresponding video sub-segments based on the video playback area; Identifying a first target object corresponding to the positioning node in the virtual guidance interface, and determining guidance attributes of the first target object, wherein the guidance attributes include movement attributes and action attributes; determining, according to the movement attribute, a direction from a preset point to a reference point of the first target object as an indication direction, and generating a guidance mark based on the indication direction; A preset gesture is retrieved based on the action attribute, and the preset gesture is positioned at a reference position of the first target object for display.

5. The method according to claim 4, characterized in that Also includes: If the first target object corresponding to the positioning node is not identified in the virtual guidance interface, retrieving a spatial map including a plurality of preset feature points; Matching the feature points in the virtual guidance interface with a plurality of preset feature points to determine the current position and current orientation of the user; The orientation from the current position to the position of the first target object is determined as a target orientation, a rotation direction from the current orientation to the target orientation is obtained, and reminder information corresponding to the rotation direction is generated.

6. The method according to claim 3, characterized in that Comparing the standard item status of the positioning node with the current item status of the virtual guide interface, and playing the next video sub-segment if the status is the same, and looping the current video sub-segment if the status is different, including: Obtaining a standard video frame selected from the video frame set corresponding to the positioning node, and comparing the similarity between the first target object corresponding to the positioning node in the standard video frame and the corresponding object in the virtual guidance interface; When the similarity is greater than or equal to the similarity threshold, it is determined that the item status of the positioning node and the virtual guidance interface is the same, and the next video sub-segment is played; When the similarity is less than the similarity threshold, it is determined that the item states of the positioning node and the virtual guidance interface are different, and the current video sub-segment is played in a loop.

7. The method according to claim 1, characterized in that When the autonomous boot mode is executed, the following steps are also included: When the first target object in the virtual guidance interface corresponds to a moving attribute, the playback speed of the customized demonstration video is updated according to an adjustment speed corresponding to an area change rate of the first target object, and the area change difference is positively correlated with the adjustment speed; When the first target item in the virtual guidance interface corresponds to an action attribute, the number of cycles of the customized demonstration video is obtained, and the playback speed of the customized demonstration video is updated according to the adjustment speed corresponding to the number of cycles.

8. The method according to claim 1, characterized in that When the collaborative guidance mode is executed, the interactive device of the user and the nursing end is synchronized, and the playback timing of the customized demonstration video is controlled according to the adjustment instruction of the nursing end, including: After synchronizing the interactive device of the user and the nursing end, the multiple video sub-segments contained in the customized demonstration video are played in sequence, and the voice data of the nursing end is obtained in real time; Identifying keywords of the voice data, wherein the keywords include instruction keywords and operation keywords; determining an adjustment instruction corresponding to the instruction keyword, and playing the video sub-segments sequentially or in a loop according to the adjustment instruction; A second target object corresponding to the current video sub-segment in the virtual guidance interface is identified, a dynamic display mark corresponding to the operation keyword is generated, and the dynamic display mark is displayed at a reference position of the second target object.

9. The method according to claim 8, characterized in that Generating a dynamic display mark corresponding to the operation keyword and displaying it at a reference position of the second target item includes: Determining the operation attributes of the operation keyword, wherein the operation attributes include movement attributes and action attributes; determining an operation direction corresponding to the operation keyword according to the movement attribute, displaying a reference mark corresponding to the movement attribute at a reference position of the second target object, and adjusting it based on the operation direction; An operation action corresponding to the operation keyword is determined based on an action attribute, a reference mark corresponding to the action attribute is displayed at a reference position of the second target item, and the second target item is adjusted based on the operation action.

10. The method according to claim 9, characterized in that When adjusting the reference mark based on the operation direction, the method further includes: Receive feedback instructions from the nursing end, identify direction information and angle information in the voice data, and adjust the reference mark according to the direction information and the angle information.

Images