Virtual reality display method and system for rehabilitation robot based on image segmentation

Abstract

The application provides a virtual reality display method and system of a rehabilitation robot based on image segmentation, and relates to the technical field of virtual display. The method can determine an intermediate 3D human body model according to an external body feature vector corresponding to a target user and an external body feature vector corresponding to a preset 3D human body model, determine a target 3D human body model according to a first bone feature vector corresponding to the target user and a second bone feature vector corresponding to the intermediate 3D model, drive the target 3D human body model to perform synchronous movement according to decomposed actions completed by the target user when performing a training task, and mark each decomposed action using a difficulty level label. The method effectively reduces the influence of external factors such as clothing and accessories of the target user on data collection and processing, can more accurately restore the real action details of the target user, is beneficial to improving the user experience, avoids a complex reconstruction process of three-dimensional point cloud data, and reduces the consumption of computing resources.

Description

Technical Field

[0001] This invention relates to the field of virtual display technology, and in particular to a virtual reality display method and system for a rehabilitation robot based on image segmentation. Background Technology

[0002] A rehabilitation robot is an intelligent device that combines robotics, sensor technology, control technology, and artificial intelligence. It is primarily used to assist users in rehabilitation training, helping them regain motor function and improve their physical condition. In recent years, with the development of virtual reality (VR) technology, combining rehabilitation robots with VR technology allows the target user and other users managing the target user to more intuitively observe the process and effects of the target user's rehabilitation training.

[0003] In existing technologies, methods for combining rehabilitation robots with VR technology include: extracting several images from several images collected by the rehabilitation robot that only present a portion of the target user's region; obtaining several three-dimensional point cloud data corresponding to the target user based on the several partial region images; generating VR data of the target user based on the several three-dimensional point cloud data; and displaying the VR data.

[0004] However, the above method also has the following technical problems:

[0005] 3D point cloud data obtained from partial area images can be affected by external factors such as the target user's clothing and accessories, making it impossible to accurately capture the user's subtle movements, such as fine finger movements or small changes in joints. As a result, the reliability of VR data generated based on 3D point cloud data is low. Therefore, VR data generated by the above method has low accuracy and is difficult to reproduce the real movement details of the target user, which may reduce the user experience of the target user and other users managing the target user. Summary of the Invention

[0006] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0007] According to a first aspect of the present invention, a virtual reality display method for a rehabilitation robot based on image segmentation is provided, the method comprising the following steps:

[0008] S1. Based on the target image segmentation model and the first target image group, obtain the third target image group corresponding to the target user. The first target image is an image of the target user collected by the rehabilitation robot during the process of the target user performing training tasks using the rehabilitation robot. The third target image is a part of the first target image that only shows the target user.

[0009] S2. Obtain and display the VR data corresponding to the target user based on the third target image group corresponding to the target user. Step S2 includes the following steps:

[0010] S21. Input the third target image group corresponding to the target user into the preset external body shape feature vector extraction model to obtain the external body shape feature vector corresponding to the target user.

[0011] S22. Determine several intermediate 3D human models based on several preset 3D human models, wherein the intermediate 3D human models are the preset 3D human models whose vector similarity between the corresponding external body shape feature vector and the target user's corresponding external body shape feature vector is not less than a preset vector similarity threshold.

[0012] S23. Determine the target 3D human model based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vectors corresponding to several intermediate 3D human models.

[0013] S24. Drive the target 3D human body model to move synchronously according to the decomposed actions completed by the target user when performing the training task, and use difficulty level labels to mark each decomposed action.

[0014] According to a second aspect of the present invention, a virtual reality display system for a rehabilitation robot based on image segmentation is provided, the system comprising:

[0015] The image segmentation module is used to obtain a third target image group corresponding to the target user based on the target image segmentation model and the first target image group. The first target image is an image of the target user collected by the rehabilitation robot during the process of the target user performing training tasks using the rehabilitation robot. The third target image is a portion of the first target image that only shows the target user.

[0016] The VR data processing module is used to acquire and display VR data corresponding to the target user based on the third target image group corresponding to the target user; the VR data processing module includes:

[0017] The external body shape feature vector acquisition unit is used to input the third target image group corresponding to the target user into the preset external body shape feature vector extraction model to obtain the external body shape feature vector corresponding to the target user.

[0018] The intermediate 3D human body model determination unit is used to determine a number of intermediate 3D human body models based on a number of preset 3D human body models. The intermediate 3D human body model is the preset 3D human body model whose vector similarity between the corresponding external body shape feature vector and the target user's corresponding external body shape feature vector is not less than a preset vector similarity threshold.

[0019] The target 3D human body model determination unit is used to determine the target 3D human body model based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vectors corresponding to several intermediate 3D human body models.

[0020] The target 3D human model driving unit is used to drive the target 3D human model to move synchronously according to the decomposed actions completed by the target user when performing training tasks, and to label each decomposed action with difficulty level labels.

[0021] The present invention has at least the following beneficial effects:

[0022] This invention provides a virtual reality display method and system for a rehabilitation robot based on image segmentation. The method, based on a target image segmentation model and a first target image group, acquires a third target image group corresponding to the target user. It then acquires and displays VR data corresponding to the target user based on the third target image group. Specifically, it determines an intermediate 3D human model based on the external body shape feature vector corresponding to the target user and the external body shape feature vector corresponding to a preset 3D human model. It also determines a target 3D human model based on a first skeletal feature vector corresponding to the target user and a second skeletal feature vector corresponding to the intermediate 3D model. Finally, it drives the target 3D human model to complete training tasks according to the target user's movements. The invention uses external body shape feature vectors and skeletal feature vectors to determine the target 3D human body model, drives the target 3D human body model to move synchronously according to the decomposed actions completed by the target user when performing training tasks, and uses difficulty level labels to mark each decomposed action. It can be seen that the invention uses external factors such as the target user's clothing and accessories to determine the target 3D human body model, drives the target 3D human body model to move synchronously according to the decomposed actions completed by the target user when performing training tasks, and uses difficulty level labels to mark each decomposed action. This effectively reduces the impact of external factors such as the target user's clothing and accessories on data collection and processing, can more accurately restore the real action details of the target user, is conducive to improving the user experience of the target user and other users managing the target user, and also avoids the complex reconstruction process of 3D point cloud data, reducing the consumption of computing resources. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a virtual reality display method for a rehabilitation robot based on image segmentation, provided as an embodiment of the present invention;

[0025] Figure 2This is a schematic diagram of the structure of a virtual reality display system for a rehabilitation robot based on image segmentation, provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar tasks and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0028] Embodiments of the present invention provide a virtual reality display method for a rehabilitation robot based on image segmentation, such as... Figure 1 As shown, the method includes the following steps:

[0029] S1. Based on the target image segmentation model and the first target image group, obtain the third target image group corresponding to the target user. The first target image group includes several first target images, which are images of the target user collected by the rehabilitation robot during the process of the target user performing training tasks using the rehabilitation robot. The third target image group includes several third target images, which are images of the target user that are only displayed in the first target images. The target user is the user who is using the rehabilitation robot.

[0030] S2. Obtain VR data corresponding to the target user based on the third target image group corresponding to the target user and display the VR data.

[0031] By following the steps above, interference from external factors can be effectively eliminated when acquiring images of only a portion of the target user during the target user's execution of the training task. Acquiring images of only a portion of the target user can be understood as acquiring images of only the target user's limbs, so that the virtual reality displays only the target user's movements during the execution of the training task.

[0032] Specifically, step S1 includes the following steps S11-S12:

[0033] S11. Based on the target image segmentation model and the first target image group, obtain a second target image group corresponding to several initial users. The second target image group includes several second target images. The second target images are only the parts of the first target images that display the initial users. The initial users are the users in the first target images.

[0034] Specifically, the first target images in the first target image group are arranged sequentially from morning to night according to the time they were acquired.

[0035] Specifically, the second target images in the second target image group are arranged sequentially from morning to night according to the time when their corresponding first target images were acquired.

[0036] Specifically, the second target image can be understood as an image containing the initial user obtained by segmenting the first target image along the contour boundary of the initial user.

[0037] In a specific embodiment, the training task can be understood as a rehabilitation training task.

[0038] S12. If the facial features of the target user are completely consistent with the facial features of the initial user corresponding to the second target image group, then the second target image group is used as the third target image group corresponding to the target user.

[0039] Specifically, facial features include at least the position and shape of organs such as the eyes, nose, and mouth.

[0040] Through the above steps, based on the target image segmentation model and the first target image group, several second target image groups corresponding to initial users are obtained. If the facial features of the target user are completely consistent with the facial features of the initial users corresponding to the second target image group, then the second target image group is used as the third target image group corresponding to the target user. This effectively eliminates interference from external factors when only a portion of the target user's image is displayed during the target user's training task, thus improving the accuracy of obtaining the third target image corresponding to the target user. This can be understood as follows: during the target user's training task using the rehabilitation robot, multiple users may simultaneously appear within the image acquisition area of ​​the rehabilitation robot. For example, the target user (target patient) and other users (other patients) who need to perform training tasks may simultaneously appear within the image acquisition area of ​​the rehabilitation robot, or the target user and other staff members may simultaneously appear within the image acquisition area of ​​the rehabilitation robot. In this case, the first target image acquired by the rehabilitation robot will include multiple users, such as the target user and other users who need to perform training tasks. The image segmentation model is used to segment the first target image, which can be used to segment each user in the first target image as an initial user (that is, the target user and other users who need to perform training tasks are treated as different initial users, or the target user and other staff are treated as different initial users). The portion of the first target image that only shows the initial user is segmented separately as the second target image corresponding to the initial user. In multi-person scenarios, the second target image corresponding to each initial user can be accurately extracted. Then, the facial features of the initial user and the target user are compared to determine the third target image corresponding to the target user. The image that only shows the target user can be accurately extracted from the image collected by the rehabilitation robot. Therefore, through the above steps, different users can be effectively separated in multi-person scenarios, and the image of the target user can be accurately extracted. This allows for the accurate capture of the actions performed by the target user during the training task using the rehabilitation robot, providing more accurate data support for obtaining VR data corresponding to the target user.

[0041] Specifically, step S2 includes the following steps S21-S24:

[0042] S21. Input the third target image group corresponding to the target user into the preset external body shape feature vector extraction model to obtain the external body shape feature vector corresponding to the target user.

[0043] Specifically, the external body shape feature vector is a vector that represents the features of the external body shape. The features of the external body shape include, but are not limited to, height, body proportions, body curves, body type, shoulder width, chest circumference, waist circumference, and hip circumference.

[0044] Specifically, the preset external body shape feature vector extraction model is a model obtained by those skilled in the art through training feature extraction algorithms such as CNN (convolutional neural network) for the task of external body shape feature vector extraction.

[0045] S22. Determine several intermediate 3D human models based on several preset 3D human models, wherein the intermediate 3D human models are the preset 3D human models whose vector similarity between the corresponding external body shape feature vector and the target user's corresponding external body shape feature vector is not less than a preset vector similarity threshold.

[0046] Specifically, the external body shape feature vector corresponding to the preset 3D human body model is a vector obtained by vectorizing the external body shape features of the preset 3D human body model.

[0047] Furthermore, the external body shape feature vector corresponding to the preset 3D human body model has the same vector dimension as the external body shape feature vector corresponding to the target user.

[0048] S23. Determine the target 3D human body model based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vectors corresponding to several intermediate 3D human body models. The first skeletal feature vector includes the skeletal feature vector value of each bone in the target user's body, and the second skeletal feature vector includes the skeletal feature vector value of each bone in the intermediate 3D human body model.

[0049] Specifically, the skeletal feature vector value represents the numerical value of the skeletal feature. For example, if the skeletal feature is length, the skeletal feature vector value represents the length of the bone.

[0050] Furthermore, the bone features corresponding to all bone feature vector values ​​are the same feature; for example, the bone feature corresponding to each bone feature vector value is length.

[0051] S24. Drive the target 3D human body model to move synchronously according to the decomposed actions completed by the target user when performing the training task, and use difficulty level labels to mark each decomposed action.

[0052] Specifically, the breakdown of actions performed by the target user during the training task can be determined using several images collected by the rehabilitation robot during the target user's training task.

[0053] Specifically, the training task consists of several decomposed movements.

[0054] Specifically, the difficulty level labels include hard and easy.

[0055] Furthermore, difficulties indicate that the target user's actions are not standard and they exhibit difficulties in actual execution during the process of completing the decomposed actions.

[0056] Furthermore, simply put, it means that the target user performs the decomposed actions in a standard manner and can easily complete the decomposed actions.

[0057] Specifically, after step S23 and before step S24, the method further includes: adjusting the data of the corresponding bones in the target 3D human body model based on the target user's skeletal acquisition data, so that the target 3D human body model is more similar to the target user. For example, if the skeletal acquisition data shows that a certain bone of the target user is bent to the left by 2 degrees, then the bone in the target 3D human body model corresponding to that bone is adjusted to be bent to the left by 2 degrees.

[0058] Through the above steps, an intermediate 3D human body model is determined based on the external body shape feature vector corresponding to the target user and the external body shape feature vector corresponding to the preset 3D human body model. A target 3D human body model is determined based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vector corresponding to the intermediate 3D model. The target 3D human body model is then driven to move synchronously according to the decomposed movements performed by the target user during the training task. Each decomposed movement is labeled with a difficulty level tag, effectively reducing the impact of external factors such as the target user's clothing and accessories on data acquisition and processing. This allows for a more accurate reproduction of the target user's realistic movement details. Furthermore, the synchronous movement of the target 3D human body model and the labeling of difficulty levels enhance the visualization and interactivity of the training process, providing more intuitive movement feedback for the target user and other users managing the target user. This improves the user experience for both the target user and other users managing the target user, while also avoiding the complex reconstruction process of 3D point cloud data and reducing computational resource consumption.

[0059] Specifically, step S23 includes the following steps S231-S233:

[0060] S231. Obtain the skeleton identifier list W = {W1, W2, ..., W...} i , ..., W m}, W i The bone identifier is the i-th bone in the human body, where i ranges from 1 to m, and m is the number of bones in the human body. The bone identifier can be understood as the name of the bone, such as the distal phalanx of the left little finger, the shinbone of the right leg, etc.

[0061] S232. Obtain the first skeletal feature vector A corresponding to the target user and the second skeletal feature vector B corresponding to the j-th intermediate 3D human body model. j Vector matching degree E between j Where A = {A1, A2, ..., A} i , ..., A m}, A iFor the target user body, W i The corresponding skeletal feature vector value, j takes values ​​from 1 to n, where n is the number of intermediate 3D human models, B j ={B j1 B j2 , ..., B ji , ..., B jm}, B ji For the j-th intermediate 3D human body model, W i The corresponding skeletal feature vector value, E j Meets the following conditions:

[0062] E j =(Σ m i=1 (D i ×(A i -B ji ) 2 )) 1 / 2 , where D i In order to enable W to perform training tasks that the target users are currently undertaking i The corresponding importance weight of the bones.

[0063] Specifically, importance weights are used to represent the importance of bones in training tasks. The higher the importance weight, the more important the bones are in training tasks.

[0064] Specifically, D1, D2, ..., D i , ..., D m There are importance weights of 1 or less, and there are also importance weights of less than 1.

[0065] S233. If max(E1, E2, ..., E... j , ..., E n )≥E 0 Then max(E1, E2, ..., E j , ..., E n The corresponding intermediate 3D human body model is used as the target 3D human body model, and max() is the function to obtain the maximum value. 0 The preset matching degree threshold is, as those skilled in the art know, a threshold that is preset by those skilled in the art based on vector matching degree and the importance weight of bones, and will not be elaborated here.

[0066] Specifically, step S233 further includes: if max(E1, E2, ..., E j , ..., E n ) < E 0 If the system fails to detect the target user, a prompt message will be issued, indicating that there is no 3D human body model matching the target user.

[0067] Through the above steps, different bones have varying degrees of importance in different training tasks. For example, in upper limb rehabilitation training tasks, lower limb bones are not important, and in lower limb rehabilitation training tasks, arm bones are not important. Therefore, based on the importance weight of bones in the training task, the bone feature vector values ​​of the target user's bones, and the bone feature vector values ​​of the bones in the intermediate 3D model, the vector matching degree between the first bone feature vector corresponding to the target user and the second bone feature vector corresponding to the intermediate 3D human body model is obtained. In the process of obtaining the vector matching degree, the importance of bones is taken into consideration, which helps to improve the accuracy of the vector matching degree. The intermediate 3D human body model corresponding to the largest vector matching degree that is not less than the preset matching degree threshold is used as the target 3D human body model. It is possible to obtain a highly personalized 3D human body model according to the actual bone characteristics of the target user and the needs of the training task being performed by the target user, ensuring that the target 3D human body model has a high similarity to the target user and can accurately reflect the target user's body shape and bone structure.

[0068] In one specific embodiment, step S232 further includes the following steps S10-S20 to obtain D. i :

[0069] S10. Obtain the list set of bone weight values ​​corresponding to the preset training task, C={C1, C2, ..., C...} e , ..., C f}, C e ={C e1 C e2 , ..., C ei , ..., C em}, C e This is a list of bone weight values ​​mapped to the e-th preset training task, where e ranges from 1 to f, f is the number of preset training tasks, and C... ei For the e-th preset training task, W i The corresponding bone weight value.

[0070] Specifically, the bone weight value is used to represent the importance of a bone in a preset training task. The larger the bone weight value, the more important the corresponding bone is in the preset training task.

[0071] S20. If the training task being performed by the target user is related to C... e If the corresponding preset training task is the same, then C will be... ei As D i .

[0072] By following the steps above, a list of bone weight values ​​corresponding to a preset training task is obtained. Based on the training task being performed by the target user and the preset training task, the importance weight of bones in the training task being performed by the target user is obtained from the list of bone weight values ​​corresponding to the preset training task, which is convenient and quick.

[0073] In one specific embodiment, step S10 further includes the following steps S101-S104 to obtain C. ei :

[0074] S101. Obtain the list of decomposed action identifiers F corresponding to the e-th preset training task. e ={F e1 F e2 , ..., F eg , ..., F eh(e)}, F eg Let g be the decomposition action identifier of the g-th decomposition action in the e-th preset training task, where g takes values ​​from 1 to h(e), and h(e) is the number of decomposition actions in the e-th preset training task.

[0075] Specifically, the pre-set training task consists of several decomposed actions.

[0076] S102, G e1 G e2 , ..., G eg , ..., G eh(e) The union of these sets serves as the list of key skeleton identifiers H corresponding to the e-th preset training task. e ={H e1 H e2 H ek H et(e)}, where G eg For F eg The corresponding list of key skeleton markers includes several key skeleton markers. Key skeleton markers belong to the W category, and the bones corresponding to these key skeleton markers are the bones needed in their respective decomposed movements. H k The key skeleton identifier is the kth key skeleton identifier corresponding to the preset training task. The value of k is from 1 to t(e), and t(e) is the number of key skeleton identifiers corresponding to the eth preset training task.

[0077] S103, G e1 G e2 , ..., G eg , ..., G eh(e) Contains H ek The number of key skeletal markers in the list as H ek The corresponding frequency value R ek .

[0078] S104, If W i Corresponding skeleton and H ek If the corresponding bones are the same bone, then according to R... ek Get C ei C ei Meets the following conditions:

[0079] C ei =V 0 +R ek / 10, where V 0 The preset basic skeletal weights are known to those skilled in the art. The preset basic skeletal weights are values ​​less than 1 that are pre-set by those skilled in the art according to actual needs, such as 0.1, 0.2, 0.3, which will not be elaborated here.

[0080] Specifically, step S104 further includes: if W i Corresponding skeleton and H e1 H e2 H ek H et(e) If none of the corresponding bones are the same bone, then let C ei =V 0 .

[0081] Through the above steps, the union of the lists of key bone markers corresponding to all decomposed action markers corresponding to the preset training task is taken as the key bone marker list corresponding to the preset training task. Based on the number of key bone markers including key bone markers, the frequency of occurrence of each key bone marker is determined. The higher the frequency of occurrence of a key bone marker, the more frequently the corresponding bone is used in the preset training task, and the more important the corresponding bone is. Therefore, when the bone corresponding to a bone marker is the same bone as the corresponding bone of a key bone marker in the preset training task, the bone weight value of the corresponding bone marker in the preset training task is obtained based on the frequency of occurrence of the key bone marker and the basic bone weight. If the bone corresponding to a bone marker is not the same bone as any of the corresponding bones of a key bone marker in the preset training task, the bone weight value of the corresponding bone marker in the preset training task is determined to be the basic bone weight value, so that the bone weight value can represent the importance of the bone in the preset training task. The larger the bone weight value, the higher the importance of the corresponding bone in the preset training task.

[0082] In a specific embodiment, step S24 further includes the following step: obtaining the difficulty level label corresponding to the decomposition action:

[0083] S241. When the target user completes the decomposed action, the decomposed action is taken as the target action and the execution time period corresponding to the target action is obtained. The start time of the execution time period is the time when the target user starts to execute the target action, and the end time of the specified time period is the time when the target user completes the target action.

[0084] S242. If the time point at which the first target image corresponding to the third target image corresponding to the target user is collected is within the execution time period corresponding to the target action, then the third target image is used as the fourth target image corresponding to the target user to obtain the fourth target image group corresponding to the target user, and the fourth target image group includes several fourth target images.

[0085] Specifically, the fourth target images in the fourth target image group are arranged sequentially from morning to night according to the time when their corresponding first target images were acquired.

[0086] S243. Obtain the list of key point identifiers corresponding to the target action, L={L1, L2, ..., L...} x , ..., L p}, L x Let x be the x-th key point corresponding to the target action, where x ranges from 1 to p, and p is the number of key points corresponding to the target action. Key points are joints or bones whose positions change during the execution of the target action by the target user.

[0087] S244, L x The corresponding feature vector of the first motion trajectory and L x The distance between the feature vectors of the corresponding second motion trajectory is taken as L. x The corresponding standardized score M x .

[0088] Specifically, according to the order of the fourth target image in the fourth target image group corresponding to the target user, L is sequentially... x Connect the corresponding key points in the fourth target image to obtain L. x The corresponding first motion trajectory.

[0089] Specifically, if the training task being performed by the target user is the same as the preset training task, then among the several decomposed actions corresponding to the preset training task, the decomposed action that is the same as the target action is taken as the key action, and L is set as the key action. x The corresponding key points are the motion trajectories in the standard action operation video corresponding to the key actions, which are used as L. x The corresponding second motion trajectory.

[0090] Specifically, each sub-action of the training task corresponds to a standard action operation video.

[0091] Specifically, L x The corresponding first motion trajectory is input into a preset trajectory feature vector extraction model to obtain L. x The feature vector of the corresponding first motion trajectory.

[0092] Specifically, L x The corresponding second motion trajectory is input into a preset trajectory feature vector extraction model to obtain L. x The feature vector of the corresponding second motion trajectory.

[0093] Specifically, the preset trajectory feature vector extraction model is a model obtained by those skilled in the art through training feature extraction algorithms such as CNN for the trajectory feature vector extraction task.

[0094] Specifically, M x Meets the following conditions:

[0095] M x =(Σ q y=1 (M 0 xy -M 1 xy ) 2 ) 1 / 2 M 0 xy For L x The y-th eigenvector value in the eigenvector of the corresponding first motion trajectory, where y ranges from 1 to q, and q is the number of eigenvector values ​​in the eigenvector of the first motion trajectory, M. 1 xy For L x The y-th feature vector value in the feature vector of the corresponding second motion trajectory, L x The corresponding feature vector of the first motion trajectory and L x The feature vectors of the corresponding second motion trajectory have the same vector dimension.

[0096] Specifically, the smaller the standardization score, the more similar the first motion trajectory corresponding to the key point and the second motion trajectory corresponding to the key point are. This can be understood as the more standard the first motion trajectory corresponding to the key point is.

[0097] S245. Input the facial feature vector corresponding to the facial features of the target user in each fourth target image into the preset difficulty score acquisition model to obtain the difficulty score N corresponding to the target action.

[0098] Specifically, the preset difficulty score acquisition model is a model obtained by those skilled in the art through training a neural network model for the difficulty score acquisition task.

[0099] Specifically, the higher the difficulty score, the more difficult it is for the target user to perform the target action.

[0100] S246. When Q≥U, the difficulty level label of the decomposed action corresponding to the target action is determined to be difficult; when Q<U, the difficulty level label of the decomposed action corresponding to the target action is determined to be easy. Here, Q is the comprehensive judgment score corresponding to the target action, and U is the preset judgment score. Q meets the following conditions:

[0101] Q=(Σ p x=1 M x / p)+N.

[0102] Specifically, as those skilled in the art will know, the preset judgment score is a score that is pre-set by those skilled in the art based on the standardized score and the difficulty score, and will not be elaborated here.

[0103] Through the above steps, based on the first motion trajectory corresponding to the keypoint identifiers of the target action and the second motion trajectory corresponding to the keypoint identifiers of the target action, a standardized score corresponding to the keypoint representation is obtained. The smaller the standardized score, the more similar the first and second motion trajectories of the keypoints are. The facial feature vector corresponding to the facial features of the target user in each fourth target image is input into a preset difficulty score acquisition model to obtain the difficulty score corresponding to the target action. The larger the difficulty score, the more difficult the target user's performance of the target action. This includes changes in the target user's facial features (such as changes in eyebrows, eyes, and corners of the mouth). It can display the facial expressions of the target user when performing the target action (e.g., frowning, clenching teeth, or downturned corners of the mouth). These facial expressions can intuitively reflect the psychological and physiological state of the target user when completing the target action. Therefore, the difficulty score not only quantifies the difficulty of the action, but also accurately reflects the target user's rehabilitation status. The higher the difficulty score, the stronger the facial expressions shown by the changes in the target user's facial features, reflecting that they may encounter greater obstacles or discomfort during the execution of the action. For example, the higher the difficulty score corresponding to a more complex shoulder and hip rehabilitation action performed by the target user, the more uncomfortable the target user feels during the execution of the shoulder and hip rehabilitation action, which can reflect the user's poor rehabilitation status.

[0104] Furthermore, based on the standardized score corresponding to the key point of the target action and the difficulty score corresponding to the target action, the difficulty level label of the decomposed action corresponding to the target action is determined. The difficulty level label includes difficult and easy. Difficult indicates that the target user's actions are not standard and show difficulty in actual execution, while easy indicates that the target user's actions are standard and can be easily completed. By combining the target user's facial features and the motion trajectory of key points to determine the difficulty level label of the decomposed actions corresponding to the target action, the accuracy of difficulty level label determination is improved, and the versatility of the system is enhanced. This can be understood as combining the target user's facial expressions and the motion trajectory of key points when performing the target action to determine the difficulty level label of the decomposed actions corresponding to the target action. This helps improve the accuracy of determining the difficulty level label of the decomposed actions corresponding to the target action. Furthermore, through the synchronous movement of the target 3D human model and the annotation of difficulty level labels, the visualization and interactivity of the training process are enhanced. This allows the target user and other users managing the target user to directly observe the difficult or non-standard decomposed actions, and also to identify the key points that cause the difficulty or non-standard action. This not only reflects the target user's rehabilitation status but also accurately identifies specific actions (such as difficult or non-standard decomposed actions) that affect the rehabilitation status, providing more intuitive action feedback for the target user and other users managing the target user, thus improving the user experience for both the target user and other users managing the target user.

[0105] Embodiments of the present invention also provide a virtual reality display system for a rehabilitation robot based on image segmentation, such as... Figure 2 As shown, the virtual reality display system includes:

[0106] Image segmentation module 1 is used to obtain a third target image group corresponding to the target user based on the target image segmentation model and the first target image group. The first target image group includes several first target images, which are images of the target user collected by the rehabilitation robot during the process of the target user performing training tasks using the rehabilitation robot. The third target image group includes several third target images, which are images of the target user that are only displayed in the first target images. The target user is the user who is using the rehabilitation robot.

[0107] VR data processing module 2 is used to obtain VR data corresponding to the target user based on the third target image group corresponding to the target user and display the VR data.

[0108] Specifically, the virtual reality display system also includes: a rehabilitation robot and a VR display device.

[0109] Specifically, the image segmentation module 1 also includes:

[0110] The image segmentation unit is used to obtain a second target image group corresponding to several initial users based on the target image segmentation model and the first target image group. The second target image group includes several second target images, which are only the parts of the first target images that display the initial users. The initial users are the users in the first target images.

[0111] Specifically, the first target images in the first target image group are arranged sequentially from morning to night according to the time they were acquired.

[0112] Specifically, the second target images in the second target image group are arranged sequentially from morning to night according to the time when their corresponding first target images were acquired.

[0113] Specifically, the second target image can be understood as an image containing the initial user obtained by segmenting the first target image along the contour boundary of the initial user.

[0114] In one specific embodiment, the training task can be understood as rehabilitation exercise.

[0115] The third target image group acquisition unit is used to take the second target image group as the third target image group corresponding to the target user if the facial features of the target user are completely consistent with the facial features of the initial user corresponding to the second target image group.

[0116] Specifically, facial features include at least the position and shape of organs such as the eyes, nose, and mouth.

[0117] Specifically, VR data processing module 2 also includes:

[0118] The external body shape feature vector acquisition unit is used to input the third target image group corresponding to the target user into the preset external body shape feature vector extraction model to obtain the external body shape feature vector corresponding to the target user.

[0119] Specifically, the external body shape feature vector is a vector that represents the features of the external body shape. The features of the external body shape include, but are not limited to, height, body proportions, body curves, body type, shoulder width, chest circumference, waist circumference, and hip circumference.

[0120] Specifically, the preset external body shape feature vector extraction model is a model obtained by those skilled in the art through training feature extraction algorithms such as CNN (convolutional neural network) for the task of external body shape feature vector extraction.

[0121] The intermediate 3D human body model determination unit is used to determine a number of intermediate 3D human body models based on a number of preset 3D human body models. The intermediate 3D human body model is the preset 3D human body model whose vector similarity between the corresponding external body shape feature vector and the target user's corresponding external body shape feature vector is not less than a preset vector similarity threshold.

[0122] Specifically, the external body shape feature vector corresponding to the preset 3D human body model is a vector obtained by vectorizing the external body shape features of the preset 3D human body model.

[0123] Furthermore, the external body shape feature vector corresponding to the preset 3D human body model has the same vector dimension as the external body shape feature vector corresponding to the target user.

[0124] The target 3D human body model determination unit is used to determine the target 3D human body model based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vectors corresponding to several intermediate 3D human body models. The first skeletal feature vector includes the skeletal feature vector value of each bone in the target user's body, and the second skeletal feature vector includes the skeletal feature vector value of each bone in the intermediate 3D human body models.

[0125] Specifically, the skeletal feature vector value represents the numerical value of the skeletal feature. For example, if the skeletal feature is length, the skeletal feature vector value represents the length of the bone.

[0126] Furthermore, the bone features corresponding to all bone feature vector values ​​are the same feature; for example, the bone feature corresponding to each bone feature vector value is length.

[0127] The target 3D human model driving unit is used to drive the target 3D human model to move synchronously according to the decomposed actions completed by the target user when performing training tasks, and to label each decomposed action with difficulty level labels.

[0128] Specifically, the breakdown of actions performed by the target user during the training task can be determined using several images collected by the rehabilitation robot during the target user's training task.

[0129] Specifically, the training task consists of several decomposed movements.

[0130] Specifically, the difficulty level labels include hard and easy.

[0131] Furthermore, difficulties indicate that the target user's actions are not standard and they exhibit difficulties in actual execution during the process of completing the decomposed actions.

[0132] Furthermore, simply put, it means that the target user performs the decomposed actions in a standard manner and can easily complete the decomposed actions.

[0133] Specifically, VR data processing module 2 also includes a target 3D human body model adjustment unit, which adjusts the data of the corresponding bones in the target 3D human body model based on the target user's skeletal acquisition data. For example, if the skeletal acquisition data shows that a certain bone of the target user is bent to the left by 2 degrees, then the bone in the target 3D human body model corresponding to that bone will be adjusted to be bent to the left by 2 degrees.

[0134] Specifically, the target 3D human body model determination unit also includes:

[0135] The skeletal identifier list retrieval sub-unit is used to retrieve the skeletal identifier list W = {W1, W2, ..., W...} i , ..., W m}, W i The bone identifier is the i-th bone in the human body, where i ranges from 1 to m, and m is the number of bones in the human body. The bone identifier can be understood as the name of the bone, such as the distal phalanx of the left little finger, the shinbone of the right leg, etc.

[0136] The vector matching degree acquisition subunit is used to obtain the first skeletal feature vector A corresponding to the target user and the second skeletal feature vector B corresponding to the j-th intermediate 3D human body model. j Vector matching degree E between j Where A = {A1, A2, ..., A} i , ..., A m}, A i For the target user body, W i The corresponding skeletal feature vector value, j takes values ​​from 1 to n, where n is the number of intermediate 3D human models, B j ={B j1 B j2 , ..., B ji , ..., B jm}, B ji For the j-th intermediate 3D human body model, W i The corresponding skeletal feature vector value, E j Meets the following conditions:

[0137] E j =(Σ m i=1 (D i ×(A i -B ji ) 2 )) 1 / 2 , where D i In order to enable W to perform training tasks that the target users are currently undertaking i The corresponding importance weight of the bones.

[0138] Specifically, importance weights are used to represent the importance of bones in training tasks. The higher the importance weight, the more important the bones are in training tasks.

[0139] Specifically, D1, D2, ..., D i , ..., D m There are importance weights of 1 or less, and there are also importance weights of less than 1.

[0140] The target 3D human body model is used to determine sub-units, for if max(E1, E2, ..., E j , ..., E n )≥E 0 Then max(E1, E2, ..., E j , ..., E n The corresponding intermediate 3D human body model is used as the target 3D human body model, and max() is the function to obtain the maximum value. 0 For the preset matching threshold, if max(E1, E2, ..., E j , ..., E n ) < E 0 If no matching 3D human body model is found, a prompt message will be issued. The prompt message indicates that no matching 3D human body model is found. As those skilled in the art know, the preset matching degree threshold is a threshold preset by those skilled in the art based on the vector matching degree and the importance weight of the skeleton, which will not be elaborated here.

[0141] In one specific embodiment, the vector matching degree acquisition subunit further includes:

[0142] The sub-unit for retrieving the bone weight value mapping list set is used to retrieve the bone weight value mapping list set C={C1, C2, ..., C...} corresponding to the preset training task. e , ..., C f}, C e ={C e1 C e2 , ..., C ei , ..., C em}, C e This is a list of bone weight values ​​mapped to the e-th preset training task, where e ranges from 1 to f, f is the number of preset training tasks, and C... ei For the e-th preset training task, W i The corresponding bone weight value.

[0143] Specifically, the bone weight value is used to represent the importance of a bone in a preset training task. The larger the bone weight value, the more important the corresponding bone is in the preset training task.

[0144] Importance weight acquisition subunit, used to determine if the target user's ongoing training task is related to C. e If the corresponding preset training task is the same, then C will be... ei As D i .

[0145] In one specific embodiment, the subunit for obtaining the bone weight value mapping list set further includes:

[0146] The sub-unit for obtaining the decomposed action label list is used to obtain the decomposed action label list F corresponding to the e-th preset training task. e ={F e1 F e2 , ..., F eg , ..., F eh(e)}, F eg Let g be the decomposition action identifier of the g-th decomposition action in the e-th preset training task, where g takes values ​​from 1 to h(e), and h(e) is the number of decomposition actions in the e-th preset training task.

[0147] Specifically, the pre-set training task consists of several decomposed actions.

[0148] Key skeleton identifier list to obtain sub-units, used to G e1 G e2 , ..., G eg , ..., G eh(e) The union of these sets serves as the list of key skeleton identifiers H corresponding to the e-th preset training task. e ={H e1 H e2 H ek H et(e)}, where G eg For F eg The corresponding list of key skeleton markers includes several key skeleton markers. Key skeleton markers belong to the W category, and the bones corresponding to these key skeleton markers are the bones needed in their respective decomposed movements. H k The key skeleton identifier is the kth key skeleton identifier corresponding to the preset training task. The value of k is from 1 to t(e), and t(e) is the number of key skeleton identifiers corresponding to the eth preset training task.

[0149] The frequency value acquisition subunit is used to obtain G. e1 G e2 , ..., G eg , ..., G eh(e) Contains H ek The number of key skeletal markers in the list as H ek The corresponding frequency value R ek .

[0150] The bone weight value acquisition sub-unit is used if W i Corresponding skeleton and H ek If the corresponding bones are the same bone, then according to R... ek Get C ei C ei Meets the following conditions:

[0151] C ei =V 0 +R ek / 10, where V 0 The preset basic weights of the skeleton; if W i Corresponding skeleton and H e1 H e2 H ek H et(e) If none of the corresponding bones are the same bone, then let C ei =V 0 As those skilled in the art will know, the preset basic skeletal weights are values ​​less than 1 that are pre-set by those skilled in the art according to actual needs, such as 0.1, 0.2, 0.3, which will not be elaborated here.

[0152] In one specific embodiment, the target 3D human body model driving unit further includes:

[0153] The execution time period acquisition sub-unit is used to acquire the execution time period corresponding to the target action when the target user completes the decomposed action. The start time of the execution time period is the time when the target user starts to execute the target action, and the end time of the specified time period is the time when the target user completes the target action.

[0154] The fourth target image acquisition subunit is used to acquire the fourth target image group corresponding to the target user if the time point when the first target image corresponding to the third target image corresponding to the target user is within the execution time period corresponding to the target action. The fourth target image group includes several fourth target images.

[0155] Specifically, the fourth target images in the fourth target image group are arranged sequentially from morning to night according to the time when their corresponding first target images were acquired.

[0156] The keypoint identifier list retrieval sub-unit is used to retrieve the keypoint identifier list L={L1, L2, ..., L...} corresponding to the target action. x , ..., L p}, L xLet x be the x-th key point corresponding to the target action, where x ranges from 1 to p, and p is the number of key points corresponding to the target action. Key points are joints or bones whose positions change during the execution of the target action by the target user.

[0157] Standardized score acquisition sub-unit, used to obtain L x The corresponding feature vector of the first motion trajectory and L x The distance between the feature vectors of the corresponding second motion trajectory is taken as L. x The corresponding standardized score M x .

[0158] Specifically, according to the order of the fourth target image in the fourth target image group corresponding to the target user, L is sequentially... x Connect the corresponding key points in the fourth target image to obtain L. x The corresponding first motion trajectory.

[0159] Specifically, if the training task being performed by the target user is the same as the preset training task, then among the several decomposed actions corresponding to the preset training task, the decomposed action that is the same as the target action is taken as the key action, and L is set as the key action. x The corresponding key points are the motion trajectories in the standard action operation video corresponding to the key actions, which are used as L. x The corresponding second motion trajectory.

[0160] Specifically, each sub-action of the training task corresponds to a standard action operation video.

[0161] Specifically, L x The corresponding first motion trajectory is input into a preset trajectory feature vector extraction model to obtain L. x The feature vector of the corresponding first motion trajectory.

[0162] Specifically, L x The corresponding second motion trajectory is input into a preset trajectory feature vector extraction model to obtain L. x The feature vector of the corresponding second motion trajectory.

[0163] Specifically, the preset trajectory feature vector extraction model is a model obtained by those skilled in the art through training feature extraction algorithms such as CNN for the trajectory feature vector extraction task.

[0164] Specifically, M x Meets the following conditions:

[0165] M x =(Σ q y=1 (M 0xy -M 1 xy ) 2 ) 1 / 2 M 0 xy For L x The y-th eigenvector value in the eigenvector of the corresponding first motion trajectory, where y ranges from 1 to q, and q is the number of eigenvector values ​​in the eigenvector of the first motion trajectory, M. 1 xy For L x The y-th feature vector value in the feature vector of the corresponding second motion trajectory, L x The corresponding feature vector of the first motion trajectory and L x The feature vectors of the corresponding second motion trajectory have the same vector dimension.

[0166] Specifically, the smaller the standardization score, the more similar the first motion trajectory corresponding to the key point and the second motion trajectory corresponding to the key point are. This can be understood as the more standard the first motion trajectory corresponding to the key point is.

[0167] The difficulty score acquisition subunit is used to input the facial feature vector corresponding to the facial features of the target user in each fourth target image into the preset difficulty score acquisition model in order to obtain the difficulty score N corresponding to the target action.

[0168] Specifically, the preset difficulty score acquisition model is a model obtained by those skilled in the art through training a neural network model for the difficulty score acquisition task.

[0169] Specifically, the higher the difficulty score, the more difficult it is for the target user to perform the target action.

[0170] The difficulty level label acquisition sub-unit is used to determine the difficulty level label of the decomposed action corresponding to the target action as "difficult" when Q≥U, and as "easy" when Q<U. Here, Q is the comprehensive judgment score corresponding to the target action, and U is the preset judgment score. Q meets the following conditions:

[0171] Q=(Σ p x=1 M x / p)+N.

[0172] Specifically, as those skilled in the art will know, the preset judgment score is a score that is pre-set by those skilled in the art based on the standardized score and the difficulty score, and will not be elaborated here.

[0173] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store a computer program related to implementing a method in the method embodiments, the computer program being loaded and executed by the processor to implement the method provided in the above embodiments.

[0174] Embodiments of the present invention also provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided in the above embodiments.

[0175] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0176] This invention provides a virtual reality display method and system for a rehabilitation robot based on image segmentation. The method, based on a target image segmentation model and a first target image group, acquires a third target image group corresponding to the target user. It then acquires and displays VR data corresponding to the target user based on the third target image group. Specifically, it determines an intermediate 3D human model based on the external body shape feature vector corresponding to the target user and the external body shape feature vector corresponding to a preset 3D human model. It also determines a target 3D human model based on a first skeletal feature vector corresponding to the target user and a second skeletal feature vector corresponding to the intermediate 3D model. Finally, it drives the target 3D human model to complete training tasks according to the target user's movements. The invention uses external body shape feature vectors and skeletal feature vectors to determine the target 3D human body model, drives the target 3D human body model to move synchronously according to the decomposed actions completed by the target user when performing training tasks, and uses difficulty level labels to mark each decomposed action. It can be seen that the invention uses external factors such as the target user's clothing and accessories to determine the target 3D human body model, drives the target 3D human body model to move synchronously according to the decomposed actions completed by the target user when performing training tasks, and uses difficulty level labels to mark each decomposed action. This effectively reduces the impact of external factors such as the target user's clothing and accessories on data collection and processing, can more accurately restore the real action details of the target user, is conducive to improving the user experience of the target user and other users managing the target user, and also avoids the complex reconstruction process of 3D point cloud data, reducing the consumption of computing resources.

[0177] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A virtual reality display method for a rehabilitation robot based on image segmentation, characterized in that, The method includes the following steps: S1. Based on the target image segmentation model and the first target image group, obtain the third target image group corresponding to the target user. The first target image is the image of the target user collected by the rehabilitation robot during the process of the target user performing training tasks using the rehabilitation robot. The third target image is only a part of the first target image showing the target user. S2. Obtain VR data corresponding to the target user based on the third target image group corresponding to the target user and display the VR data; step S2 includes the following steps: S21. Input the third target image group corresponding to the target user into the preset external body shape feature vector extraction model to obtain the external body shape feature vector corresponding to the target user; S22. Determine several intermediate 3D human models based on several preset 3D human models, wherein the intermediate 3D human models are the preset 3D human models whose vector similarity between the corresponding external body shape feature vector and the target user's corresponding external body shape feature vector is not less than a preset vector similarity threshold. S23. Determine the target 3D human model based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vectors corresponding to several intermediate 3D human models. S24. Drive the target 3D human model to move synchronously according to the decomposed movements completed by the target user when performing the training task, and label each decomposed movement with a difficulty level label, including: When the target user completes the decomposed action, the decomposed action is taken as the target action and the corresponding execution time period is obtained; If the first target image corresponding to the third target image of the target user is collected at a time point within the execution time period, then the third target image is used as the fourth target image corresponding to the target user to obtain a fourth target image group; L x The corresponding feature vector of the first motion trajectory and L x The distance between the feature vectors of the corresponding second motion trajectory is taken as L. x The corresponding standardized score M x L x Let x be the x-th key point corresponding to the target action, 1≤x≤p, where p is the number of key points corresponding to the target action. Key points are joints or bones whose positions change during the target user's execution of the target action. The first motion trajectory is obtained based on the position of the key point in the fourth target image. The second motion trajectory is the motion trajectory of the key point in the corresponding standard action operation video. When (Σ) p x=1 M x When / p)+N≥U, the difficulty label of the decomposed action corresponding to the target action is determined to be difficult; otherwise, the difficulty label of the decomposed action corresponding to the target action is determined to be easy. U is the preset judgment score; N is the difficulty score corresponding to the target action obtained by the model based on the facial feature vector corresponding to the facial features of the target user in each fourth target image and the preset difficulty score.

2. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 1, characterized in that, In step S23, the first skeletal feature vector includes the skeletal feature vector value of each bone in the target user's body, and the second skeletal feature vector includes the skeletal feature vector value of each bone in the intermediate 3D human body model.

3. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 2, characterized in that, Step S23 includes the following steps: S231. Obtain the list of skeletal identifiers W = {W1, W2, ..., W...} i , ..., W m }, W i is the bone identifier of the i-th bone in the human body, where i ranges from 1 to m, and m is the number of bones in the human body; S232. Obtain the first skeletal feature vector A corresponding to the target user and the second skeletal feature vector B corresponding to the j-th intermediate 3D human body model. j Vector matching degree E between j Where A = {A1, A2, ..., A} i , ..., A m }, A i For the target user body, W i The corresponding skeletal feature vector value, j takes values ​​from 1 to n, where n is the number of intermediate 3D human models, B j ={B j1 B j2 , ..., B ji , ..., B jm }, B ji For the j-th intermediate 3D human body model, W i The corresponding skeletal feature vector value, E j Meets the following conditions: E j =(Σ m i=1 (D i ×(A i -B ji ) 2 )) 1 / 2 , where D i In order to enable W to perform training tasks that the target users are currently undertaking i The corresponding importance weights of the bones; S233. If max(E1, E2, ..., E... j , ..., E n )≥E 0 Then max(E1, E2, ..., E j , ..., E n The corresponding intermediate 3D human body model is used as the target 3D human body model, and max() is the function to obtain the maximum value. 0 This is the preset matching threshold.

4. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 3, characterized in that, Importance weights are used to represent the importance of bones in training tasks. The higher the importance weight, the more important the bones are in the training task.

5. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 3, characterized in that, D i Obtain it through the following steps: S10. Obtain the list set of bone weight values ​​corresponding to the preset training task, C={C1, C2, ..., C...} e , ..., C f }, C e ={C e1 C e2 , ..., C ei , ..., C em }, C e This is a list of bone weight values ​​mapped to the e-th preset training task, where e ranges from 1 to f, f is the number of preset training tasks, and C... ei For the e-th preset training task, W i The corresponding bone weight value; S20. If the training task being performed by the target user is related to C... e If the corresponding preset training task is the same, then C will be... ei As D i .

6. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 5, characterized in that, Bone weight values ​​are used to represent the importance of bones in a preset training task. The higher the bone weight value, the more important the corresponding bone is in the preset training task.

7. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 5, characterized in that, C ei Obtain it through the following steps: S101. Obtain the list of decomposed action identifiers F corresponding to the e-th preset training task. e ={F e1 F e2 , ..., F eg , ..., F eh(e) }, F eg Let g be the decomposition action identifier of the g-th decomposition action in the e-th preset training task, where g takes values ​​from 1 to h(e), and h(e) is the number of decomposition actions in the e-th preset training task. S102, G e1 G e2 , ..., G eg , ..., G eh(e) The union of these sets serves as the list of key skeleton identifiers H corresponding to the e-th preset training task. e ={H e1 H e2 H ek H et(e) }, where G eg For F eg The corresponding list of key skeleton markers includes several key skeleton markers. Key skeleton markers belong to the W category, and the bones corresponding to these key skeleton markers are the bones needed in their respective decomposed movements. H ek This is the k-th key skeleton identifier corresponding to the e-th preset training task, where k ranges from 1 to t(e), and t(e) is the number of key skeleton identifiers corresponding to the e-th preset training task. S103, G e1 G e2 , ..., G eg , ..., G eh(e) Contains H ek The number of key skeletal markers in the list as H ek The corresponding frequency value R ek ; S104, If W i Corresponding skeleton and H ek If the corresponding bones are the same bone, then according to R... ek Get C ei C ei Meets the following conditions: C ei =V 0 +R ek / 10, where V 0 The preset basic weights for the skeleton.

8. The virtual reality display method for a rehabilitation robot based on image segmentation according to claim 7, characterized in that, Step S104 also includes: if W i Corresponding skeleton and H e1 H e2 H ek H et(e) If none of the corresponding bones are the same bone, then let C ei =V 0 .

9. A virtual reality display system for a rehabilitation robot based on image segmentation, characterized in that, The system includes: The image segmentation module is used to obtain the third target image group corresponding to the target user based on the target image segmentation model and the first target image group. The first target image is an image of the target user collected by the rehabilitation robot during the process of the target user performing training tasks using the rehabilitation robot. The third image is a part of the first target image that only shows the target user. A VR data processing module is used to acquire VR data corresponding to the target user based on a third target image group corresponding to the target user and display the VR data; the VR data processing module includes: The external body shape feature vector acquisition unit is used to input the third target image group corresponding to the target user into the preset external body shape feature vector extraction model to obtain the external body shape feature vector corresponding to the target user; The intermediate 3D human body model determination unit is used to determine a number of intermediate 3D human body models based on a number of preset 3D human body models. The intermediate 3D human body model is the preset 3D human body model whose vector similarity between the corresponding external body shape feature vector and the target user's corresponding external body shape feature vector is not less than a preset vector similarity threshold. The target 3D human body model determination unit is used to determine the target 3D human body model based on the first skeletal feature vector corresponding to the target user and the second skeletal feature vectors corresponding to several intermediate 3D human body models. The target 3D human model driving unit is used to drive the target 3D human model to perform synchronized movements based on the decomposed movements completed by the target user during the training task, and to label each decomposed movement with a difficulty level label, including: When the target user completes the decomposed action, the decomposed action is taken as the target action and the corresponding execution time period is obtained; If the first target image corresponding to the third target image of the target user is collected at a time point within the execution time period, then the third target image is used as the fourth target image corresponding to the target user to obtain a fourth target image group; L x The corresponding feature vector of the first motion trajectory and L x The distance between the feature vectors of the corresponding second motion trajectory is taken as L. x The corresponding standardized score M x L x Let x be the x-th key point corresponding to the target action, 1≤x≤p, where p is the number of key points corresponding to the target action. Key points are joints or bones whose positions change during the target user's execution of the target action. The first motion trajectory is obtained based on the position of the key point in the fourth target image. The second motion trajectory is the motion trajectory of the key point in the corresponding standard action operation video. When (Σ) p x=1 M x When / p)+N≥U, the difficulty label of the decomposed action corresponding to the target action is determined to be difficult; otherwise, the difficulty label of the decomposed action corresponding to the target action is determined to be easy. U is the preset judgment score; N is the difficulty score corresponding to the target action obtained by the model based on the facial feature vector corresponding to the facial features of the target user in each fourth target image and the preset difficulty score.

Images