Amputation rehabilitation training system based on mixed reality

Abstract

The application provides a mixed reality-based amputation rehabilitation training system, which scans and photographs a healthy side of a target object's amputated part, recognizes an image of the healthy side, obtains three-dimensional structural features of a marker associated with the healthy side, photographs the marker in a mixed reality scenario, obtains an image of the marker, recognizes the image of the marker, obtains spatial position information of the marker relative to a mixed reality display device, combines the spatial position information of the mixed reality display device, matches the healthy side and a virtual object in an image display space of the mixed reality display device, and displays the virtual object in the image display space, so as to improve the visual fidelity of the healthy side of the patient to regenerate a complete limb by using the mixed reality technology according to real data of the patient, and reduce the cost of rehabilitation training and improve the training effect without relying on data of others.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mixed reality, and particularly relates to a mixed reality-based amputation rehabilitation training system. BACKGROUND

[0002] Phantom limb refers to the pain felt by the amputee or the patient with a residual limb in the absence of the limb. The existing method for treating phantom limb pain mainly includes mirror therapy, which reflects the image of the patient's healthy limb through a mirror, so that the patient's brain believes that the residual limb still exists. However, the mirror therapy can only provide a two-dimensional mirror image, and the patient is difficult to form a three-dimensional image through imagination, and is also prone to visual illusion. In order to overcome the defect that the mirror therapy cannot provide a three-dimensional image, a large amount of healthy patient limb image data is collected and a three-dimensional image is displayed at the end of the residual limb by using VR technology. This requires the configuration of a motion capture system to capture the overall limb movement of the patient, and then a high-definition display is used to display a three-dimensional image. However, in the process of implementing the mirror therapy by using the VR technology, the patient can only watch the virtual world, and the patient's visual system can only enter the logic of the virtual world, which breaks the connection between the virtual world and the real world, reduces the treatment effect, and also requires a large amount of healthy limb data to be collected, which increases the cost of the mirror therapy. SUMMARY

[0003] In view of the defects of the prior art, the present application provides a mixed reality-based amputation rehabilitation training system, which scans and photographs the healthy limb of the amputated part of the target object and identifies the image to obtain the three-dimensional structural features of the marker associated with the healthy limb. The marker is photographed in the mixed reality environment to obtain the image of the marker. The image of the marker is identified to obtain the spatial position information of the marker relative to the mixed reality display device. The spatial position information of the mixed reality display device is combined to match the healthy limb and the virtual object in the image display space of the mixed reality display device, so that the virtual object is displayed in the image display space. According to the real data of the patient, the mixed reality technology is used to improve the visual fidelity of the patient to regenerate a complete limb from the healthy limb, and the cost of rehabilitation training is reduced and the training effect is improved without relying on the data of others. The patient can wear the mixed reality display device for a long time, and the rehabilitation training can be integrated into daily life to reduce the feeling of separation between the virtual environment and the real environment during the rehabilitation training, so that the patient can always be completely immersed in the virtual rehabilitation training environment. The system integrates motion capture and image presentation, and motion capture can be realized by using the camera of the mixed reality display device, and the display of the mixed reality display device is used to realize the presentation of the immersive three-dimensional image, so that the reliability and effect of the rehabilitation training are effectively improved.

[0004] The present application provides a mixed reality-based amputation rehabilitation training system, which comprises:

[0005] a scanning module configured to scan a healthy side limb of an amputated part of a target object to obtain a first image of the healthy side limb;

[0006] a first image recognition module configured to recognize the first image to obtain a three-dimensional structural feature of a marker associated with the healthy side limb;

[0007] a mixed reality camera module configured to capture the marker based on the three-dimensional structural feature to obtain a second image of the marker;

[0008] a second image recognition module configured to recognize the second image to obtain spatial position information of the marker relative to a mixed reality display device;

[0009] a posture detection module configured to collect spatial position information of the mixed reality display device;

[0010] a matching module configured to match the healthy side limb and a virtual object in an image display space of the mixed reality display device based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device;

[0011] an image display control module configured to display the virtual object in the image display space based on a matching result of the healthy side limb and the virtual object in the image display space.

[0012] Further, the scanning module is configured to scan a healthy side limb of an amputated part of a target object to obtain a first image of the healthy side limb, including:

[0013] scanning the healthy side limb of the amputated part of the target object and a calibration object worn by the target object to obtain a first image containing the healthy side limb and the calibration object; wherein the calibration object is a proportional model corresponding to a contralateral limb of the amputated part, and the calibration object is provided with the marker.

[0014] Further, the first image recognition module is configured to recognize the first image to obtain a three-dimensional structural feature of a marker associated with the healthy side limb, including:

[0015] performing pixel contour recognition processing on the first image to obtain three-dimensional shape contour feature information of the marker.

[0016] Further, the mixed reality camera module is configured to capture the marker based on the three-dimensional structural feature to obtain a second image of the marker, including:

[0017] generate an identifiable contour of the marker in a photographing space of the mixed reality camera module based on the three-dimensional shape contour feature information of the marker; and track and photograph the marker based on the identifiable contour to obtain a second image corresponding to the healthy-side limb driving the marker to move synchronously during the movement of the healthy-side limb.

[0018] Further, the second image recognition module identifies the second image to obtain the spatial position information of the marker relative to the mixed reality display device, including:

[0019] identifying the second image to obtain the spatial position information of the marker in a photographing space coordinate system of the mixed reality camera module;

[0020] based on the relative position relationship between the mixed reality camera module and the mixed reality display device and the spatial position information of the marker in the photographing space coordinate system, obtaining the spatial position information of the marker relative to the mixed reality display device.

[0021] Further, the posture detection module collects the spatial position information of the mixed reality display device, including:

[0022] collecting motion posture data of the mixed reality display device during the process in which the target object wears the mixed reality display device; and analyzing the motion posture data to obtain the spatial position information of the mixed reality display device in a real environment space.

[0023] Further, the matching module matches the healthy-side limb and the virtual object in an image display space of the mixed reality display device based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, including:

[0024] based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, performing static matching and dynamic matching of the healthy-side limb and the virtual object in the image display space of the mixed reality display device; wherein the static matching includes static position matching of the healthy-side limb and the virtual object in the image display space of the mixed reality display device; and the dynamic matching includes action consistency matching of the healthy-side limb and the virtual object in the image display space of the mixed reality display device.

[0025] Further, the static matching of the healthy-side limb and the virtual object in the image display space of the mixed reality display device includes:

[0026] based on the spatial position information of the marker relative to the spatial position information of the mixed reality display device and the spatial position information of the mixed reality display device, obtain the spatial position information of the marker in the world coordinate system of the real environment space where the mixed reality display device is located;

[0027] based on the coordinate system transformation relationship between the world coordinate system of the real environment space where the mixed reality display device is located and the image display space coordinate system corresponding to the image display space, and the spatial position information of the marker in the world coordinate system of the real environment space where the mixed reality display device is located, obtain the spatial position information of the marker in the image display space coordinate system;

[0028] based on the spatial position information of the marker in the image display space coordinate system, perform static position matching of the healthy side limb and the virtual object in the image display space, so that the healthy side limb and the virtual object always maintain a relative position unchanged state in the image display space.

[0029] Further, the dynamic matching of the healthy side limb and the virtual object in the image display space of the mixed reality display device comprises:

[0030] based on the spatial position information of the marker in the image display space coordinate system, determine the action posture information of the marker in the image display space;

[0031] based on the action posture information of the healthy side limb in the image display space, determine the action posture information of the virtual object in the image display space, so that the action postures of the healthy side limb and the virtual object in the image display space are matched.

[0032] Further, the image display control module displays the virtual object in the image display space based on the matching result of the healthy side limb and the virtual object in the image display space, comprising:

[0033] based on the spatial position information and the action posture information of the virtual object in the image display space, display the virtual object in the image display space.

[0034] Compared with the prior art, the amputation rehabilitation training system based on mixed reality of the application scans and photographs the healthy side of the amputated part of the target object and identifies the image, obtains the three-dimensional structural features of the markers associated with the healthy side, and photographs the markers in the mixed reality environment to obtain the image of the markers; the image of the markers is identified to obtain the spatial position information of the markers relative to the mixed reality display device, and the spatial position information of the mixed reality display device is combined to match the healthy side and the virtual object in the image display space of the mixed reality display device, so that the virtual object is displayed in the image display space. According to the real data of the patient, the visual fidelity of the patient to the healthy side to regenerate a complete limb is improved by using the mixed reality technology, and the cost of rehabilitation training is reduced and the training effect is improved without relying on the data of others; the patient can wear the mixed reality display device for a long time, integrate rehabilitation training into daily life, reduce the sense of separation between virtual environment and real environment in the rehabilitation training process, and enable the patient to be completely immersed in the virtual rehabilitation training environment; the system integrates motion capture and image presentation, motion capture can be realized by using the camera of the mixed reality display device, and immersive three-dimensional image presentation can be realized by using the display of the mixed reality display device, thereby effectively improving the reliability and effect of rehabilitation training. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Figure 1 The structure schematic diagram of the amputation rehabilitation training system based on mixed reality provided by the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0038] Reference Figure 1 The structure schematic diagram of the amputation rehabilitation training system based on mixed reality provided by the present application. The amputation rehabilitation training system based on mixed reality comprises:

[0039] scan a healthy side of the amputated part of the target object to obtain a first image of the healthy side;

[0040] identify the first image to obtain a three-dimensional structural feature of a marker associated with the healthy side;

[0041] capture the marker based on the three-dimensional structural feature to obtain a second image of the marker;

[0042] identify the second image to obtain spatial position information of the marker relative to the mixed reality display device;

[0043] detect the spatial position information of the mixed reality display device;

[0044] match the healthy side and a virtual object in an image display space of the mixed reality display device based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device;

[0045] display the virtual object in the image display space based on a matching result of the healthy side and the virtual object in the image display space.

[0046] In the above manner, the amputation rehabilitation training system based on mixed reality scans and photographs a healthy side of an amputated part of a target object and identifies an image to obtain a three-dimensional structural feature of a marker associated with the healthy side. The marker is then photographed in a mixed reality scenario to obtain an image of the marker. The image of the marker is identified to obtain spatial position information of the marker relative to a mixed reality display device, and the spatial position information of the marker and the spatial position information of the mixed reality display device are combined to match the healthy side and a virtual object in an image display space of the mixed reality display device. The virtual object is then displayed in the image display space. The system improves the visual fidelity of a patient's healthy side regenerating a complete limb using mixed reality technology based on real data of the patient, without relying on data from others, thereby reducing the cost of rehabilitation training and improving the training effect. The patient can wear the mixed reality display device for a long time, integrate rehabilitation training into daily life, reduce the sense of separation between the virtual environment and the real environment during rehabilitation training, and enable the patient to be fully immersed in the virtual rehabilitation training environment at all times. The system integrates motion capture and image presentation, uses a camera provided by the mixed reality display device to realize motion capture, and uses a display provided by the mixed reality display device to realize the presentation of immersive three-dimensional images, thereby effectively improving the reliability and effect of rehabilitation training.

[0047] Optionally, the scanning module scans the contralateral limb of the amputated part of the target object to obtain a first image of the contralateral limb, including:

[0048] The scanning module scans the contralateral limb of the amputated part of the target object and the calibration object worn by the contralateral limb to obtain a first image containing the contralateral limb and the calibration object; wherein the calibration object is a proportional model corresponding to the ipsilateral limb of the amputated part, and the calibration object is provided with the marker.

[0049] In the above manner, in order to effectively and accurately visually restore the contralateral limb (i.e. the amputated part) of the amputated part of the target object such as a patient, a corresponding proportional limb model can be printed based on the original limb shape of the patient by means of 3D printing technology, which is used as a calibration object worn on the contralateral limb of the patient, and a corresponding marker is provided on the limb model. The calibration object is rigidly connected to the contralateral limb of the patient, and the marker is also rigidly connected to the calibration object, so that the marker is also rigidly connected to the contralateral limb of the patient, so that the marker moves synchronously during the movement of the contralateral limb of the patient. At this time, the spatial movement coordinates and trajectory of the marker can be equivalent to the spatial movement coordinates and trajectory of the contralateral limb of the patient. In addition, the marker can have a unique shape, such as a certain uniqueness in the overall shape or surface contour, so as to effectively and significantly distinguish the marker from other objects. By panoramic scanning the contralateral limb of the amputated part of the patient and the calibration object worn by the contralateral limb, a first image containing the contralateral limb of the patient, the calibration object and the marker is obtained, so that the marker can be effectively visually positioned. By wearing the calibration object on the contralateral limb of the patient, the movement of the contralateral limb of the patient can be accurately and reliably tracked and photographed, thereby forming activity data about the patient himself, providing a reliable basis for simulating the patient's limbs in a mixed reality scene.

[0050] Optionally, the first image recognition module identifies the first image to obtain the three-dimensional structural features of the marker associated with the contralateral limb, including:

[0051] The first image recognition module identifies the first image to obtain the three-dimensional structural features of the marker associated with the contralateral limb, including:

[0052] In the above manner, the first image is identified and processed to extract the pixel contour feature information of the first image, and the three-dimensional shape contour feature information of the marker is further extracted from the pixel contour feature information, so that the marker can be accurately visually recognized, and the three-dimensional shape contour of the marker can be used as a reference for subsequent precise tracking and photographing of the marker during the movement of the contralateral limb of the patient.

[0053] Optionally, the mixed reality camera module captures the marker based on the three-dimensional structural features to obtain a second image of the marker, including:

[0054] Based on the three-dimensional shape contour feature information of the marker, a recognizable contour of the marker in the camera space of the mixed reality camera module is generated; based on the recognizable contour, the marker is tracked and captured to obtain a second image corresponding to the situation that the healthy limb drives the marker to move synchronously during the movement of the healthy limb.

[0055] In actual operation, the three-dimensional shape contour feature information of the marker can be processed by the 3D Unity platform to generate a recognizable contour of the marker in the camera space of the mixed reality camera module (such as a camera provided by the mixed reality display device). The processing method of the 3D Unity platform for the three-dimensional shape contour feature information is a conventional technical means in the field, which will not be described in detail here. The mixed reality camera module can track and capture the marker based on the recognizable contour during the movement of the healthy limb of the patient to obtain a second image corresponding to the situation that the healthy limb of the patient drives the marker to move synchronously during the movement of the healthy limb, thereby forming activity data of the healthy limb of the patient, and providing reliable and rich data support for simulating the original limb of the patient in a mixed reality scene.

[0056] Optionally, the second image recognition module identifies the second image to obtain spatial position information of the marker relative to the mixed reality display device, including:

[0057] The second image is identified and processed to obtain spatial position information of the marker in the camera space coordinate system of the mixed reality camera module;

[0058] Based on the relative position relationship between the mixed reality camera module and the mixed reality display device and the spatial position information of the marker in the camera space coordinate system, the spatial position information of the marker relative to the mixed reality display device is obtained.

[0059] In the above manner, after the second image is captured, the second image is identified and processed in the camera space coordinate system of the mixed reality camera module to obtain the spatial position information (i.e. spatial coordinates) of the marker in the camera space coordinate system of the mixed reality camera module. Then, in combination with the relative position relationship between the mixed reality camera module and the mixed reality display device (i.e. the relative coordinate position relationship between the mixed reality camera module and the mixed reality display device in the world coordinate system) and the spatial position information of the marker in the camera space coordinate system, the spatial position information of the marker relative to the mixed reality display device is obtained, so that the marker is positionally calibrated in the virtual display space of the mixed reality display device.

[0060] Optionally, the posture detection module collects spatial position information of the mixed reality display device, including:

[0061] Collecting motion posture data of the mixed reality display device during the process that the target object wears the mixed reality display device; analyzing the motion posture data to obtain spatial position information corresponding to the mixed reality display device in a real environment space.

[0062] In the above manner, the mixed reality display device is also provided with a posture detection module such as a gyroscope or a three-axis acceleration sensor. When the mixed reality display device synchronously moves due to the movement of the patient's head during the process that the patient wears the mixed reality display device, the posture detection module collects motion posture data of the mixed reality display device, and then analyzes and processes the motion posture data to obtain spatial position information (i.e. spatial coordinates in a world coordinate system) corresponding to the mixed reality display device in a real environment space.

[0063] Optionally, the matching module matches the healthy side limb and the virtual object in the image display space of the mixed reality display device based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, including:

[0064] The matching module matches the healthy side limb and the virtual object in the image display space of the mixed reality display device based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, including static matching and dynamic matching of the marker. The static matching includes static position matching of the healthy side limb and the virtual object in the image display space of the mixed reality display device. The dynamic matching includes action consistency matching of the healthy side limb and the virtual object in the image display space of the mixed reality display device.

[0065] By the above manner, in order to ensure that the patient can watch the normal combination between the healthy side limb and the virtual object (i.e. the virtual image of the part of the limb corresponding to the amputated limb) and the coherent and coordinated movement between the healthy side limb and the virtual object in the display space provided by the mixed reality display device after the patient wears the mixed reality display device, the healthy side limb and the virtual object need to be statically matched and dynamically matched by reference in the image display space of the mixed reality display device; the static position matching of the healthy side limb and the virtual object in the image display space of the mixed reality display device is realized through the static matching, so as to ensure that the healthy side limb and the virtual object are combined effectively in the mixed reality display device; the action consistency matching of the healthy side limb and the virtual object in the image display space of the mixed reality display device is realized through the dynamic matching by reference, so that the patient can watch the limb movement image consistent with that before amputation in the image display space.

[0066] Optionally, the static matching of the healthy side limb and the virtual object in the image display space of the mixed reality display device comprises:

[0067] Based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, the spatial position information of the marker in the world coordinate system of the real environment space where the mixed reality display device is located is obtained.

[0068] Based on the coordinate system transformation relationship between the world coordinate system of the real environment space where the mixed reality display device is located and the image display space coordinate system corresponding to the image display space, and the spatial position information of the marker in the world coordinate system of the real environment space where the mixed reality display device is located, the spatial position information of the marker in the image display space coordinate system is obtained.

[0069] Based on the spatial position information of the marker in the image display space coordinate system, the static position matching of the healthy side limb and the virtual object in the image display space is performed, so that the relative position between the healthy side limb and the virtual object in the image display space remains unchanged.

[0070] By the above manner, through the above coordinate system transformation processing, the static position matching of the healthy side limb and the virtual object in the image display space is realized, so that the relative position between the healthy side limb and the virtual object in the image display space remains unchanged. In this way, the patient will not experience the separation or excessive gap between the healthy side limb and the virtual object during the image watching process in the image display space of the mixed reality display device, and the realism of the image is effectively improved.

[0071] Optionally, the dynamic matching by reference of the healthy side limb and the virtual object in the image display space of the mixed reality display device comprises:

[0072] determine the action posture information of the marker in the image display space based on the spatial position information of the marker in the image display space coordinate system;

[0073] determine the action posture information of the virtual object in the image display space based on the action posture information of the healthy side limb in the image display space, so that the action postures of the healthy side limb and the virtual object in the image display space are matched.

[0074] In the above manner, since the marker is rigidly connected with the monitored limb, the action posture information of the marker in the image display space can be directly used as the action posture information of the healthy side limb in the image display space, that is, the action posture information of the marker in the image display space is equivalent to the action posture information of the healthy side limb in the image display space. Then, the action posture information of the virtual object in the image display space is determined based on the action posture information of the healthy side limb in the image display space, so that the action postures of the virtual object in the image display space and the healthy side limb in the image display space are coordinated with each other, avoiding the situation that the action postures of the two are out of sync or not coordinated, and ensuring the coordination and authenticity of the patient watching the image through the mixed reality display device.

[0075] Optionally, the image display control module displays the virtual object in the image display space based on the matching result of the healthy side limb and the virtual object in the image display space, including:

[0076] display the virtual object in the image display space based on the position information and the action posture information of the virtual object in the image display space.

[0077] In the above manner, the virtual object is displayed in the image display space based on the position information and the action posture information of the virtual object in the image display space. When the patient wears the mixed reality display device, the patient can watch the image of the virtual object corresponding to the virtual environment displayed in the virtual environment and the healthy side limb existing in the real environment at the same time. The images of the healthy side limb and the virtual object are processed in the visual system of the patient, so that the patient can have the visual consciousness that the amputated limb part has grown out and can move freely, thereby realizing the rehabilitation training of the patient.

[0078] From the content of the above embodiment, the amputation rehabilitation training system based on mixed reality scans and photographs the healthy side of the target object's amputated part, and identifies the image, obtains the three-dimensional structure characteristics of the marker associated with the healthy side, and photographs the marker in the mixed reality environment to obtain the image of the marker; the image of the marker is identified to obtain the spatial position information of the marker relative to the mixed reality display device, and the spatial position information of the mixed reality display device is combined to match the healthy side and the virtual object in the image display space of the mixed reality display device, so as to display the virtual object in the image display space. According to the real data of the patient, the visual fidelity of the patient to the healthy side of the regenerated complete limb is improved by using the mixed reality technology, and the cost of rehabilitation training is reduced and the training effect is improved without relying on the data of others; the patient can wear the mixed reality display device for a long time, integrate the rehabilitation training into daily life, reduce the sense of separation between the virtual environment and the real environment in the rehabilitation training process, and can make the patient completely immersed in the virtual rehabilitation training environment; the system integrates motion capture and image presentation, and can realize motion capture by using the camera of the mixed reality display device, and can realize the presentation of immersive three-dimensional image by using the display of the mixed reality display device, thereby effectively improving the reliability and effect of rehabilitation training.

[0079] The above is only one specific embodiment of the present application, and any improvement made on the basis of the concept of the present application is considered to be within the protection scope of the present application.

Claims

1. A mixed reality-based amputation rehabilitation training system, including: The scanning module scans and captures images of the healthy limb at the amputation site of the target object to obtain a first image of the healthy limb. The first image recognition module recognizes the first image and obtains the three-dimensional structural features of the marker associated with the healthy limb; A mixed reality camera module, based on the three-dimensional structural features, captures an image of the landmark to obtain a second image of the landmark; The second image recognition module recognizes the second image to obtain the spatial position information of the marker relative to the mixed reality display device; The attitude detection module collects the spatial position information of the mixed reality display device, including: Collect motion posture data of the mixed reality display device during the process of the target object wearing the mixed reality display device; analyze the motion posture data to obtain the spatial position information of the mixed reality display device in the real environment space; The matching module, based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, matches the healthy limb and the virtual object in the image display space of the mixed reality display device, including: Based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, static matching and dynamic alignment matching are performed on the healthy limb and the virtual object in the image display space of the mixed reality display device; wherein, the static matching includes static position matching of the healthy limb and the virtual object in the image display space of the mixed reality display device; the dynamic alignment matching includes motion consistency matching of the healthy limb and the virtual object in the image display space of the mixed reality display device; The image display control module displays the virtual object in the image display space based on the matching result of the healthy limb and the virtual object in the image display space.

2. The amputation rehabilitation training system based on mixed reality according to claim 1, characterized in that: The scanning module scans and captures images of the unaffected limb at the amputation site of the target object, obtaining a first image of the unaffected limb, including: The healthy limb of the amputated site of the target object and the calibration object worn thereon are scanned and photographed to obtain a first image containing the healthy limb and the calibration object; wherein, the calibration object is a scale model corresponding to the affected limb of the amputated site, and the calibration object is provided with the marker.

3. The amputation rehabilitation training system based on mixed reality according to claim 2, characterized in that: The first image recognition module recognizes the first image and obtains the three-dimensional structural features of the marker associated with the healthy limb, including: The first image is subjected to pixel contour recognition processing to obtain the three-dimensional shape contour feature information of the marker.

4. The amputation rehabilitation training system based on mixed reality according to claim 3, characterized in that: The mixed reality camera module captures an image of the landmark based on the three-dimensional structural features, obtaining a second image of the landmark, including: Based on the three-dimensional shape contour feature information of the marker, a recognizable contour of the marker is generated in the camera space of the mixed reality camera module; based on the recognizable contour, the marker is tracked and photographed to obtain a second image corresponding to the movement of the marker synchronously driven by the healthy limb during the activity.

5. The amputation rehabilitation training system based on mixed reality according to claim 4, characterized in that: The second image recognition module recognizes the second image to obtain the spatial position information of the marker relative to the mixed reality display device, including: The second image is processed for recognition to obtain the spatial position information of the marker in the shooting space coordinate system of the mixed reality camera module; Based on the relative positional relationship between the mixed reality camera module and the mixed reality display device, and the spatial position information of the marker in the shooting space coordinate system, the spatial position information of the marker relative to the mixed reality display device is obtained.

6. The amputation rehabilitation training system based on mixed reality according to claim 5, characterized in that: Static matching of the healthy limb and the virtual object in the image display space of the mixed reality display device includes: Based on the spatial position information of the marker relative to the mixed reality display device and the spatial position information of the mixed reality display device, the spatial position information of the marker in the world coordinate system of the real environment space where the mixed reality display device is located is obtained; Based on the coordinate system transformation relationship between the world coordinate system of the real environment space where the mixed reality display device is located and the image display space coordinate system corresponding to the image display space, and the spatial position information of the marker in the world coordinate system of the real environment space where the mixed reality display device is located, the spatial position information of the marker in the image display space coordinate system is obtained. Based on the spatial position information of the marker in the image display space coordinate system, static position matching is performed on the healthy limb and the virtual object in the image display space, so that the healthy limb and the virtual object always maintain a constant relative position in the image display space.

7. The amputation rehabilitation training system based on mixed reality according to claim 6, characterized in that: Dynamically matching the healthy limb and the virtual object within the image display space of the mixed reality display device includes: Based on the spatial position information of the marker in the image display space coordinate system, determine the motion posture information of the marker in the image display space; Based on the movement and posture information of the healthy limb in the image display space, the movement and posture information of the virtual object in the image display space is determined so that the movement and posture of the healthy limb and the virtual object in the image display space are matched.

8. The amputation rehabilitation training system based on mixed reality according to claim 7, characterized in that: The image display control module displays the virtual object in the image display space based on the matching result of the healthy limb and the virtual object in the image display space, including: Based on the virtual object's position and motion information in the image display space, the virtual object is displayed in the image display space.

Images