Upper limb rehabilitation training method and device, electronic equipment and storage medium

Abstract

The application provides a kind of upper limb rehabilitation training method, device, electronic equipment and storage medium, it is related to medical equipment technical field, method includes: obtaining the multiple continuous eye images of the eye of user based on the preset guide action of virtual reality equipment configuration is generated;Determine the center position of eyeball included in eye image based on eye image;Determine the plane position coordinates of center position in eye image;Based on preset guide action, the length of arm of user and the plane position coordinates in multiple eye images, determine the expected motion trajectory and expected motion speed of virtual reality equipment;Based on expected motion trajectory and expected motion speed control virtual reality equipment movement, for user with virtual reality equipment carries out upper limb rehabilitation training.The above-mentioned mode, by tracking the eyeball of user, the expected motion trajectory and expected motion speed of virtual reality equipment movement are determined, realize the purpose of using eye control virtual reality equipment movement.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to an upper limb rehabilitation training method, device, electronic device, and storage medium. Background Technology

[0002] Approximately two-thirds of stroke survivors retain upper limb injuries. Studies have shown that early active rehabilitation training is beneficial for the recovery of limb motor function, and virtual reality technology is often used for upper limb rehabilitation training. Virtual reality technology has three characteristics: immersion, interactivity, and information intensity. Using virtual reality technology for rehabilitation guidance during upper limb training can effectively improve the guidance of attention and maintain the continuity of upper limb rehabilitation exercises.

[0003] When using virtual reality technology to perform motor imagery therapy for upper limb rehabilitation training, patients usually need to control the virtual reality device to make relevant movements and then imagine the movements. However, this method only focuses on patients with some muscle strength who can move voluntarily. Patients in the flaccid paralysis stage cannot control the virtual reality device, so the use of motor imagery in conjunction with virtual reality device training is still lacking. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an upper limb rehabilitation training method, device, electronic device and storage medium, which can determine the expected movement trajectory and expected movement speed of virtual reality device by tracking the user's eyeball, so as to achieve the purpose of controlling virtual reality device with eyes and overcome the problem that patients in the flaccid paralysis stage cannot control virtual reality device.

[0005] In a first aspect, embodiments of the present invention provide an upper limb rehabilitation training method, comprising: acquiring multiple consecutive eye images generated by a user's eyes based on preset guided actions configured in a virtual reality device; determining the center position of the eyeball included in the eye images based on the eye images; determining the planar position coordinates of the center position in the eye images; determining the expected movement trajectory and expected movement speed of the virtual reality device based on the preset guided actions, the user's arm length, and the planar position coordinates in the multiple eye images; and controlling the movement of the virtual reality device based on the expected movement trajectory and expected movement speed, so that the user can use the virtual reality device for upper limb rehabilitation training.

[0006] In a preferred embodiment of the present invention, the above-described method for determining the center position of the eyeball in the eye image based on the eye image includes: selecting a bounding box in the eye image to obtain a region of interest image in the eye image; performing grayscale processing on the region of interest image to obtain a grayscale image; determining the gradient direction of multiple pixels included in the grayscale image; and taking the position of the point where the lines intersect most frequently in the gradient direction as the center position of the eyeball.

[0007] In a preferred embodiment of the present invention, the above-mentioned determination of the expected motion trajectory and expected motion speed of the virtual reality device based on the preset guided action, the user's arm length, and the planar position coordinates in multiple eye images includes: determining the spatial trajectory coordinates of the robotic arm of the virtual reality device based on the preset guided action; determining the Y-direction coordinates corresponding to the multiple eye images based on the spatial trajectory coordinates and the user's arm length; and determining the expected motion trajectory and expected motion speed of the robotic arm of the virtual reality device based on the Y-direction coordinates and the planar position coordinates in the multiple eye images.

[0008] In a preferred embodiment of the present invention, the above-mentioned determination of the expected motion trajectory and expected motion speed of the robotic arm of the virtual reality device based on the Y-direction coordinate and the planar position coordinates in multiple eye images includes: determining the original motion trajectory of the robotic arm of the virtual reality device based on the Y-direction coordinate and the planar position coordinates in multiple eye images; filtering the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device; and determining the expected motion speed based on the expected motion trajectory.

[0009] In a preferred embodiment of the present invention, the above-mentioned filtering of the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device includes: determining the coordinates of a first original motion trajectory and a second original motion trajectory adjacent to the original motion trajectory; comparing the first original motion trajectory coordinates and the second original motion trajectory coordinates to determine whether the second original motion trajectory coordinates are reasonable; if they are not reasonable, then filtering the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device.

[0010] In a preferred embodiment of the present invention, the above-mentioned determination of the expected motion speed based on the expected motion trajectory includes: determining the distance between any two adjacent expected motion trajectory coordinates in the expected motion trajectory; determining the expected motion sub-velocities corresponding to any two adjacent expected motion trajectory coordinates based on multiple distances; and determining the expected motion speed based on multiple expected motion sub-velocities.

[0011] In a preferred embodiment of the present invention, the virtual reality device includes a first robotic arm and a second robotic arm. Controlling the virtual reality device to move based on a expected motion trajectory and a expected motion speed includes: performing kinematic analysis on the expected motion trajectory based on the length of the first robotic arm and the length of the second robotic arm to obtain expected motion parameters; and controlling the first robotic arm and the second robotic arm to move at the expected motion speed based on the expected motion parameters.

[0012] Secondly, embodiments of the present invention also provide an upper limb rehabilitation training device, comprising: an eye image acquisition module, used to acquire multiple consecutive eye images generated by a user's eyes based on preset guided actions configured in a virtual reality device; a center position determination module, used to determine the center position of the eyeball included in the eye images based on the eye images; a planar position coordinate determination module, used to determine the planar position coordinates of the center position in the eye images; a trajectory and speed determination module, used to determine the expected motion trajectory and expected motion speed of the virtual reality device based on the preset guided actions, the user's arm length, and the planar position coordinates in the multiple eye images; and a motion control module, used to control the motion of the virtual reality device based on the expected motion trajectory and expected motion speed, so that the user can use the virtual reality device for upper limb rehabilitation training.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the upper limb rehabilitation training method of the first aspect described above.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the upper limb rehabilitation training method described in the first aspect.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] This invention provides an upper limb rehabilitation training method, device, electronic device, and storage medium. By acquiring multiple consecutive eye images generated by a user's eyes based on preset guided actions configured in a virtual reality device, the planar coordinates of the center position of the eyeball in the eye images are determined. Then, based on the preset guided actions, the user's arm length, and the planar coordinates in the multiple eye images, the expected movement trajectory and expected movement speed of the virtual reality device are determined. This achieves the goal of controlling the virtual reality device for movement using the eyes. By controlling the movement of the virtual reality device based on the expected movement trajectory and expected movement speed, the user can use the virtual reality device for upper limb rehabilitation training, overcoming the problem that patients in the flaccid paralysis stage cannot control the virtual reality device.

[0017] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0018] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1a A flowchart of an upper limb rehabilitation training method provided in an embodiment of the present invention;

[0021] Figure 1b A structural diagram of a virtual reality device provided in an embodiment of the present invention;

[0022] Figure 2 A flowchart of another upper limb rehabilitation training method provided in an embodiment of the present invention;

[0023] Figure 3 A flowchart of another upper limb rehabilitation training method provided in an embodiment of the present invention;

[0024] Figure 4 A flowchart of another upper limb rehabilitation training method provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of an upper limb rehabilitation training device provided in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Approximately two-thirds of stroke survivors retain upper limb injuries. Studies have shown that early active rehabilitation training is beneficial for the recovery of their limb motor function. Motor imagery therapy, based on the theory of central nervous system plasticity, uses motor intention to re-establish damaged motor conduction pathways. When patients have motor intentions, the corresponding brain electrical activity also increases and generalizes, manifested as increased frequency and amplitude. Continuous stimulation may enhance or reconstruct central synapses, achieving a reorganization of the nervous system.

[0029] Virtual reality technology has three characteristics: immersion, interactivity, and information intensity. Therefore, virtual reality technology is often used to implement imagery therapy for upper limb rehabilitation training. Using virtual reality technology to guide rehabilitation during upper limb training can effectively improve the guidance effect of attention and maintain the continuity of upper limb rehabilitation training.

[0030] When using virtual reality technology to implement motor imagery therapy for upper limb rehabilitation training, graphic prompts and verbal prompts are often used to guide patients to perform related motor imagery. Patients usually need to control the virtual reality device to issue relevant instructions, and then perform motor imagery in response to the instructions. However, this method only focuses on patients with some muscle strength who can move voluntarily. Patients in the flaccid paralysis stage cannot control the virtual reality device, so the use of motor imagery in conjunction with virtual reality device training is still lacking.

[0031] Based on this, the present invention provides an upper limb rehabilitation training method, device, electronic device, and storage medium that can obtain multiple consecutive eye images generated by the user's eyes based on preset guided actions configured in the virtual reality device, determine the planar position coordinates of the center position of the eyeball in the eye images, and then determine the expected movement trajectory and expected movement speed of the virtual reality device based on the preset guided actions, the user's arm length, and the planar position coordinates in the multiple eye images. This achieves the purpose of controlling the virtual reality device for movement using the eyes, thereby controlling the movement of the virtual reality device based on the expected movement trajectory and expected movement speed, so that the user can use the virtual reality device for upper limb rehabilitation training, overcoming the problem that patients in the flaccid paralysis stage cannot control the virtual reality device.

[0032] To facilitate understanding of this embodiment, a detailed description of an upper limb rehabilitation training method disclosed in this embodiment of the invention will be provided first.

[0033] Example 1

[0034] This invention provides an upper limb rehabilitation training method. Figure 1a This is a flowchart illustrating an upper limb rehabilitation training method provided in an embodiment of the present invention. Figure 1a As shown, this upper limb rehabilitation training method may include the following steps:

[0035] Step S101: Acquire multiple consecutive eye images of the user's eyes based on preset guided actions configured by the virtual reality device.

[0036] Virtual reality devices may include cameras that can capture images of the user's eyes, producing multiple consecutive eye images.

[0037] The user's eyes can move according to the preset guided actions configured in the virtual reality device. During the eye movement, multiple consecutive eye images are generated. It can be understood that multiple consecutive eye images can form a video of the user's eye movement process, and each eye image can be one frame in the video.

[0038] Action games can be imported into virtual reality devices. These games contain preset guided actions, which users can use to imagine the movement of their upper limbs. As the user's upper limbs move, their eyes will also follow the preset guided actions.

[0039] Step S102: Determine the center position of the eyeball included in the eye image based on the eye image.

[0040] You can select the area containing the eye in the eye image, determine the position of the eyeball within the area containing the eye, and then determine the center position of the eyeball.

[0041] Step S103: Determine the planar position coordinates of the center position in the eye image.

[0042] The planar coordinates of the center position within the region containing the eye can be obtained by comparing the center position with the position of the region containing the eye in the eye image.

[0043] Step S104: Based on the preset guided actions, the user's arm length, and the planar position coordinates in multiple eye images, determine the expected motion trajectory and expected motion speed of the virtual reality device.

[0044] The planar motion trajectory of the eye following a preset guided action can be determined by the planar position coordinates in multiple eye images. The planar position coordinates can be coordinates in the X and Z directions.

[0045] The movement of a user's eyes is generated by imagining movements based on preset guided actions. For example, when a user is playing a badminton game using a virtual reality device, they imagine the swinging motion of their arm according to the preset guided actions. The user's eyes will move with the imagined position of the arm. Therefore, the angle of the user's arm rotation can be approximated as the angle of eye rotation. Thus, the user's arm can be taken as the Y-direction, and the length of the user's arm can be taken as the radius. The range of the ball corresponding to this radius is the maximum range of the user's swing. Based on the planar motion trajectory, the length of the user's arm, and the spatial trajectory coordinates corresponding to each preset guided action, each Y-direction coordinate can be deduced. By combining each Y-direction coordinate with the planar motion trajectory, the spatial motion trajectory of the eyeball can be obtained. Then, the expected motion trajectory of the virtual reality device can be determined based on the spatial motion trajectory of the eyeball, and the expected motion speed can be determined based on the time interval between the expected motion trajectory and the eye image.

[0046] Step S105: Control the movement of the virtual reality device based on the expected movement trajectory and expected movement speed, so that the user can use the virtual reality device for upper limb rehabilitation training.

[0047] It can control virtual reality devices to move according to the expected movement trajectory and speed to assist users in upper limb rehabilitation training.

[0048] Specifically, the virtual reality device may include a first robotic arm and a second robotic arm. Based on the lengths of the first and second robotic arms, kinematic analysis is performed on the expected motion trajectory to obtain the expected motion parameters. Based on the expected motion parameters, the first and second robotic arms are controlled to move at the expected motion speed.

[0049] Furthermore, in practical applications, virtual reality devices may exhibit structural biases, such as the bias of the first irregular link and the bias of the second irregular link. Therefore, based on the length of the first robotic arm, the length of the second robotic arm, and the biases of the first and second irregular links, kinematic analysis can be performed on the expected motion trajectory to obtain the expected motion parameters.

[0050] The expected motion parameters may include the coordinates of the end effector of the second robotic arm in the X, Y and Z directions, such as Px, Py and Pz.

[0051] The coordinate rotation matrices of the rotation axes of the first and second robotic arms can be determined by the DH (Denavit-Hartenberg) parameter method. The coordinate rotation matrices are then multiplied together and simplified to obtain the expected motion parameters. Based on the expected motion parameters, the motion trajectories of the first and second robotic arms are planned and the virtual reality device is controlled to move.

[0052] For ease of understanding, Figure 1b The present invention provides a structural diagram of a virtual reality device 10, wherein the virtual reality device 10 includes: an electrical control box 11, a connecting rod 12, a first irregular connecting rod 13, a second irregular connecting rod 14, a first robotic arm 15, and a second robotic arm 16.

[0053] Specifically, the electrical control box 11 houses a motor driver 111 and a main control board 112. The electrical control box 11 is rotatably connected to the first irregular connecting rod 13 via a connecting rod 12, with the rotating shaft being a rotating shaft 131. The first irregular connecting rod 13 is rotatably connected to the second irregular connecting rod 14 via a rotating shaft 141. The second irregular connecting rod 14 is rotatably connected to the first robotic arm 15 via a rotating shaft 151. The first robotic arm 15 is rotatably connected to the second robotic arm 16 via a rotating shaft 161. Through the cooperation of the motor driver 111 and the main control board 112, the rotating shafts 131, 141, 151, and 161 can be driven to rotate, thereby moving the connecting rod 12, the first irregular connecting rod 13, the second irregular connecting rod 14, the first robotic arm 15, and the second robotic arm 16.

[0054] It should be noted that, Figure 1b The structure of the virtual reality device shown is merely exemplary, and the upper limb rehabilitation training method provided by this invention is not limited to this. Figure 1b In the virtual reality device shown.

[0055] The upper limb rehabilitation training method provided in this invention acquires multiple consecutive eye images generated by the user's eyes based on preset guided actions configured in a virtual reality device. It determines the planar coordinates of the center position of the eyeball in the eye images. Then, based on the preset guided actions, the user's arm length, and the planar coordinates in the multiple eye images, it determines the expected movement trajectory and expected movement speed of the virtual reality device. This achieves the goal of controlling the virtual reality device with the eyes. By controlling the movement of the virtual reality device based on the expected movement trajectory and expected movement speed, the user can use the virtual reality device for upper limb rehabilitation training, overcoming the problem that patients in the flaccid paralysis stage cannot control the virtual reality device.

[0056] Example 2

[0057] This invention also provides another upper limb rehabilitation training method; this method is implemented based on the method of the above embodiments; the method focuses on describing the specific implementation of determining the center position of the eyeball included in the eye image based on the eye image.

[0058] Figure 2 A flowchart of another upper limb rehabilitation training method provided in an embodiment of the present invention is shown below. Figure 2 As shown, this upper limb rehabilitation training method may include the following steps:

[0059] Step S201: Select the eye image by drawing a bounding box to obtain the region of interest image in the eye image.

[0060] You can select the region containing the eye in an eye image, and the image containing the eye will be used as the region of interest.

[0061] Step S202: Perform grayscale processing on the region of interest image to obtain a grayscale image.

[0062] The region of interest can be processed in grayscale using the maximum value method, the average value method, and the weighted average method. Alternatively, a device capable of grayscale processing can be used to process the region of interest in grayscale; no specific limitation is made here.

[0063] Step S203: Determine the gradient direction of the multiple pixels included in the grayscale image.

[0064] In a grayscale image, the gradient angle between a pixel and its eight neighboring pixels is calculated, and the gradient direction is the one with the largest gradient angle.

[0065] Step S204: The position of the point where the lines intersect most frequently in the gradient direction is taken as the center position of the eyeball.

[0066] Specifically, multiple intersection points of the lines containing the gradient directions of multiple pixels can be determined, and the position of the point where the lines intersect the most among these intersection points can be taken as the center position of the eyeball.

[0067] The upper limb rehabilitation training method provided in this embodiment of the invention performs grayscale processing on the region of interest image obtained by selecting an eye image to obtain a grayscale image, and determines the center position of the eyeball based on the gradient direction of each pixel in the grayscale image, thus accurately determining the center position of the eyeball and providing a basis for determining the planar position coordinates of the center position in the eye image.

[0068] Example 3

[0069] This invention also provides another upper limb rehabilitation training method; this method is implemented based on the method in the above embodiments; this method focuses on describing the specific implementation of determining the expected motion trajectory and expected motion speed of the virtual reality device based on preset guided actions, the user's arm length, and the planar position coordinates in multiple eye images.

[0070] Figure 3 A flowchart of another upper limb rehabilitation training method provided in an embodiment of the present invention is shown below. Figure 3 As shown, this upper limb rehabilitation training method may include the following steps:

[0071] Step S301: Determine the spatial trajectory coordinates of the robotic arm of the virtual reality device based on the preset guided actions.

[0072] Since the preset guided actions are pre-configured in the virtual reality device, the spatial trajectory coordinates of the robotic arm can be obtained directly.

[0073] Step S302: Determine the Y-direction coordinates corresponding to multiple eye images based on the spatial trajectory coordinates and the user's arm length.

[0074] For example, the Y-axis coordinate can be calculated using the following formula:

[0075] (x-x0) 2 +(y-y0) 2 +(z-z0) 2 =R 2

[0076] Where x, y, and z are the X-direction coordinates, Y-direction coordinates, and Z-direction coordinates of the robotic arm's spatial trajectory, respectively, and x0 and z0 are the X-direction coordinates and Z-direction coordinates of the planar position coordinates.

[0077] The value of y0, i.e. the Y-direction coordinate corresponding to the eye image, can be derived from the above formula. The Y-direction coordinate corresponding to each eye image is calculated using the above formula, and the parameters in the above formula correspond to each other.

[0078] Step S303: Based on the Y-direction coordinates and the planar position coordinates in multiple eye images, determine the expected motion trajectory and expected motion speed of the robotic arm of the virtual reality device.

[0079] By combining the Y-axis coordinates corresponding to multiple eye images with the planar position coordinates in the multiple eye images, the spatial motion trajectory in the eye images can be obtained, that is, the spatial motion trajectory of the eyeball. Since the spatial motion trajectory of the eyeball is derived from the spatial trajectory coordinates of the robotic arm, the spatial motion coordinates of each robotic arm correspond to one of the coordinates on the spatial motion trajectory of the eyeball. Therefore, the expected motion trajectory of the virtual reality device can be determined by the spatial motion trajectory of the eyeball, and the expected motion speed can be determined based on the expected motion trajectory and the time interval between multiple consecutive eye images.

[0080] The upper limb rehabilitation training method provided in this invention determines the Y-direction coordinates corresponding to multiple eye images by using spatial trajectory coordinates and the user's arm length. Then, using the Y-direction coordinates and the planar position coordinates in the multiple eye images, the expected motion trajectory and expected motion speed of the robotic arm of the virtual reality device are determined. This achieves the goal of controlling the virtual reality device with the eyes and overcomes the problem that patients in the flaccid paralysis stage cannot control the virtual reality device.

[0081] Example 4

[0082] This invention also provides another upper limb rehabilitation training method; this method is implemented based on the method in the above embodiments; this method focuses on describing the specific implementation of determining the expected motion trajectory and expected motion speed of the robotic arm of the virtual reality device based on the Y-direction coordinate and the planar position coordinates in multiple eye images.

[0083] Figure 4 A flowchart of another upper limb rehabilitation training method provided in an embodiment of the present invention is shown below. Figure 4 As shown, this upper limb rehabilitation training method may include the following steps:

[0084] Step S401: Based on the Y-direction coordinates and the planar position coordinates in multiple eye images, determine the original motion trajectory of the robotic arm of the virtual reality device.

[0085] The original motion trajectory is the motion trajectory of the robotic arm determined by the spatial motion trajectory of the eyeball, and there are coordinates with large fluctuations in the original motion trajectory.

[0086] Step S402: Filter the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device.

[0087] In practical applications, users blink, and their eyeballs may twitch due to various reasons, such as eye diseases or eye strain from prolonged screen time. These fluctuations can cause significant variations in the original motion trajectory, leading to movements of the virtual reality device that are irrelevant to the user's intentions. Therefore, filtering can be performed on the original motion trajectory to remove coordinates with large fluctuations, thus obtaining the expected motion trajectory of the virtual reality device's robotic arm and eliminating the impact of blinking and eyeball twitching on the device's movement.

[0088] Specifically, filtering the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device may include: determining the coordinates of the first and second original motion trajectories adjacent to the original motion trajectory; comparing the first and second original motion trajectory coordinates to determine whether the second original motion trajectory coordinates are reasonable; if they are not reasonable, filtering the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device.

[0089] In this process, it can be determined whether the fluctuation of the second original motion trajectory coordinates relative to the first original motion trajectory coordinates is large. If it is large, it is unreasonable, and the unreasonable second original motion trajectory coordinates are filtered.

[0090] Among them, a fluctuation range threshold can be preset. If the fluctuation of the second original motion trajectory coordinates relative to the first original motion trajectory coordinates exceeds the fluctuation range threshold, it is determined that the fluctuation of the second original motion trajectory coordinates is large.

[0091] Step S403: Determine the expected motion speed based on the expected motion trajectory.

[0092] Specifically, determining the expected motion speed based on the expected motion trajectory may include: determining the distance between any two adjacent expected motion trajectory coordinates; determining the expected motion sub-velocities corresponding to any two adjacent expected motion trajectory coordinates based on multiple distances; and determining the expected motion speed based on multiple expected motion sub-velocities.

[0093] Further, the interval time between any two adjacent expected motion trajectory coordinates is determined. It should be noted that this interval time is the same as the time interval between two consecutive eye images. The expected motion sub-velocities between any two adjacent expected motion trajectory coordinates are determined by the interval time and distance. The expected motion sub-velocities are the velocities at the location of one of the expected motion trajectory coordinates on the expected motion trajectory. In this way, the expected motion speed can be determined by multiple expected motion sub-velocities.

[0094] The upper limb rehabilitation training method provided in this embodiment of the invention obtains the expected movement trajectory by filtering the original movement trajectory, and then determines the expected movement speed based on the expected movement trajectory, thus ensuring the accuracy of the expected movement trajectory and the expected movement speed and improving the accuracy of using eye-controlled virtual reality devices for movement.

[0095] Example 5

[0096] Corresponding to the above method embodiments, this invention provides an upper limb rehabilitation training device. Figure 5 This is a schematic diagram of the structure of an upper limb rehabilitation training device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the upper limb rehabilitation training device may include:

[0097] The eye image acquisition module 501 is used to acquire multiple consecutive eye images generated by the user's eyes based on preset guided actions configured by the virtual reality device.

[0098] The center position determination module 502 is used to determine the center position of the eyeball included in the eye image based on the eye image.

[0099] The planar position coordinate determination module 503 is used to determine the planar position coordinates of the center position in the eye image.

[0100] The trajectory and speed determination module 504 is used to determine the expected motion trajectory and expected motion speed of the virtual reality device based on preset guided actions, the user's arm length, and planar position coordinates in multiple eye images.

[0101] The motion control module 505 is used to control the movement of the virtual reality device based on the expected motion trajectory and expected motion speed, so that the user can use the virtual reality device for upper limb rehabilitation training.

[0102] The upper limb rehabilitation training device provided in this invention acquires multiple consecutive eye images generated by the user's eyes based on preset guided actions configured in the virtual reality device. It determines the planar position coordinates of the center position of the eyeball in the eye images. Then, based on the preset guided actions, the user's arm length, and the planar position coordinates in the multiple eye images, it determines the expected movement trajectory and expected movement speed of the virtual reality device. This achieves the goal of controlling the virtual reality device with the eyes. By controlling the movement of the virtual reality device based on the expected movement trajectory and expected movement speed, the device allows the user to use it for upper limb rehabilitation training, overcoming the problem that patients in the flaccid paralysis stage cannot control the virtual reality device.

[0103] In some embodiments, the center position determination module is further configured to select a region of interest (ROI) in the eye image to obtain a region of interest image; perform grayscale processing on the ROI image to obtain a grayscale image; determine the gradient direction of multiple pixels included in the grayscale image; and take the position of the point where the lines intersect most frequently in the gradient direction as the center position of the eyeball.

[0104] In some embodiments, the trajectory and speed determination module is further configured to determine the spatial trajectory coordinates of the robotic arm of the virtual reality device based on a preset guided action; determine the Y-direction coordinates corresponding to multiple eye images based on the spatial trajectory coordinates and the user's arm length; and determine the expected motion trajectory and expected motion speed of the robotic arm of the virtual reality device based on the Y-direction coordinates and the planar position coordinates in the multiple eye images.

[0105] In some embodiments, the trajectory and velocity determination module is further configured to determine the original motion trajectory of the robotic arm of the virtual reality device based on the Y-direction coordinate and the planar position coordinates in multiple eye images; to filter the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device; and to determine the expected motion velocity based on the expected motion trajectory.

[0106] In some embodiments, the trajectory and speed determination module is further configured to determine the coordinates of a first original motion trajectory and a second original motion trajectory adjacent to the original motion trajectory; compare the first original motion trajectory coordinates and the second original motion trajectory coordinates to determine whether the second original motion trajectory coordinates are reasonable; if they are not reasonable, filter the original motion trajectory to obtain the expected motion trajectory of the robotic arm of the virtual reality device.

[0107] In some embodiments, the trajectory and velocity determination module is further configured to determine the distance between any two adjacent expected motion trajectory coordinates in the expected motion trajectory; determine the expected motion sub-velocity corresponding to any two adjacent expected motion trajectory coordinates based on multiple distances; and determine the expected motion velocity based on multiple expected motion sub-velocities.

[0108] In some embodiments, the motion control module is further configured to perform kinematic analysis on the expected motion trajectory based on the lengths of the first robotic arm and the second robotic arm to obtain expected motion parameters; and control the first robotic arm and the second robotic arm to move at the expected motion speed based on the expected motion parameters.

[0109] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0110] Example 6

[0111] This invention also provides an electronic device for running the above-described upper limb rehabilitation training method; see [link to previous document]. Figure 6 The diagram shows the structure of an electronic device, which includes a memory 600 and a processor 601. The memory 600 is used to store one or more computer instructions, which are executed by the processor 601 to implement the above-mentioned upper limb rehabilitation training method.

[0112] Furthermore, Figure 6 The electronic device shown also includes a bus 602 and a communication interface 603, with the processor 601, communication interface 603 and memory 600 connected via the bus 602.

[0113] The memory 600 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 603 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 602 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0114] Processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 601 or by instructions in software form. Processor 601 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 600. Processor 601 reads information from memory 600 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0115] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described upper limb rehabilitation training method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0116] The computer program product for upper limb rehabilitation training provided in this embodiment of the invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0117] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0118] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An upper limb rehabilitation training device, characterized in that, The method comprises the following steps: An eye image acquisition module is configured to acquire a plurality of continuous eye images of a user's eyes generated based on a preset guide action of a virtual reality device; A center position determination module is configured to determine a center position of an eyeball included in the eye images based on the eye images; A plane position coordinate determination module is configured to determine a plane position coordinate of the center position in the eye images; A trajectory and speed determination module is configured to determine an expected motion trajectory and an expected motion speed of the virtual reality device based on the preset guide action, a length of an arm of the user, and the plane position coordinates in the plurality of eye images; A motion control module is configured to control the motion of the virtual reality device based on the expected motion trajectory and the expected motion speed, so that the user performs upper limb rehabilitation training by using the virtual reality device. The trajectory and speed determination module determines the expected motion trajectory and the expected motion speed of the virtual reality device based on the preset guide action, the length of the arm of the user, and the plane position coordinates in the plurality of eye images, specifically as follows: Taking the arm of the user as the Y direction, the length of the arm of the user as the radius, and the range of the ball corresponding to the radius as the maximum range of the user's swing, each Y direction coordinate is inversely solved according to the plane motion trajectory, the length of the arm of the user, and each spatial trajectory coordinate corresponding to the preset guide action, each Y direction coordinate is combined with the plane motion trajectory to obtain a spatial motion trajectory of the eyeball, the expected motion trajectory of the virtual reality device is determined according to the spatial motion trajectory of the eyeball, and the expected motion speed is determined according to the time interval between the expected motion trajectory and the eye images.

2. The apparatus of claim 1, wherein, The specific execution steps of the upper limb rehabilitation training device include: An eye image acquisition module is configured to acquire a plurality of continuous eye images of a user's eyes generated based on a preset guide action of a virtual reality device; A center position determination module is configured to determine a center position of an eyeball included in the eye images based on the eye images; A plane position coordinate determination module is configured to determine a plane position coordinate of the center position in the eye images; A trajectory and speed determination module is configured to determine an expected motion trajectory and an expected motion speed of the virtual reality device based on the preset guide action, a length of an arm of the user, and the plane position coordinates in the plurality of eye images; A motion control module is configured to control the motion of the virtual reality device based on the expected motion trajectory and the expected motion speed, so that the user performs upper limb rehabilitation training by using the virtual reality device.

3. The apparatus of claim 2, wherein, The center position determination module determines the center position of the eyeball included in the eye images based on the eye images, specifically as follows: The eye image is framed to obtain a region of interest image in the eye image; The region of interest image is subjected to grayscale processing to obtain a grayscale image; The gradient directions of a plurality of pixel points included in the grayscale image are determined; The position of a point with the most straight line intersections in the gradient direction is taken as the center position of the eyeball.

4. The apparatus of claim 2, wherein, The trajectory and speed determination module determines the expected motion trajectory and the expected motion speed of the virtual reality device based on the preset guide action, the length of the arm of the user, and the plane position coordinates in the plurality of eye images, specifically as follows: The spatial trajectory coordinates of a mechanical arm of the virtual reality device are determined based on the preset guide action; determining Y direction coordinates corresponding to the eye images respectively based on the spatial trajectory coordinates and the arm length of the user; determining an expected motion trajectory and an expected motion speed of a mechanical arm of the virtual reality device based on the Y direction coordinates and plane position coordinates in the eye images.

5. The apparatus of claim 4, wherein, The method of determining the expected motion trajectory and the expected motion speed of the mechanical arm of the virtual reality device based on the Y direction coordinates and the plane position coordinates in the eye images comprises: determining an original motion trajectory of the mechanical arm of the virtual reality device based on the Y direction coordinates and the plane position coordinates in the eye images; performing filtering processing on the original motion trajectory to obtain the expected motion trajectory of the mechanical arm of the virtual reality device; determining the expected motion speed based on the expected motion trajectory.

6. The apparatus of claim 5, wherein, The method of performing filtering processing on the original motion trajectory to obtain the expected motion trajectory of the mechanical arm of the virtual reality device comprises: determining first and second adjacent original motion trajectory coordinates of the original motion trajectory; comparing the first and second original motion trajectory coordinates to determine whether the second original motion trajectory coordinate is reasonable; if not, performing filtering processing on the original motion trajectory to obtain the expected motion trajectory of the mechanical arm of the virtual reality device.

7. The apparatus of claim 5, wherein, The method of determining the expected motion speed based on the expected motion trajectory comprises: determining distances between any two adjacent expected motion trajectory coordinates in the expected motion trajectory; determining expected motion sub speeds corresponding to any two adjacent expected motion trajectory coordinates based on the distances; determining the expected motion speed based on the expected motion sub speeds.

8. The apparatus of claim 2, wherein, The virtual reality device comprises a first mechanical arm and a second mechanical arm, and the method of controlling the virtual reality device to move based on the expected motion trajectory and the expected motion speed comprises: performing kinematic analysis on the expected motion trajectory based on the length of the first mechanical arm and the length of the second mechanical arm to obtain expected motion parameters; controlling the first mechanical arm and the second mechanical arm to move at the expected motion speed based on the expected motion parameters.

9. An electronic device, comprising: The device comprises a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the specific execution steps of the upper limb rehabilitation training device according to any one of claims 2 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to implement the specific execution steps of the upper limb rehabilitation training device according to any one of claims 2 to 7.

Images