An upper limb home rehabilitation system based on Sphero spherical robot

Abstract

The invention discloses an upper limb home rehabilitation system based on a Sphero spherical robot. The hardware system comprises the Sphero spherical robot, a terminal computer, a camera with a bracket and scene coordinate paper, and the software system comprises a Visual Studio development module, an OpenCV machine vision library module, a GUI interface input module and a Sphero robot identification and tracking module. The invention realizes rehabilitation training for patients based on the interaction between the Sphero spherical robot and the patients, controls the movement of the Sphero spherical robot through the software system, and the patients perform rehabilitation training for the upper limb muscles in a way of chasing and grasping with their hands. The camera with a bracket cooperates with the software system to realize the recognition and tracking of the Sphero spherical robot and the recognition of the successful grasping of the patients, thereby increasing the interest of the rehabilitation training, and realizes the selection of training with different difficulties through the reasonably designed scene coordinate paper, thereby increasing the effect of the rehabilitation training.

Description

Technical Field

[0001] The present invention relates to the technical field of upper limb rehabilitation training, and in particular to an upper limb home rehabilitation system based on a Sphero spherical robot. Background Art

[0002] Stroke is the leading cause of death among Chinese residents. According to surveys, about 55%-75% of stroke patients will experience hemiplegia. Limb injuries caused by hemiplegia, especially upper limb injuries, will make it difficult for patients to carry out daily activities, seriously affecting their daily lives. With the improvement of people's living standards in recent years, the proportion of obese and diabetic people has increased, the incidence of stroke has also increased year by year, and the average age of onset has gradually decreased, which seriously threatens the health of Chinese residents.

[0003] At present, rehabilitation treatment for stroke patients is mostly performed by rehabilitation trainers in hospitals. Since stroke patients have difficulty in moving and reduced motor ability, it is very inconvenient for them to travel to and from the hospital. In addition, my country has a large population base and there is a serious shortage of professional rehabilitation trainers. The accuracy and standardization of artificial rehabilitation are difficult to guarantee. Therefore, the present invention proposes an upper limb home rehabilitation system based on a Sphero spherical robot to solve the problems existing in the prior art. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to propose an upper limb home rehabilitation system based on a Sphero spherical robot. The upper limb home rehabilitation system based on the Sphero spherical robot realizes rehabilitation training for patients based on the interaction between the Sphero spherical robot and the patient. The movement of the robot is controlled by a software system, and the patient performs rehabilitation training of the upper limb muscles in a chasing and grasping manner with his hands, which increases the fun of the training. The system also realizes training options of different difficulty levels through reasonably designed scene coordinate paper, thereby increasing the effect of the rehabilitation training. At the same time, a camera with a bracket is used in conjunction with the software system to realize visual tracking, recognition and judgment of training results. The system is affordable and easy to install and operate, which greatly increases the universality of the system.

[0005] To achieve the purpose of the present invention, the present invention is implemented through the following technical solutions: a home rehabilitation system for upper limbs based on a Sphero spherical robot, including a hardware system and a software system, the hardware system includes a Sphero spherical robot, a terminal computer, a camera with a bracket and a scene coordinate paper, the Sphero spherical robot is connected to the terminal computer through a built-in Bluetooth communication module, the camera with a bracket is connected to the terminal computer through a USB port, the camera with a bracket is placed on one side of the scene coordinate paper and the camera is vertically illuminated to the scene coordinate paper, the Sphero spherical robot is placed on the scene coordinate paper, and the software system includes a Visual Studio development module, an OpenCV machine vision library module, a GUI interface input module and a Sphero robot recognition and tracking module.

[0006] Further improvements are: the OpenCV machine vision library module, GUI interface input module and Sphero robot recognition and tracking module are implemented based on the Visual Studio development module, the OpenCV machine vision library module provides recognition visual data for the Sphero spherical robot, the GUI interface input module inputs patient information and selects training mode and query of training data, and the Sphero robot recognition and tracking module uses Hough circle detection to realize robot recognition and tracking.

[0007] Further improvements are: the Sphero spherical robot has a built-in Bluetooth communication module, a three-axis speed gyroscope module, a three-axis accelerometer module and a power module. The Sphero spherical robot also includes an open API platform for controlling the power, motion trajectory, RGBLED light color value and data exchange of the left-drive and right-drive wheels of the Sphero spherical robot.

[0008] A further improvement is that the camera with a bracket is a wide-angle camera and faces downward to take images, the height of the camera with a bracket is adjusted so that the scene coordinate paper covers the entire imaging range of the camera with a bracket, and the scene coordinate paper is set according to the average height of adult males and adult females in my country.

[0009] Further improvements are: the GUI interface input module includes a main interface and a training data interface, the main interface is used to input the patient's ID number, then select Bluetooth connection to control the Sphero spherical robot, and connect the camera with a bracket, and finally select the training difficulty and number of trainings to start training; click the query button on the main interface to enter the training data interface, and query all the user's training data within three months in the training data interface.

[0010] A further improvement is that the Hough circle detection principle is to achieve recognition and tracking by obtaining the position of the circle in the Sphero spherical robot image, which is specifically divided into two steps. The first step is to detect the center of the circle, and the second step is to estimate the radius.

[0011] Further improvement is that: the first step of detecting the center of the circle is to obtain the video stream through a camera with a bracket, then read the video and save it by frame, and then perform binarization and median filtering on the image. The processed image is subjected to canny edge detection, and the sobel operator is used to calculate the gradient of the figure in the x and y directions. Then, the circular space N (a, b) is initialized, and the gradient straight lines of all figures are drawn in the two-dimensional Hough space and accumulated. Finally, the possible center of the circle is obtained after statistical sorting N (a, b).

[0012] Further improvement is: in the second step of estimating the radius, first calculate the distance of all the circumferences of one of the circular cutters, then set two thresholds according to the size of the circle to be detected, define the radius range of the circle, then sort the retained distances, initialize the radius space R, traverse all non-zero points in the edge graph, for all radii r that satisfy N(r)+=1, and finally obtain the estimated radius value.

[0013] The beneficial effects of the present invention are as follows: the present invention realizes rehabilitation training for patients based on the interaction between the Sphero spherical robot and the patients, controls its movement through a software system, and the patients perform rehabilitation training for the upper limb muscles in a manner of chasing and grasping with their hands, uses a camera with a bracket in conjunction with the software system to realize recognition and tracking of the Sphero spherical robot and recognition of successful grasping by the patients, thereby increasing the fun of rehabilitation training, and realizes training selection of different difficulty levels through reasonably designed scene coordinate paper, thereby increasing the effect of rehabilitation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram of the hardware system connection architecture of Example 1 of the present invention.

[0015] Figure 2 This is a flowchart of the software system operation of Example 1 of the present invention.

[0016] Figure 3 This is a flow chart of detecting the center of a circle according to Embodiment 1 of the present invention.

[0017] Figure 4 This is a flow chart of radius estimation in Example 1 of the present invention.

[0018] Figure 5 This is a schematic diagram of point selection on the coordinate paper of the hardware test scenario in Example 2 of the present invention. DETAILED DESCRIPTION

[0019] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0020] Example 1

[0021] according to Figure 1 , 2 As shown in Figures 3 and 4, this embodiment provides an upper limb home rehabilitation system based on a Sphero spherical robot, including a hardware system and a software system. The hardware system includes a Sphero spherical robot, a terminal computer, a camera with a bracket and a scene coordinate paper. The Sphero spherical robot is connected to the terminal computer via a built-in Bluetooth communication module, and the camera with a bracket is connected to the terminal computer via a USB port. The camera with a bracket is placed on one side of the scene coordinate paper and the camera is vertically illuminated to the scene coordinate paper. The Sphero spherical robot is placed on the scene coordinate paper. The terminal computer controls the random movement of the Sphero spherical robot via the Bluetooth communication module. The patient grabs the moving robot within a specified time, and then uses the camera with a bracket to identify and determine whether the training is completed.

[0022] The software system includes Visual Studio development module, OpenCV machine vision library module, GUI interface input module and Sphero robot recognition and tracking module. The OpenCV machine vision library module, GUI interface input module and Sphero robot recognition and tracking module are implemented based on the Visual Studio development module. The OpenCV machine vision library module provides recognition visual data for the Sphero spherical robot. The GUI interface input module inputs patient information and selects training mode and query of training data. The Sphero robot recognition and tracking module uses Hough circle detection to realize robot recognition and tracking. The user first connects to the Sphero spherical robot through the Bluetooth communication module, and then enters the user ID in the GUI interface input module, selects the difficulty and number of training according to needs, and then starts training. At this time, the camera starts to acquire and process images, and stores data in real time until the training is completed.

[0023] The Sphero spherical robot has built-in Bluetooth communication module, three-axis speed gyroscope module, three-axis accelerometer module and power module. The Sphero spherical robot also includes an open API platform for controlling the power, motion trajectory, RGBLED light color value and data exchange of the left and right drive wheels of the Sphero spherical robot.

[0024] The camera with a bracket is a wide-angle camera and faces downward to take images. The height of the camera with a bracket is adjusted so that the scene coordinate paper covers the entire imaging range of the camera with a bracket. The scene coordinate paper is set according to the average height of adult males and adult females in my country.

[0025] The coordinate system length of the scene coordinate paper is set to 60cm, the width is set to 50cm, the coordinate system is fan-shaped, and the fan-shaped marking line is drawn with a radius difference of 2cm. Different line colors are used at the radius of 20cm and 40cm to divide the coordinate system into three areas: simple mode, general mode and difficult mode. During training, the Sphero spherical robot moves in the corresponding area according to the selected difficulty.

[0026] The operating environment of the software system is the Windows 10 operating system. The GUI interface input module includes the main interface and the training data interface. The main interface is used to enter the patient's ID number, then select Bluetooth connection to control the Sphero spherical robot, and connect the camera with a bracket. Finally, select the training difficulty and number of trainings to start training; click the query button on the main interface to enter the training data interface, and query all the user's training data within three months in the training data interface.

[0027] The Hough circle detection principle is to achieve recognition and tracking by obtaining the position of the circle in the Sphero spherical robot image. It is divided into two steps. The first step is to detect the center of the circle, and the second step is to estimate the radius.

[0028] The first step is to detect the center of the circle. First, obtain the video stream through the camera with a bracket, then read the video and save it frame by frame, and then perform binarization and median filtering on the image. The processed image is subjected to Canny edge detection, and the Sobel operator is used to calculate the gradient of the figure in the x and y directions. Then, the circular space N(a, b) is initialized, and the gradient straight lines of all figures are drawn in the two-dimensional Hough space and accumulated. Finally, the possible center of the circle is obtained after statistical sorting N(a, b).

[0029] The second step is to estimate the radius. First, calculate the distance of all the circumferences of one of the circular cutters, then set two thresholds according to the size of the circle to be detected, define the radius range of the circle, and then sort the retained distances, initialize the radius space R, traverse all non-zero points in the edge graph, and for all radii r that satisfy N(r)+=1, finally get the possible radius value.

[0030] Example 2

[0031] according to Figure 5 As shown, this embodiment provides an upper limb home rehabilitation system based on the Sphero spherical robot, including hardware testing and software testing. The hardware test is to test the recognition rate of the Sphero spherical robot under different light conditions, specifically including

[0032] 1) Place the hardware system and run the software system in a well-lit environment, select 5 test points on the scene coordinate paper, and conduct 10 experiments on each test point;

[0033] 2) Start the software and connect the camera with the bracket to the Sphero spherical robot;

[0034] 3) Start the test, place the Sphero spherical robot at test point 1, and start timing. When the camera with the bracket recognizes the Sphero spherical robot and circles the robot with a red circle, stop timing, turn off the system, move the Sphero spherical robot to the next test point, and repeat the above steps until the test is completed;

[0035] 4) Repeat the above experimental steps 1-3 in a dim light environment to perform a low-light environment recognition test. The test results are shown in Table 1 below.

[0036] Table 1 Sphero spherical robot recognition test results

[0037]

[0038] Hardware testing also includes testing the communication quality of the Bluetooth communication module at different distances with or without obstructions, including

[0039] 1) Place the hardware system and run the software in an unobstructed environment;

[0040] 2) Start the system, place the Sphero spherical robot 1m away from the terminal computer, click the Bluetooth connection button, and observe the Bluetooth connection. If the Sphero spherical robot lights up blue light, it is successfully connected. Disconnect the Bluetooth connection and place the Sphero spherical robot 5m away from the device to observe the Bluetooth connection. If it is successfully connected, disconnect the Bluetooth connection and place the Sphero spherical robot 10m away from the device to observe the Bluetooth connection. Repeat the above experiment 10 times and record the data.

[0041] 3) Place a 5 cm thick template between the terminal computer and the Sphero spherical robot, and repeat the above steps 1-2 to test the Bluetooth connection quality with obstructions. The test results are shown in Table 2 below.

[0042] Table 2 Bluetooth connection test results

[0043] distance Number of connections without obstructions / times Connection times with obstructions 1m 10 10 5m 10 10 10m 10 9

[0044] Software testing includes Bluetooth connection button testing, camera connection button testing, and start button testing. The Bluetooth connection button testing specifically includes:

[0045] 1) Build the hardware system and run the software first;

[0046] 2) Start the system, click the Bluetooth connection button, and wait for the Bluetooth to connect to the Sphero spherical robot. If the Sphero spherical robot lights up blue, the connection is successful. Click the Bluetooth disconnect button to disconnect the Bluetooth. After the Sphero spherical robot light goes out, wait for 5 seconds and click the Bluetooth connection button again. Repeat the experiment 10 times according to the above process and record the data.

[0047] The camera connection button test with bracket includes

[0048] 1) Build the hardware system and run the software first;

[0049] 2) Start the system, click the camera connection button with a bracket, and wait for the camera with a bracket to connect to the terminal computer. If the camera with a bracket displays a real-time image on the camera screen display, the connection is successful. Click the camera disconnect button with a bracket to disconnect the camera. After the real-time image on the camera screen is disconnected and returns to the initial blank screen, wait for 5S, click the camera connection button with a bracket again, and repeat the experiment 10 times according to the above process and record the data.

[0050] The start button test includes

[0051] 1) Build the hardware system and run the software first;

[0052] 2) Start the system, select the training difficulty and number of times, press the Start Training button, and enter the training interface. If you successfully jump to the training interface and the timer counts normally, the connection is successful. Click the Exit Training button, if you jump to the main interface, the exit is successful. After jumping back to the main interface, wait for 5S, click the Start Training button again, repeat the experiment 10 times according to the above process, and record the data.

[0053] When the upper limb home rehabilitation system based on the Sphero spherical robot is used and "Start Training" is clicked, the machine vision library such as OpenCV is used to correctly identify the human hand that enters the camera field of view with a bracket. When the Sphero spherical robot is completely blocked by the human hand, it is judged that the patient has grasped the Sphero spherical robot; if the grasping is successful, it is recorded as a training session, and the system waits for 3 seconds before continuing the next training. Under natural conditions, the Sphero spherical robot will enter a dormant state if no command is sent within 10 seconds.

[0054] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A home rehabilitation system for upper limbs based on a Sphero spherical robot, characterized by: The system comprises a hardware system and a software system. The hardware system comprises a Sphero spherical robot, a terminal computer, a camera with a bracket and a scene coordinate paper. The Sphero spherical robot is connected to the terminal computer through a built-in Bluetooth communication module. The camera with a bracket is connected to the terminal computer through a USB port. The camera with a bracket is placed on one side of the scene coordinate paper and the camera is vertically illuminated to the scene coordinate paper. The Sphero spherical robot is placed on the scene coordinate paper. The terminal computer controls the random movement of the Sphero spherical robot through the Bluetooth communication module. The patient grabs the moving robot within a specified time, and then uses the camera with a bracket to identify and determine whether the training is completed. The software system comprises a Visual Studio development module, an OpenCV machine vision library module, a GUI interface input module and a Sphero robot identification and tracking module.

2. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 1, characterized in that: The OpenCV machine vision library module, GUI interface input module and Sphero robot recognition and tracking module are implemented based on the Visual Studio development module. The OpenCV machine vision library module provides recognition visual data for the Sphero spherical robot. The GUI interface input module inputs patient information and selects training mode and query of training data. The Sphero robot recognition and tracking module uses Hough circle detection to realize robot recognition and tracking.

3. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 1, characterized in that: The Sphero spherical robot has a built-in Bluetooth communication module, a three-axis speed gyroscope module, a three-axis accelerometer module and a power module. The Sphero spherical robot also includes an open API platform for controlling the power, motion trajectory, RGBLED light color value and data exchange of the left-drive and right-drive wheels of the Sphero spherical robot.

4. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 1, characterized in that: The camera with the bracket is a wide-angle camera and faces downward to take images. The height of the camera with the bracket is adjusted so that the scene coordinate paper covers the entire imaging range of the camera with the bracket. The scene coordinate paper is set according to the average height of adult males and adult females in my country.

5. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 1, characterized in that: The GUI interface input module includes a main interface and a training data interface. The main interface is used to input the patient's ID number, then select Bluetooth connection to control the Sphero spherical robot, and connect the camera with a bracket, and finally select the training difficulty and number of trainings to start training; click the query button on the main interface to enter the training data interface, and query all the user's training data within three months in the training data interface.

6. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 2, characterized in that: The Hough circle detection principle is to achieve recognition and tracking by obtaining the position of the circle in the Sphero spherical robot image. It is specifically divided into two steps. The first step is to detect the center of the circle, and the second step is to estimate the radius.

7. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 6, characterized in that: The first step is to detect the center of the circle. First, a video stream is obtained through a camera with a bracket, and then the video is read and saved frame by frame. The image is then binarized and median filtered. The processed image is subjected to canny edge detection, and the sobel operator is used to calculate the gradient of the figure in the x and y directions. Then, the circular space N (a, b) is initialized, and the gradient straight lines of all figures are drawn in the two-dimensional Hough space and accumulated. Finally, the possible center of the circle is obtained after statistical sorting N (a, b).

8. The upper limb home rehabilitation system based on the Sphero spherical robot according to claim 6, characterized in that: The second step estimates the radius by first calculating the distances of all the circumferences of one of the circular cutters, then setting two thresholds according to the size of the circle to be detected, defining the radius range of the circle, and then sorting the retained distances, initializing the radius space R, traversing all non-zero points in the edge graph, and for all radii r that satisfy N(r)+=1, finally obtaining the estimated radius value.

Images