Intelligent artificial limb control system and control method for visual impairment amputation patient
Through the voice interaction and multi-sensory feedback of the intelligent prosthetic control system, the problem of perceived and grasping tasks in daily life of patients with visually impaired amputation is solved, and environmental perception and object positioning of patients with visually impaired amputation is realized, providing suitable intelligent prosthesis Control plan.
Patent Information
- Application Number
- CN202411063462.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing prosthetic control system cannot be used for patients with visually impaired amputations who lose both vision and grasping ability, and it is difficult to complete daily life activities without help.
An intelligent prosthetic control system was designed, combining voice interaction, environmental perception, grab guide, prosthetic coordinated control, auditory feedback and tactile feedback to perform environmental perception and object positioning through global cameras and local cameras, using vibrating feedback arm rings and independent The sound source module provides multi-threshold, multi-stage navigation and grab guide.
It realizes the perception, positioning and grasping tasks of visually impaired amputation patients in the indoor environment, and provides a set of practically available intelligent prosthetic control systems to meet basic living needs. It is suitable for environmental perception, mobile navigation and objects of visually impaired amputation patients. position.
Smart Images

Figure CN118975876A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electromechanical control technology, and in particular to an intelligent prosthesis control system and a control method thereof for visually impaired amputees. Background Art
[0002] Human grasping skills are developed through interactive learning and depend critically on vision and touch. Visually impaired amputees lose their visual and partial tactile abilities to perceive their surroundings, making it a daunting challenge for them to manipulate their prosthetic hands to achieve reaching and grasping tasks. Existing prosthetic control methods rely heavily on the visual judgment of amputees, requiring patients to rely on the human visual environment perception and system supervision to approach and aim at objects, which is extremely difficult for visually impaired amputees.
[0003] Prosthetic control methods based on electromyographic signals, such as finite state machines, pattern recognition, synchronous control, and deep learning, require patients to align their prostheses with objects before executing corresponding myoelectric control. New shared control methods based on multi-sensor fusion can achieve a certain degree of prosthetic control automation through environmental context information. However, the above-mentioned human-machine collaborative control methods are still based on the condition that patients actively complete environmental perception and object positioning through human vision, and cannot be applied to visually impaired amputees.
[0004] Guiding the blind is a prerequisite for visually impaired amputees to complete grasping tasks. The key to the guiding task is to help visually impaired amputees obtain information from the environment accurately and quickly to promote their perception and understanding of the environment. In addition to the high-risk and high-cost retinal prosthesis technology, the use of hearing or touch to provide sensory compensation is the main idea of guiding the blind. Such as pedestrian navigation systems based on voice interaction, auxiliary robots based on vibration touch, or mobile canes. Mobile navigation tasks tend to guide people to positions on a two-dimensional plane. For object positioning, it is necessary to find objects in three-dimensional space. Using wearable machine vision devices to help visually impaired patients locate and grasp the required objects in the environment through auditory feedback (voice prompts, "radar" prompts, mapping based on image to sound features, etc.) has been proven to be effective, and the use of continuous approach feedback has a better positioning effect. In the above technologies, the hands (touch) of visually impaired patients play a key role. For visually impaired individuals with amputations, the above methods will have great limitations.
[0005] Existing prosthetic control systems and methods are not suitable for visually impaired amputees who have lost both vision and gripping abilities. Currently, there is no complete prosthetic solution for visually impaired amputees, which makes it difficult for them to complete some basic daily activities without help. Summary of the invention
[0006] The present invention aims to solve the problem that the existing prosthetic control systems and control methods in the prior art cannot be applied to visually impaired amputees who have lost both vision and grasping ability. Currently, there is no complete prosthetic solution for visually impaired amputees, which makes it difficult for visually impaired amputees to complete some basic daily life activities without help.
[0007] To solve the above technical problems, the present invention is achieved through the following technical solutions:
[0008] Solution 1: The present invention proposes an intelligent prosthesis control system for visually impaired amputees, the intelligent prosthesis control system comprising a wireless headset, a head-mounted camera, a vibration feedback armband, an intelligent prosthesis, an independent sound source module, a prosthesis receiving cavity, a backpack, a main controller and a mobile power supply.
[0009] The wireless headset is worn on the user's ear, the head-mounted camera is worn on the user's head and connected to the main controller via a data cable, the vibration feedback armband is worn on the user's upper arm, and exchanges data with the main controller via wireless communication, the prosthetic receiving cavity is worn at the end of the user's residual limb, the intelligent prosthesis is fixed to the prosthetic receiving cavity, and the intelligent prosthesis and the main controller complete data exchange via wireless communication, the main controller and the mobile power supply are placed in the backpack, the mobile power supply provides power to the main controller, the independent sound source module is placed under the target object, and the independent sound source module and the main controller exchange data via wireless communication.
[0010] Furthermore, a preferred embodiment is provided, wherein the head-mounted camera comprises an elastic headband, an adjustment bracket and a global camera, the elastic headband is worn on the user's head, the adjustment bracket is fixed to the elastic headband, the global camera is fixed to the adjustment bracket, and the adjustment bracket is used to adjust the field of view angle of the global camera.
[0011] Furthermore, a preferred embodiment is provided, in which the vibration feedback armband comprises a main vibration module base, a vibration armband battery, a vibration armband driving board, a main vibration module upper cover, a vibration armband controller, a vibration armband switch, a vibration motor, a sub-vibration module and an elastic connector; the vibration armband battery, the vibration armband driving board and the vibration armband controller are fixed to the main vibration module base, the vibration armband battery provides power for the vibration armband driving board and the vibration armband controller, the vibration armband switch is fixed to the side of the main vibration module base, the vibration motor is fixed to the main vibration module base and the inner side of the sub-vibration module, so that the vibration motor is used to press onto the user's upper arm skin, and the main vibration module base and the sub-vibration module and the sub-vibration module are connected via the elastic connector.
[0012] Furthermore, a preferred embodiment is provided, wherein the number of the sub-vibration modules is five, the number of the vibration motors is twelve, each of the sub-vibration modules and the main vibration module base has two vibration motors, and the vibration motors are connected to the vibration armband drive board via a flexible flat cable.
[0013] Furthermore, a preferred embodiment is provided, in which the vibration feedback armband is worn on the upper arm of the user, and the method for performing data interaction with the main controller through wireless communication is as follows: the base of the main vibration module is located on the upper side of the upper arm, and the vibration armband controller receives the signal of the main controller, controls the vibration of the vibration motor, and generates vibration feedback information representing different directions by activating the vibration motors at different positions, and the vibrations of all the vibration motors are used to provide feedback information of successful grasping.
[0014] Furthermore, a preferred embodiment is provided, wherein the intelligent prosthesis comprises a prosthetic hand, a wrist rotation motor, a wrist flexion motor, an intelligent prosthesis controller, a wrist drive board, a prosthetic battery, a local camera and a wrist shell, wherein the wrist rotation motor and the wrist flexion motor are fixed to the wrist shell to drive the prosthetic hand to rotate and flex, the prosthesis controller is wirelessly connected to the main controller, and controls the movement of the prosthetic hand, the wrist rotation motor and the wrist flexion motor according to the signal of the main controller, the prosthetic battery provides power for the intelligent prosthesis, the prosthesis controller wrist drive board and the prosthetic battery are fixed to the inside of the wrist shell, and the local camera is fixed to the side of the wrist shell so that the field of view of the local camera can include the inner part of the palm of the prosthetic hand.
[0015] Furthermore, a preferred embodiment is provided, wherein the independent sound source module includes a sound source module cover, a sound source module speaker, a sound source module switch, a sound source module battery, a sound source module controller, a music playback module, a power management module, a sound source module base, a magnetic suction cup and a memory card; the sound source module controller receives wireless transmission information from the main controller, controls the music playback module to play or pause the music files in the memory card, and the sound source module battery provides power for the independent sound source module.
[0016] Furthermore, a preferred embodiment is provided, wherein the independent sound source module is in the shape of a disk or a small cuboid.
[0017] Furthermore, a preferred embodiment is provided, wherein the main controller obtains the category and position information of the object in the field of view by identifying and detecting the images obtained by the global camera and the local camera, generates voice prompts and vibration feedback information, and controls the wireless headset and the vibration feedback armband to generate voice prompts and vibration feedback according to the generated information, so as to guide the user to complete environmental perception, mobile navigation and grasping guidance tasks. When the main controller detects that the target object is within the grasping range of the prosthetic hand, the main controller controls the prosthetic hand to close to complete the grasping of the target object.
[0018] Solution 2: An intelligent prosthesis control method for visually impaired amputees, the intelligent prosthesis control method is implemented based on the intelligent prosthesis control system for visually impaired amputees described in any one of Solution 1, and the intelligent prosthesis system control method comprises:
[0019] S1. The user first carries the backpack on his back, wears the vibration feedback armband on his upper arm, wears the prosthetic receiving cavity on the end of the residual limb, then fixes the intelligent prosthesis to the prosthetic receiving cavity, wears the head-mounted camera on the head, and adjusts the adjustment bracket to ensure that the field of view of the global camera is facing the front of the user, connects the global camera and the main controller through a data cable, and finally wears the wireless headset on the user's ear, turns on the main controller, and runs the upper computer program, then correctly turns on the intelligent prosthesis, the vibration feedback armband and the wireless headset, and waits for the wireless connection to be established between the various parts. When the user receives the "system startup successful" prompt tone generated by the main controller through the wireless headset, it means that the system has started correctly;
[0020] S2, selecting different independent sound source modules according to different target object characteristics, and placing the independent sound source module and the target object together, turning on the independent sound source module through the sound source module switch, waiting for the independent sound source module to establish a wireless connection with the main controller, and after the correct connection, the independent sound source module will play a successful connection prompt tone;
[0021] S3, the user inputs the name of the target object to be grabbed to the main controller through a voice command, the main controller recognizes the user's voice command through the voice interaction module and wakes up the independent sound source module bound to the target object, the independent sound source module plays the music set in advance, the user judges the position of the target object according to the sound emitted by the independent sound source module, and keeps approaching the target object;
[0022] S4. When the distance between the user and the target object is greater than two meters, the main controller recognizes the image of the global camera through the environment perception module to obtain the object information in front of the user. The environment perception module in the main controller divides the field of view of the global camera into a nine-square grid, and generates a position description statement of the target object according to the position of the target object, and then generates a prompt sound through the speech synthesis function, which is played to the user through the wireless headset. The user continues to approach the object according to the sound and voice prompt emitted by the independent sound source module;
[0023] S5. When the distance between the user and the target object is between one meter and two meters, the main controller will feed back the position information of the target object to the user through the vibration feedback armband, transmit the direction information of the target object to the user by activating the vibration motors at different positions, and transmit the distance information of the target object to the user by changing the vibration intensity and frequency of the vibration motors. The user continues to approach the object according to the sound emitted by the independent sound source module and the vibration prompt generated by the vibration feedback armband. When the user is one meter away from the target object, the main controller sends a voice prompt to the user through the wireless headset that the user has reached the contact range.
[0024] S6. When the user receives a prompt sound indicating that the user has reached the contact range, the user lifts the intelligent prosthesis, and the main controller recognizes the image of the local camera through the grasping guidance module to obtain the positional relationship between the target object and the prosthetic hand, and generates a moving direction prompt sound based on the relative positional relationship to guide the user to move the prosthetic hand closer to the target object;
[0025] S7, when the distance between the artificial hand and the target object is greater than 20 centimeters, the grasping guidance module in the main controller uses a large threshold to calculate the moving direction prompt sound, and the main controller generates the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand left, right, up, down and forward;
[0026] S8, when the distance between the artificial hand and the target object is less than 20 centimeters, the grasping guidance module in the main controller uses a small threshold to calculate the moving direction prompt sound, and the main controller continues to calculate and generate the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand to continue to approach the target object. When the grasping guidance module detects that the target object is within the grasping range of the artificial hand, the main controller sends a prompt sound to the user through the wireless headset that the target object has reached the grasping range;
[0027] S9, the main controller sends a "reaching the grasping range" prompt sound to the user, and controls the movement of the artificial hand, the wrist rotation motor and the wrist flexion motor through the prosthesis coordination control module, so that the artificial hand can stably grasp the target object;
[0028] S10, when the object is successfully grasped, the main controller controls the vibration feedback armband to start all the vibration motors to convey a signal of successful grasping to the user in the form of vibration feedback, and at the same time, the main controller puts the independent sound source module bound to the target object into sleep mode through the wireless communication module;
[0029] S11. After grasping the object, the user performs corresponding operations. When the object needs to be released, the user issues a voice command to open the prosthetic hand. The wireless headset receives the user's voice command and sends it to the main controller. The main controller recognizes the command through the voice interaction module, and controls the prosthetic hand to open through the prosthetic collaborative control module to release the target object.
[0030] The present invention is beneficial in that:
[0031] The present invention provides an intelligent prosthesis system and a control method thereof for visually impaired amputees, which include voice interaction, environmental perception, grasping guidance, prosthesis collaborative control, auditory feedback and tactile feedback modules, and can help visually impaired amputees complete environmental perception, mobile navigation, object positioning and grasping tasks in indoor scenes.
[0032] The present invention aims at the problem that the existing prosthetic control systems and control methods in the prior art are not applicable to visually impaired amputees who have lost both vision and grasping ability. At present, there is no complete prosthetic solution for visually impaired amputees, which makes it difficult for visually impaired amputees to complete some basic daily life activities without help. An intelligent prosthetic control system and control method for visually impaired amputees is proposed, which combines voice interaction, environmental perception, grasping guidance, prosthetic collaborative control, auditory feedback and tactile feedback to achieve a multi-stage, multi-channel, multi-threshold mobile navigation and grasping guidance strategy.
[0033] The control system described in the present invention uses intelligent voice interaction, uses a global camera for environmental perception, uses a local camera to guide the user to aim at the object, uses a multi-finger prosthesis to grasp the object, and realizes auditory and tactile feedback through spatial sound sources, voice prompts and vibration feedback. It can help visually impaired amputees complete perception, positioning, navigation and grasping tasks in indoor environments. The modules and the modules and users work together to form a closed-loop human-computer interaction system to meet the basic living needs of visually impaired amputees. The system and method are natural and intuitive, conform to the daily behavior and activity habits of humans, and provide a practical and usable intelligent prosthetic control system solution for visually impaired amputees.
[0034] The present invention is also applicable to the field of environmental perception, mobile navigation, object positioning and grasping tasks for visually impaired amputees in indoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a schematic diagram of the overall composition of an intelligent prosthetic control system for visually impaired amputees as described in Implementation Example 1.
[0036] Figure 2 This is a schematic diagram of the head-mounted camera structure and global orientation division in an intelligent prosthetic control system for visually impaired amputees as described in Implementation Example 2.
[0037] Figure 3 This is a schematic diagram of the structure of a vibration feedback armband in an intelligent prosthetic control system for visually impaired amputees as described in Implementation Method 3.
[0038] Figure 4 This is a schematic diagram of a wearing method and feedback method of a vibration feedback armband in an intelligent prosthesis control system for visually impaired amputees as described in Implementation Method 3.
[0039] Figure 5 This is a schematic diagram of the structure of an intelligent prosthesis in an intelligent prosthesis control system for visually impaired amputees as described in Implementation Example 6.
[0040] Figure 6 This is a schematic diagram of the structure of an independent sound source module in an intelligent prosthesis control system for visually impaired amputees as described in Implementation Example 1.
[0041] Figure 7 This is a schematic diagram of the functional module composition and control flow of an intelligent prosthesis control system for visually impaired amputees described in Implementation Example 1.
[0042] Figure 8 This is a schematic diagram of the calculation rules for the moving direction prompt sound in an intelligent prosthesis control system for visually impaired amputees as described in the tenth embodiment.
[0043] Fig. 9 This is a schematic diagram of the overall task flow of an intelligent prosthetic control system for visually impaired amputees as described in Implementation Example 10.
[0044] In the figure, wireless headset 1, head-mounted camera 2, elastic hairband 21, adjustment bracket 22, global camera 23, vibration feedback armband 3, main vibration module base 31, vibration armband battery 32, vibration armband drive board 33, main vibration module cover 34, vibration armband controller 35, vibration armband switch 36, vibration motor 37, sub-vibration module 38, elastic connector 39, intelligent prosthesis 4, prosthetic hand 41, wrist rotation motor 42, abdominal flexion motor 43, intelligent prosthesis controller 44, wrist drive board 45, prosthesis battery 46, local camera 47, wrist shell 48, independent sound source module 5, sound source module cover 51, sound source module speaker 52, sound source module switch 53, sound source module battery 54, sound source module controller 55, music playback module 56, power management module 57, sound source module base 58, magnetic suction cup 59, memory card 60, prosthesis receiving cavity 6, backpack 7, main controller 8, mobile power supply 9. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solutions and advantages of the implementation methods of the present application clearer, the technical solutions in the implementation methods of the present application will be clearly and completely described below in conjunction with the drawings in the implementation methods of the present application. Obviously, the described implementation methods are only part of the implementation methods of the present application, not all of the implementation methods.
[0046] Embodiment 1: This embodiment provides an intelligent prosthesis control system for visually impaired amputees, the intelligent prosthesis control system comprising a wireless headset 1, a head-mounted camera 2, a vibration feedback armband 3, an intelligent prosthesis 4, an independent sound source module 5, a prosthesis receiving cavity 6, a backpack 7, a main controller 8 and a mobile power supply 9.
[0047] The wireless headset 1 is worn on the user's ear, the head-mounted camera 2 is worn on the user's head and connected to the main controller 8 via a data cable, the vibration feedback armband 3 is worn on the user's upper arm, and exchanges data with the main controller 8 via wireless communication, the prosthesis receiving cavity 6 is worn at the end of the user's residual limb, the intelligent prosthesis 4 is fixed to the prosthesis receiving cavity 6, and the intelligent prosthesis 4 and the main controller 8 complete data exchange via wireless communication, the main controller 8 and the mobile power supply 9 are placed in the backpack 7, the mobile power supply 9 provides power for the main controller 8, the independent sound source module 5 is placed under the target object, and the independent sound source module 5 and the main controller 8 exchange data via wireless communication.
[0048] See also Figure 1 This embodiment describes the control system described in this embodiment. Through intelligent voice interaction, the control system uses a global camera for environmental perception, a local camera to guide the user to aim at the object, and a multi-fingered prosthetic hand to grasp the object. It realizes auditory and tactile feedback through spatial sound sources, voice prompts and vibration feedback, which can help visually impaired amputees complete perception, positioning, navigation and grasping tasks in indoor environments. The modules and the modules and users work together to form a closed-loop human-computer interaction system to meet the basic living needs of visually impaired amputees. The system and method are natural and intuitive, conform to the daily behavior and activity habits of humans, and provide a practical and usable intelligent prosthetic control system solution for visually impaired amputees.
[0049] Embodiment 2. This embodiment is a further limitation of the intelligent prosthesis control system for visually impaired amputees described in embodiment 1. The head-mounted camera 2 includes an elastic headband 21, an adjustment bracket 22 and a global camera 23. The elastic headband 21 is worn on the user's head, the adjustment bracket 22 is fixed to the elastic headband 21, and the global camera 23 is fixed to the adjustment bracket 22. The adjustment bracket 22 is used to adjust the field of view angle of the global camera 23.
[0050] Embodiment 3. This embodiment further limits the intelligent prosthesis control system for visually impaired amputees described in embodiment 1. The vibration feedback armband 3 includes a main vibration module base 31, a vibration armband battery 32, a vibration armband driving board 33, a main vibration module upper cover 34, a vibration armband controller 35, a vibration armband switch 36, a vibration motor 37, a sub-vibration module 38 and an elastic connector 39; the vibration armband battery 32, the vibration armband driving board 33, and the vibration armband controller 35 are fixed to the main vibration module base 31, the vibration armband battery 32 provides power for the vibration armband driving board 33 and the vibration armband controller 35, the vibration armband switch 36 is fixed to the side of the main vibration module base 31, the vibration motor 37 is fixed to the inner side of the main vibration module base 31 and the sub-vibration module 38, so that the vibration motor 37 is used to press on the user's upper arm skin, and the main vibration module base 31 and the sub-vibration module 38 and the sub-vibration module 38 are connected by the elastic connector 39.
[0051] Embodiment 4. This embodiment is a further limitation of the intelligent prosthetic control system for visually impaired amputees described in embodiment 3. The number of sub-vibration modules 38 is five, and the number of vibration motors 37 is twelve. Each of the sub-vibration modules 38 and the main vibration module base 31 has two vibration motors 37. The vibration motors 37 are connected to the vibration arm ring drive board 33 via a flexible flat cable.
[0052] Embodiment 5: This embodiment is a modification of the smart device for visually impaired amputees described in Embodiment 3.
[0053] Further limitation of the prosthetic control system is that the vibration feedback armband 3 is worn on the user's upper arm, and the method for data interaction with the main controller 8 through wireless communication is: the main vibration module base 31 is located on the upper side of the upper arm, and the vibration armband controller 35 receives the signal of the main controller 8, controls the vibration of the vibration motor 37, and generates vibration feedback information representing different directions by activating the vibration motors 37 at different positions, and the vibrations of all the vibration motors 37 are used to provide feedback information of successful grasping.
[0054] Embodiment 6. This embodiment is a further limitation of the intelligent prosthesis control system for visually impaired amputees described in embodiment 1. The intelligent prosthesis 4 includes a prosthetic hand 41, a wrist rotation motor 42, a wrist flexion motor 43, an intelligent prosthesis controller 44, a wrist drive board 45, a prosthesis battery 46, a local camera 47 and a wrist shell 48. The wrist rotation motor 42 and the wrist flexion motor 43 are fixed to the wrist shell 48 to drive the prosthetic hand 41 to rotate and flex. The prosthesis controller 44 is wirelessly connected to the main controller 8 to control the movement of the prosthetic hand 41, the wrist rotation motor 42 and the wrist flexion motor 43 according to the signal of the main controller 8. The prosthesis battery 46 provides power for the intelligent prosthesis 4. The wrist drive board 45 and the prosthesis battery 46 of the prosthesis controller 44 are fixed to the inside of the wrist shell 48. The local camera 47 is fixed to the side of the wrist shell 48 so that the field of view of the local camera 47 can include the inner part of the palm of the prosthetic hand 41.
[0055] Embodiment 7. This embodiment is a further limitation of the intelligent prosthetic control system for visually impaired amputees described in embodiment 1. The independent sound source module 5 includes a sound source module cover 51, a sound source module speaker 52, a sound source module switch 53, a sound source module battery 54, a sound source module controller 55, a music playback module 56, a power management module 57, a sound source module base 58, a magnetic suction cup 59 and a memory card 60; the sound source module controller 55 receives wireless transmission information from the main controller 8, controls the music playback module 56 to play or pause the music files in the memory card 60, and the sound source module battery 54 provides power for the independent sound source module 5.
[0056] Embodiment 8: This embodiment further limits the intelligent prosthesis control system for visually impaired amputees described in embodiment 7. The shape of the independent sound source module 5 is a disc or a small cuboid.
[0057] Implementation method nine. This implementation method is a further limitation of the intelligent prosthesis control system for visually impaired amputees described in implementation method one. The main controller 8 obtains the category and position information of objects in the field of view by identifying and detecting the images obtained by the global camera 23 and the local camera 47, generates voice prompts and vibration feedback information, and controls the wireless headset 1 and the vibration feedback armband 3 to generate voice prompts and vibration feedback according to the generated information, so as to guide the user to complete environmental perception, mobile navigation and grasping guidance tasks. When the main controller 8 detects that the target object is within the grasping range of the prosthetic hand 41, the main controller 8 controls the prosthetic hand 41 to close to complete the grasping of the target object.
[0058] Embodiment 10: This embodiment proposes an intelligent prosthesis control method for visually impaired amputees. The intelligent prosthesis control method is implemented based on the intelligent prosthesis control system for visually impaired amputees described in any one of embodiments 1 to 9. The intelligent prosthesis control method includes:
[0059] S1. The user first carries the backpack 7 on his back, wears the vibration feedback armband 3 on his upper arm, wears the prosthesis receiving cavity 6 on the end of the residual limb, then fixes the intelligent prosthesis 4 to the prosthesis receiving cavity 6, wears the head-mounted camera 2 on the head, and adjusts the adjustment bracket 22 to ensure that the field of view of the global camera 23 is facing the front of the user, connects the global camera 23 and the main controller 8 through a data cable, and finally wears the wireless headset 1 on the user's ear, turns on the main controller 8, and runs the upper computer program, then correctly turns on the intelligent prosthesis 4, the vibration feedback armband 3 and the wireless headset 1, and waits for the wireless connection to be established between the various parts. When the user receives the "system startup successful" prompt tone generated by the main controller 8 through the wireless headset 1, it means that the system has started correctly;
[0060] S2, selecting different independent sound source modules 5 according to different target object characteristics, and placing the independent sound source module 5 and the target object together, turning on the independent sound source module 5 through the sound source module switch 53, waiting for the independent sound source module 5 to establish a wireless connection with the main controller 8, and after the correct connection, the independent sound source module 5 will play a "connection successful" prompt tone;
[0061] S3, the user inputs the name of the target object to be grabbed to the main controller 8 through a voice command, the main controller 8 recognizes the user's voice command through the voice interaction module and wakes up the independent sound source module 5 bound to the target object, the independent sound source module 5 plays the music set in advance, the user judges the position of the target object according to the sound emitted by the independent sound source module 5, and keeps approaching the target object;
[0062] S4, when the distance between the user and the target object is greater than two meters, the main controller 8 recognizes the image of the global camera 23 through the environmental perception module to obtain the object information in front of the user. The environmental perception module in the main controller 8 divides the field of view of the global camera 23 into a nine-square grid, and generates a position description statement of the target object according to the position of the target object, and then generates a prompt sound through the speech synthesis function, which is played to the user through the wireless headset 1. The user continues to approach the object according to the sound and voice prompts emitted by the independent sound source module 5;
[0063] S5. When the distance between the user and the target object is between one meter and two meters, the main controller 8 will feed back the orientation information of the target object to the user through the vibration feedback armband 3, transmit the direction information of the target object to the user by activating the vibration motors 37 at different positions, and transmit the distance information of the target object to the user by changing the vibration intensity and frequency of the vibration motors 37. The user continues to approach the object according to the sound emitted by the independent sound source module 5 and the vibration prompt generated by the vibration feedback armband 3. When the user is one meter away from the target object, the main controller 8 sends a voice prompt of "reaching the contact range" to the user through the wireless headset 1;
[0064] S6. When the user receives the prompt sound of "reaching the contact range", he lifts the intelligent prosthesis 4, and the main controller 8 recognizes the image of the local camera 47 through the grasping guidance module to obtain the positional relationship between the target object and the prosthetic hand 41, and generates a moving direction prompt sound according to the relative positional relationship to guide the user to move the prosthetic hand 41 closer to the target object;
[0065] S7, when the distance between the artificial hand 41 and the target object is greater than 20 centimeters, the grasping guidance module in the main controller 8 uses a large threshold to calculate the moving direction prompt sound, and the main controller 8 generates the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand 41 left, right, up, down and forward;
[0066] S8, when the distance between the artificial hand 41 and the target object is less than 20 centimeters, the grasping guidance module in the main controller 8 uses a small threshold to calculate the moving direction prompt sound, and the main controller 8 continues to calculate and generate the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand 41 to continue to approach the target object. When the grasping guidance module detects that the target object is within the grasping range of the artificial hand 41, the main controller 8 sends a prompt sound of "reaching the grasping range" to the user through the wireless headset 1;
[0067] S9, the main controller 8 sends a prompt tone of "reaching the grasping range" to the user, and controls the movement of the artificial hand 41, the wrist rotation motor 42 and the wrist flexion motor 43 through the prosthesis coordination control module, so that the artificial hand 41 stably grasps the target object;
[0068] S10, when the object is successfully grasped, the main controller 8 controls the vibration feedback armband 3 to start all the vibration motors 37, and conveys a signal of successful grasping to the user in the form of vibration feedback. At the same time, the main controller 8 puts the independent sound source module 5 bound to the target object into sleep mode through the wireless communication module;
[0069] S11. After the user grasps the object, he performs corresponding operations. When the object needs to be released, the user issues a voice command of "open the prosthetic hand". The wireless headset 1 receives the user's voice command and sends it to the main controller 8. The main controller 8 recognizes the command through the voice interaction module, and controls the prosthetic hand 41 to open through the prosthetic collaborative control module to release the target object.
[0070] Implementation eleven: This implementation provides an example, which is used to explain the above implementations one to ten. The specific example is as follows:
[0071] See also Figures 1 to 9The present embodiment is described. The present embodiment describes an intelligent prosthesis control system for visually impaired amputees, comprising a wireless headset 1, a head-mounted camera 2, a vibration feedback armband 3, an intelligent prosthesis 4, an independent sound source module 5, a prosthesis receiving cavity 6, a backpack 7, a main controller 8 and a mobile power supply 9. The wireless headset 1 is worn on the user's ear, the head-mounted camera 2 is worn on the user's head and connected to the main controller 8 via a data cable, the vibration feedback armband 3 is worn on the user's upper arm, and exchanges data with the main controller 8 via wireless communication, the prosthesis receiving cavity 6 is worn at the end of the user's residual limb, the intelligent prosthesis 4 is fixed to the prosthesis receiving cavity 6, the intelligent prosthesis 4 and the main controller 8 complete data exchange via wireless communication, the main controller 8 and the mobile power supply 9 are placed in the backpack 7, the mobile power supply 9 provides power for the main controller 8, the independent sound source module 5 is placed under the target object, and the independent sound source module 5 and the main controller 8 exchange data via wireless communication.
[0072] Furthermore, the wireless headset 1 is a bone conduction headset, which ensures that the user can normally receive voice prompts while keeping the auditory channel unobstructed, so that the spatial audio emitted by the independent sound source module 5 can be clearly heard.
[0073] The head-mounted camera 2 is composed of an elastic headband 2.1, an adjustment bracket 22 and a global camera 23. The elastic headband 21 is worn on the user's head, the adjustment bracket 22 is fixed to the elastic headband 21, and the global camera 23 is fixed to the adjustment bracket 22. The adjustment bracket 22 can adjust the field of view angle of the global camera 23.
[0074] Furthermore, the global camera 23 is a depth camera, which can simultaneously obtain color and depth images in a global field of view.
[0075] The vibration feedback armband 3 is composed of a main vibration module base 31, a vibration armband battery 32, a vibration armband driving board 33, a main vibration module upper cover 34, a vibration armband controller 35, a vibration armband switch 36, a vibration motor 37, a sub-vibration module 38 and an elastic connecting piece 39. The vibration armband battery 32, the vibration armband driving board 33 and the vibration armband controller 35 are fixed to the main vibration module base 31, the vibration armband battery 32 provides power for the vibration armband driving board 33 and the vibration armband controller 35, the vibration armband switch 36 is fixed to the side of the main vibration module base 31, the vibration motor 37 is fixed to the inner side of the main vibration module base 31 and the sub-vibration module 38, so that the vibration motor 37 can be pressed onto the user's upper arm skin, and the main vibration module base 31 and the sub-vibration module 38 and the sub-vibration module 38 are connected by the elastic connecting piece 39.
[0076] Furthermore, there are five sub-vibration modules 38 and twelve vibration motors 37. Each sub-vibration module 38 and the main vibration module base 31 each have two vibration motors 37. The vibration motor 37 is connected to the vibration arm ring drive board 33 via a flexible flat cable.
[0077] Furthermore, the elastic connector 39 may be made of rubber, which can ensure that the vibration feedback armband 3 can adapt to arms of different thicknesses and ensure that the vibration motor 37 has better contact with the user's skin, thereby producing an obvious vibration feedback effect.
[0078] Furthermore, when wearing the vibration feedback armband 3, it should be ensured that the main vibration module base 31 is located on the upper side of the upper arm, and the vibration armband controller 35 receives the signal of the main controller 8, controls the vibration of the vibration motor 37, and generates vibration feedback information representing different directions by activating the vibration motors 37 at different positions. The vibration of all the vibration motors 37 represents feedback information of successful grasping. The vibration intensity and frequency of the vibration motor 37 are related to the distance between the user and the target object. The closer the distance between the user and the target object, the higher the vibration intensity and frequency of the vibration motor 37.
[0079] The intelligent prosthesis 4 is composed of a prosthetic hand 41, a wrist rotation motor 42, a wrist flexion motor 43, an intelligent prosthesis controller 44, a wrist drive board 45, a prosthesis battery 46, a local camera 47 and a wrist shell 48. The wrist rotation motor 4.2 and the wrist flexion motor 43 are fixed to the wrist shell 48 to drive the prosthetic hand 41 to rotate and flex. The prosthesis controller 44 is wirelessly connected to the main controller 8 to control the movement of the prosthetic hand 41, the wrist rotation motor 42 and the wrist flexion motor 43 according to the signal of the main controller 8. The prosthesis battery 46 provides power for the intelligent prosthesis 4. The prosthesis controller 44, the wrist drive board 45 and the prosthesis battery 46 are fixed to the inside of the wrist shell 48. The local camera 47 is fixed to the side of the wrist shell 48 so that the field of view of the local camera 47 can include the inner part of the palm of the prosthetic hand 41.
[0080] Furthermore, the local camera 47 is a depth camera that can simultaneously obtain color and depth images in a local field of view. The image information of the local camera 47 is transmitted to the prosthesis controller 44 via a data cable, and then wirelessly transmitted to the main controller 8 via the prosthesis controller 44. Image processing is completed in the main controller 8. When the prosthetic hand 41 approaches the target object, the local camera 47 can clearly capture and accurately identify the target object.
[0081] The independent sound source module 5 is composed of a sound source module cover 51, a sound source module speaker 52, a sound source module switch 53, a sound source module battery 54, a sound source module controller 55, a music playback module 56, a power management module 57, a sound source module base 58, a magnetic suction cup 59 and a memory card 60. The sound source module controller 55 receives wireless transmission information from the main controller 8, controls the music playback module 56 to play or pause the music files in the memory card 60, and the sound source module battery 54 provides power for the independent sound source module 5.
[0082] Furthermore, the independent sound source module 5 has various forms and can be designed according to the characteristics of different target objects. The disc-shaped independent sound source module 5 can be placed under objects such as water bottles and water bottles. The small rectangular-shaped independent sound source module 5 can be fixed to the metal door handle through the magnetic suction cup 59. Different independent sound source modules 5 are bound one by one to different target objects. Each independent sound source module 5 is assigned a specific ID number, and the main controller 8 activates the corresponding independent sound source module 5 according to different voice commands of the user.
[0083] Furthermore, the backpack 7 is a hard shell mesh backpack, which not only ensures storage space for the main controller 8 and the mobile power supply 9, but also achieves good heat dissipation.
[0084] Furthermore, the main controller 8 may be an embedded AI development board that can run speech recognition, speech synthesis, and deep learning algorithms, can realize voice interaction, and can simultaneously detect multiple objects of different categories.
[0085] The main controller 8 obtains the category and position information of the object in the field of view by identifying and detecting the images obtained by the global camera 23 and the local camera 47, generates voice prompts and vibration feedback information, and controls the wireless headset 1 and the vibration feedback armband 3 to generate voice prompts and vibration feedback according to the generated information, thereby guiding the user to complete environmental perception, mobile navigation and grasping guidance tasks. When the main controller 8 detects that the target object is within the grasping range of the prosthetic hand 41, the main controller 8 controls the prosthetic hand 41 to close to complete the grasping of the target object.
[0086] According to one aspect of one or more embodiments of the present disclosure, a smart prosthesis control method for visually impaired amputees is provided.
[0087] An intelligent prosthesis control method for visually impaired amputees is implemented based on the intelligent prosthesis control system for visually impaired amputees. The control method comprises:
[0088] S1. The user first carries the backpack 7 on his back, wears the vibration feedback armband 3 on his upper arm, wears the prosthesis receiving cavity 6 on the end of the residual limb, and then correctly fixes the intelligent prosthesis 4 to the prosthesis receiving cavity 6, then wears the head-mounted camera 2 on the head, and adjusts the adjustment bracket 22 to ensure that the field of view of the global camera 2.3 is facing directly in front of the user, connects the global camera 23 and the main controller 8 through a data cable, and finally wears the wireless headset 1 on the user's ear, turns on the main controller 8, and runs the upper computer program, then correctly turns on the intelligent prosthesis 4, the vibration feedback armband 3 and the wireless headset 1, and waits for the wireless connection to be established between the various parts. When the user receives the "system startup successful" prompt tone generated by the main controller 8 through the wireless headset 1, it means that the system has started correctly;
[0089] S2, selecting different independent sound source modules 5 according to different characteristics of the target object, and placing the independent sound source module 5 and the target object together, turning on the independent sound source module 5 through the sound source module switch 53, waiting for the independent sound source module 5 to establish a wireless connection with the main controller 8, and after the correct connection, the independent sound source module 5 will play a "connection successful" prompt tone;
[0090] S3, the user inputs the name of the target object to be grabbed to the main controller 8 through a voice command, the main controller 8 recognizes the user's voice command through the voice interaction module and wakes up the independent sound source module 5 bound to the target object, the independent sound source module 5 plays the music set in advance, the user determines the position of the target object according to the sound emitted by the independent sound source module 5, and keeps approaching the target object;
[0091] S4, when the distance between the user and the target object is greater than two meters, the main controller 8 recognizes the image of the global camera 23 through the environment perception module to obtain the object information in front of the user, and the environment perception module in the main controller 8 divides the field of view of the global camera 23 into a nine-square grid, and generates a position description statement of the target object according to the position of the target object, such as: "The bottle is three meters in front", and then generates a prompt sound through the speech synthesis function, which is played to the user through the wireless headset 1, and the user continues to approach the object according to the sound and voice prompts emitted by the independent sound source module 5;
[0092] S5. When the distance between the user and the target object is between one meter and two meters, the main controller 8 will feed back the position information of the target object to the user through the vibration feedback armband 3, transmit the direction information of the target object to the user by activating the vibration motors 37 at different positions, and transmit the distance information of the target object to the user by changing the vibration intensity and frequency of the vibration motors 37. The user continues to approach the object according to the sound emitted by the independent sound source module 5 and the vibration prompt generated by the vibration feedback armband 3. When the user is one meter away from the target object, the main controller 8 sends a voice prompt of "reaching the contact range" to the user through the wireless headset 1;
[0093] S6. When the user receives the prompt sound of "reaching the contact range", he lifts the intelligent prosthesis 4, and the main controller 8 recognizes the image of the local camera 47 through the grasping guidance module to obtain the positional relationship between the target object and the prosthetic hand 41, and generates a moving direction prompt sound according to the relative positional relationship to guide the user to move the prosthetic hand 41 closer to the target object;
[0094] S7, when the distance between the artificial hand 41 and the target object is greater than 20 centimeters, in order to improve the efficiency of guidance, the grasping guidance module in the main controller 8 uses a large threshold to calculate the moving direction prompt sound, the threshold is determined according to the length and width of the target object, and the large threshold refers to a distance of 5 centimeters from the outer contour of the target object. The main controller 8 generates a moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand 41 to the left, right, up, down and forward;
[0095] S8, when the distance between the artificial hand 41 and the target object is less than 20 centimeters, the grasping guidance module in the main controller 8 uses a small threshold to calculate the moving direction prompt sound, and the small threshold refers to two centimeters away from the outer contour of the target object. The main controller 8 continues to calculate and generate the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand 41 to continue to approach the target object. When the grasping guidance module detects that the target object is within the grasping range of the artificial hand 41, the main controller 8 sends a prompt sound of "reaching the grasping range" to the user through the wireless headset 1;
[0096] S9, the main controller 8 sends a "reaching the grasping range" prompt sound to the user, and controls the movement of the artificial hand 41, the wrist rotation motor 42 and the wrist flexion motor 43 through the prosthesis coordination control module, so that the artificial hand 41 can grasp the target object stably with a suitable posture and gesture;
[0097] S10, when the object is successfully grasped, the main controller 8 controls the vibration feedback armband 3 to start all the vibration motors 37, and conveys a signal of successful grasping to the user in the form of vibration feedback. At the same time, the main controller 8 puts the independent sound source module 5 bound to the target object into sleep mode through the wireless communication module;
[0098] S11. After the user grasps the object, he performs corresponding operations. When the object needs to be released, the user issues a voice command of "open the artificial hand". The wireless headset 1 receives the user's voice command and sends it to the main controller 8. The main controller 8 recognizes the command through the voice interaction module, and controls the artificial hand 41 to open through the prosthesis collaborative control module to release the target object.
[0099] In actual use, according to Figure 1 Wear all the components as shown, start the system to run the host computer software, and wait for each part to establish a connection with the main controller 8. The user can complete the entire process from finding an object to completing the grasping as shown in the figure. Fig. 9 As shown, it can be divided into four stages: awakening, reaching, aligning and grasping.
[0100] Specifically, the user issues a voice command of "wake up the water bottle". After the main controller 8 recognizes the command, it wakes up the independent sound source module 5 under the water bottle to play the set music. The user locates and approaches the water bottle according to the spatial sound source emitted by the independent sound source module 5. During this process, the main controller 8 recognizes the water bottle in the environment through the global camera 23, and further assists the user to locate and approach the water bottle through voice prompts that the distance to the water bottle is greater than two meters and vibration feedback that the distance to the water bottle is between one meter and two meters. When the user is one meter away from the water bottle, the main controller 8 issues a voice prompt of "reaching the contact range". At this time, the user lifts the intelligent prosthesis 4, and the main controller 8 recognizes and calculates the relative position relationship between the water bottle and the prosthetic hand 41 through the local camera 47, and generates a moving direction prompt sound based on the position relationship. The calculation rule of the moving direction prompt sound is as follows: Figure 8 As shown, different calculation thresholds are set according to the distance between the artificial hand 41 and the water bottle. Under the condition of ensuring the alignment accuracy, the efficiency of alignment is improved. Under the guidance of the moving direction prompt sound, the user drives the artificial hand 41 to continuously approach the water bottle. When the main controller 8 detects that the water bottle is within the graspable range of the artificial hand 41, it sends a prompt sound of "reaching the grasping range" to the user, and at the same time puts the independent sound source module 5 located under the water bottle into sleep mode. At this point, the system helps the user complete the tasks of environment recognition, mobile navigation and object grasping.
[0101] After the user grabs the object and completes the corresponding operation, if the user needs to put down the object, he only needs to issue a voice command of "open the artificial hand", and the main controller 8 will control the artificial hand 41 to open, thereby releasing the object.
[0102] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.
[0103] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An intelligent prosthetic control system for visually impaired amputees. It is characterized in that The intelligent prosthesis control system comprises a wireless headset (1), a head-mounted camera (2), a vibration feedback armband (3), an intelligent prosthesis (4), an independent sound source module (5), a prosthesis receiving cavity (6), a backpack (7), a main controller (8) and a mobile power supply (9). The wireless headset (1) is worn on the user's ear, the head-mounted camera (2) is worn on the user's head and connected to the main controller (8) via a data cable, the vibration feedback armband (3) is worn on the user's upper arm and exchanges data with the main controller (8) via wireless communication, the prosthesis receiving cavity (6) is worn at the end of the user's residual limb, the intelligent prosthesis (4) is fixed to the prosthesis receiving cavity (6), the intelligent prosthesis (4) and the main controller (8) complete data exchange via wireless communication, the main controller (8) and the mobile power supply (9) are placed in the backpack (7), the mobile power supply (9) provides power to the main controller (8), the independent sound source module (5) is placed under the target object, and the independent sound source module (5) and the main controller (8) exchange data via wireless communication.
2. The intelligent prosthetic control system for visually impaired amputees according to claim 1, It is characterized in that The head-mounted camera (2) comprises an elastic headband (21), an adjustment bracket (22) and a global camera (23); the elastic headband (21) is worn on the user's head; the adjustment bracket (22) is fixed to the elastic headband (21); the global camera (23) is fixed to the adjustment bracket (22); and the adjustment bracket (22) is used to adjust the field of view angle of the global camera (23).
3. The intelligent prosthetic control system for visually impaired amputees according to claim 1, It is characterized in that The vibration feedback armband (3) comprises a main vibration module base (31), a vibration armband battery (32), a vibration armband driving board (33), a main vibration module upper cover (34), a vibration armband controller (35), a vibration armband switch (36), a vibration motor (37), a sub-vibration module (38) and an elastic connecting piece (39); the vibration armband battery (32), the vibration armband driving board (33) and the vibration armband controller (35) are fixed to the main vibration module base (31); the vibration armband battery (32) is the vibration module of the vibration armband. The arm ring driving board (33) and the vibration arm ring controller (35) provide power, the vibration arm ring switch (36) is fixed to the side of the main vibration module base (31), the vibration motor (37) is fixed to the main vibration module base (31) and the inner side of the sub-vibration module (38), so that the vibration motor (37) is used to press on the upper arm skin of the user, and the main vibration module base (31) and the sub-vibration module (38) and the sub-vibration module (38) are connected through the elastic connecting member (39).
4. The intelligent prosthetic control system for visually impaired amputees according to claim 3, It is characterized in that The number of the sub-vibration modules (38) is five, the number of the vibration motors (37) is twelve, each of the sub-vibration modules (38) and the main vibration module base (31) is provided with two vibration motors (37), and the vibration motors (37) are connected to the vibration arm ring drive board (33) via a flexible flat cable.
5. The intelligent prosthetic control system for visually impaired amputees according to claim 3, It is characterized in that The vibration feedback armband (3) is worn on the upper arm of the user, and the method for exchanging data with the main controller (8) through wireless communication is as follows: the main vibration module base (31) is located on the upper side of the upper arm, the vibration armband controller (35) receives the signal of the main controller (8), controls the vibration of the vibration motor (37), and generates vibration feedback information representing different directions by activating the vibration motors (37) at different positions, and the vibration of all the vibration motors (37) is used to provide feedback information of successful grasping.
6. The intelligent prosthetic control system for visually impaired amputees according to claim 1, It is characterized in that The intelligent prosthesis (4) comprises a prosthetic hand (41), a wrist rotation motor (42), a wrist flexion motor (43), an intelligent prosthesis controller (44), a wrist drive board (45), a prosthesis battery (46), a local camera (47) and a wrist shell (48). The wrist rotation motor (42) and the wrist flexion motor (43) are fixed to the wrist shell (48) to drive the prosthetic hand (41) to rotate and flex. The prosthesis controller (44) is connected to the main controller (8) via a wireless connection. The signal of the main controller (8) controls the movement of the artificial hand (41), the wrist rotation motor (42) and the wrist flexion motor (43); the prosthetic battery (46) provides power for the intelligent prosthesis (4); the wrist drive board (45) and the prosthetic battery (46) of the prosthetic controller (44) are fixed to the inside of the wrist shell (48); the local camera (47) is fixed to the side of the wrist shell (48) so that the field of view of the local camera (47) can include the inner part of the palm of the artificial hand (41).
7. The intelligent prosthetic control system for visually impaired amputees according to claim 1, It is characterized in that The independent sound source module (5) comprises a sound source module upper cover (51), a sound source module speaker (52), a sound source module switch (53), a sound source module battery (54), a sound source module controller (55), a music playback module (56), a power management module (57), a sound source module base (58), a magnetic suction cup (59) and a memory card (60); the sound source module controller (55) receives wireless transmission information from the main controller (8), controls the music playback module (56) to play or pause the music file in the memory card (60), and the sound source module battery (54) provides power for the independent sound source module (5).
8. The intelligent prosthetic control system for visually impaired amputees according to claim 7, It is characterized in that The independent sound source module (5) is in the shape of a disc or a small rectangular parallelepiped.
9. The intelligent prosthetic control system for visually impaired amputees according to claim 1, It is characterized in that The main controller (8) obtains the category and position information of the object in the field of view by identifying and detecting the images obtained by the global camera (23) and the local camera (47), generates voice prompts and vibration feedback information, and controls the wireless headset (1) and the vibration feedback armband (3) to generate voice prompts and vibration feedback according to the generated information, so as to guide the user to complete environmental perception, mobile navigation and grasping guidance tasks. When the main controller (8) detects that the target object is within the grasping range of the artificial hand (41), the main controller (8) controls the artificial hand (41) to close, so as to complete the grasping of the target object.
10. An intelligent prosthesis control method for visually impaired amputees. It is characterized in that The intelligent prosthesis control method is implemented based on the intelligent prosthesis control system for visually impaired amputees according to any one of claims 1 to 9, and the intelligent prosthesis control method comprises: S1, the user first carries the backpack (7) on his back, wears the vibration feedback armband (3) on his upper arm, then wears the prosthesis receiving cavity (6) on the end of the residual limb, then fixes the intelligent prosthesis (4) to the prosthesis receiving cavity (6), wears the head-mounted camera (2) on the head, and adjusts the adjustment bracket (22) to ensure that the field of view of the global camera (23) is facing the front of the user, connects the global camera (23) and the main controller (8) through a data cable, and finally wears the wireless headset (1) on the user's ear, turns on the main controller (8), and runs the upper computer program, then correctly turns on the intelligent prosthesis (4), the vibration feedback armband (3) and the wireless headset (1), and waits for the wireless connection to be established between the various parts. When the user receives the system startup success prompt tone generated by the main controller (8) through the wireless headset (1), it means that the system is correctly started; S2, selecting different independent sound source modules (5) according to different characteristics of the target object, placing the independent sound source module (5) and the target object together, turning on the independent sound source module (5) through the sound source module switch (53), waiting for the independent sound source module (5) to establish a wireless connection with the main controller (8), and after the connection is correctly established, the independent sound source module (5) will play a "connection successful" prompt tone; S3, the user inputs the name of the target object to be grasped to the main controller (8) through a voice command, the main controller (8) recognizes the user's voice command through the voice interaction module and wakes up the independent sound source module (5) bound to the target object, the independent sound source module (5) plays the music set in advance, the user determines the position of the target object according to the sound emitted by the independent sound source module (5), and continuously approaches the target object; S4, when the distance between the user and the target object is greater than two meters, the main controller (8) recognizes the image of the global camera (23) through the environment perception module to obtain information about the object in front of the user, the environment perception module in the main controller (8) divides the field of view of the global camera (23) into a nine-square grid, and generates a position description statement of the target object according to the position of the target object, and then generates a prompt sound through the speech synthesis function, which is played to the user through the wireless headset (1), and the user continues to approach the object according to the sound and voice prompts emitted by the independent sound source module (5); S5. When the distance between the user and the target object is between one meter and two meters, the main controller (8) will feed back the orientation information of the target object to the user through the vibration feedback armband (3), transmit the direction information of the target object to the user by activating the vibration motors (37) at different positions, and transmit the distance information of the target object to the user by changing the vibration intensity and frequency of the vibration motors (37). The user continues to approach the object according to the sound emitted by the independent sound source module (5) and the vibration prompt generated by the vibration feedback armband (3). When the user is one meter away from the target object, the main controller (8) sends a voice prompt to the user through the wireless headset (1) indicating that the user has reached the contact range. S6. When the user receives a prompt sound indicating that the contact range has been reached, the user lifts the intelligent prosthesis (4), and the main controller (8) recognizes the image of the local camera (47) through a grasping guidance module to obtain a positional relationship between the target object and the prosthetic hand (41), and generates a moving direction prompt sound based on the relative positional relationship to guide the user to move the prosthetic hand (41) closer to the target object; S7, when the distance between the artificial hand (41) and the target object is greater than 20 centimeters, the grasping guidance module in the main controller (8) uses a large threshold to calculate the moving direction prompt sound, and the main controller (8) generates the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand (41) left, right, up, down and forward; S8, when the distance between the artificial hand (41) and the target object is less than 20 centimeters, the grasping guidance module in the main controller (8) uses a small threshold to calculate the moving direction prompt sound, and the main controller (8) continues to calculate and generate the moving direction prompt sound through the grasping guidance module to prompt the user to move the artificial hand (41) to continue to approach the target object. When the grasping guidance module detects that the target object is within the grasping range of the artificial hand (41), the main controller (8) sends a prompt sound to the user through the wireless headset (1) indicating that the target object has reached the grasping range; S9, the main controller (8) sends a prompt sound to the user indicating that the grasping range has been reached, and controls the movement of the artificial hand (41), the wrist rotation motor (42) and the wrist flexion motor (43) through the prosthetic limb coordination control module, so that the artificial hand (41) can stably grasp the target object; S10, when the object is successfully grasped, the main controller (8) controls the vibration feedback armband (3) to start all the vibration motors (37) to convey a signal of successful grasping to the user in the form of vibration feedback, and at the same time, the main controller (8) puts the independent sound source module (5) bound to the target object into sleep mode through the wireless communication module; S11. After grasping the object, the user performs corresponding operations. When the object needs to be released, the user issues a voice command to open the artificial hand. The wireless headset (1) receives the user's voice command and sends it to the main controller (8). The main controller (8) recognizes the command through the voice interaction module and controls the artificial hand (41) to open through the prosthetic collaborative control module to release the target object.
Citation Information
Patent Citations
Multi-modal information guided environmental perception myoelectric artificial limb system and environmental perception method
CN112587285A
Modularized closed-loop artificial limb control system for upper limb amputation patient
CN114681172A
Intelligent prosthetic hand fusing multi-modal information to serve blind person and control method of intelligent prosthetic hand
CN117653429A
Soft neuroprosthetic limb with simultaneous myoelectric control and tactile feedback
WO2021257640A1
