Intelligent wheelchair driving control system based on voiceprint eye movement control

By using voiceprint eye-tracking control and 3D semantic mapping technology, the problem of low accuracy in wheelchair positioning and obstacle avoidance for ALS patients has been solved, enabling precise navigation and safe movement of wheelchairs.

CN118975895APending Publication Date: 2024-11-19HEFEI INST OF TECH INNOVATION ENG CHINESE ACAD OF SCI
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202411052051.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing wheelchair drive control systems are not suitable for ALS patients, as their positioning and obstacle avoidance accuracy is low, resulting in ALS patients being unable to move independently.

Method used

Employing voiceprint eye-tracking control technology, combined with an RGB-D camera and LiDAR, the system detects the user's eye movement trajectory through sound waves, generates control commands, constructs a 3D semantic map for localization and obstacle avoidance, and uses the artificial potential field method to plan the path, achieving precise navigation for the wheelchair.

Benefits of technology

This technology enables ALS patients to effectively control their wheelchairs through eye movements, improving positioning and obstacle avoidance accuracy and ensuring safe movement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975895A_ABST
    Figure CN118975895A_ABST
Patent Text Reader

Abstract

The invention relates to wheelchair driving control, in particular to an intelligent wheelchair driving control system based on voiceprint eye movement control, which comprises a control unit, and the control unit sends detection sound waves to the eyes of a user through a sound wave generator and receives echo signals of the detection sound waves reflected by the eyes of the user through a sound wave sensor array. The control unit performs signal processing and feature extraction on the echo signal through a signal processing and analyzing module, determines an eye movement track of the user, and generates a corresponding control instruction based on the eye movement track of the user by using a control instruction generating module; according to the technical scheme provided by the invention, the defects that a control mode in the prior art is not suitable for an asymptomatic patient, the positioning precision is relatively low and the obstacle avoidance precision is relatively low can be effectively overcome.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to wheelchair drive control, and in particular to an intelligent wheelchair drive control system based on voiceprint eye movement control. Background Art

[0002] ALS is a neurodegenerative disease that primarily affects motor neurons, causing muscle weakness and atrophy. As the disease progresses, patients will gradually lose their mobility and speech, and eventually need to rely on assistive devices to maintain their daily lives.

[0003] As the symptoms of ALS worsen, patients will begin to experience muscle atrophy in their fingers and upper limbs, making it difficult for them to take care of themselves. The time and economic costs of care are high, causing great inconvenience to the patients themselves and their families. Most existing wheelchair drive control systems rely on users to manually input control commands to move the wheelchair to where the user wants to go. However, this control method is not suitable for ALS patients, and existing wheelchair drive control systems have problems such as low positioning accuracy and low obstacle avoidance accuracy, which makes it impossible for ALS patients to use wheelchairs for autonomous movement. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides an intelligent wheelchair drive control system based on voiceprint and eye movement control, which can effectively overcome the defects of the prior art that the control method is not suitable for ALS patients, as well as the low positioning accuracy and low obstacle avoidance accuracy.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] An intelligent wheelchair driving control system based on voiceprint eye movement control comprises a control unit, wherein the control unit sends detection sound waves to the user's eyes through a sound wave generator, and uses a sound wave sensor array to receive echo signals reflected by the detection sound waves from the user's eyes, wherein the control unit performs signal processing and feature extraction on the echo signals through a signal processing and analysis module, determines the user's eye movement trajectory, and uses a control instruction generation module to generate corresponding control instructions based on the user's eye movement trajectory;

[0009] The control unit collects environmental images of the surrounding environment through the environmental image acquisition module, and generates a three-dimensional environmental map based on the environmental image sequence using the environmental map construction module; the control unit identifies the category information of the surrounding environmental objects through the target detection module, and integrates the category information of the surrounding environmental objects into the three-dimensional environmental map using the semantic map construction module to generate a three-dimensional semantic map; the control unit locates the starting point and the target point based on the three-dimensional semantic map through the positioning module;

[0010] The control unit performs preliminary driving path planning based on the starting point and the target point through the driving path planning module, and obtains obstacle information of the surrounding environment through the obstacle information acquisition module. The control unit screens the obstacles in the surrounding environment through the target obstacle determination module, obtains the target obstacle, and uses the obstacle avoidance driving module to drive and control the wheelchair to avoid the target obstacle and drive to the target point.

[0011] Preferably, the signal processing and analysis module performs signal processing and feature extraction on the echo signal to determine the user's eye movement trajectory, including:

[0012] Perform high-frequency filtering and signal amplification processing on the echo signal to convert the echo signal into an echo profile;

[0013] Extract the voiceprint frequency features reflecting the distance from the eyes in the echo profile, and use the voiceprint frequency features to determine the user's eye movement trajectory;

[0014] Among them, the deep convolutional neural network model ResNet18 is used to extract the voiceprint frequency characteristics reflecting the distance from the eyes in the echo profile.

[0015] Preferably, the control instruction generation module generates corresponding control instructions based on the user's eye movement trajectory, including:

[0016] The user's gaze direction is determined according to the user's eye movement trajectory, and the target object or target area in the user's sight is determined, and the position of the target object or target area is converted into a corresponding control instruction.

[0017] Preferably, the environment image acquisition module acquires environment images of the surrounding environment, and the environment map construction module generates a three-dimensional environment map based on the environment image sequence, including:

[0018] The RGB-D camera carried on the wheelchair is used to collect environmental images of the surrounding environment to form an environmental image sequence;

[0019] A 3D environment map is generated based on the environment image sequence using the SLAM algorithm.

[0020] Preferably, the target detection module identifies the category information of the surrounding environment objects, including:

[0021] Perform semantic segmentation on key frames in the environmental image sequence, and perform target detection on surrounding environmental objects in the key frames to determine the category information of surrounding environmental objects;

[0022] Among them, the region-based convolutional neural network mask model Mask R-CNN is used to perform semantic segmentation on the key frames in the environmental image sequence, and to detect the surrounding objects in the key frames.

[0023] Preferably, the positioning module locates the starting point and the target point based on the three-dimensional semantic map, including:

[0024] The RGB-D camera carried on the wheelchair is used to collect real-time environment images of the surrounding environment, perform semantic segmentation on the real-time environment images, and perform target detection on the surrounding environment objects in the real-time environment images to determine the category information of the surrounding environment objects;

[0025] Based on the 3D semantic map, the wheelchair's position is roughly estimated using the category information of surrounding objects.

[0026] Obtain the distance between the wheelchair and surrounding objects, and accurately locate the wheelchair based on the rough position estimate to determine the starting point;

[0027] Determine the target point position according to the control instruction;

[0028] Among them, the time-of-flight method ToF is used to obtain the distance between the wheelchair and the surrounding objects.

[0029] Preferably, the obstacle information acquisition module acquires obstacle information of the surrounding environment, and the target obstacle determination module screens obstacles in the surrounding environment to acquire target obstacles, including:

[0030] Integrate the data collected by LiDAR and RGB-D camera to obtain obstacle information in the surrounding environment;

[0031] It is determined whether the distance between the obstacle and the preliminary driving path is less than a preset distance threshold according to the obstacle information of the surrounding environment. If it is less than the preset distance threshold, the obstacle is regarded as a target obstacle.

[0032] Preferably, the obstacle avoidance driving module drives and controls the wheelchair to avoid the target obstacle and travel to the target point, including:

[0033] Based on the preliminary driving path, the artificial potential field method is used to drive and control the wheelchair to avoid the target obstacles and drive to the target point, including:

[0034] S1, driving and controlling the wheelchair to travel along the preliminary travel path, and determining whether the wheelchair falls into a local optimal state during the travel process;

[0035] S2. If the wheelchair gets stuck in a local optimum state, introduce a repulsive force deflection model to deflect the repulsive force to reduce the angle between the gravitational force and the repulsive force; otherwise, continue to drive and control the wheelchair to travel along the preliminary travel path.

[0036] S3. Repeat S1 - S2 until the wheelchair reaches the target point.

[0037] Preferably, if the wheelchair gets stuck in a local optimum state in S2, after introducing a repulsive force deflection model to deflect the repulsive force, it includes:

[0038] S21. Determine whether the deflected wheelchair gets stuck in a local optimum state. If it still gets stuck in a local optimum state, set a virtual sub - target point, drive and control the wheelchair to move to the virtual sub - target point, and enter S22; otherwise, continue to drive and control the wheelchair to travel along the preliminary travel path.

[0039] S22. Determine whether the moved wheelchair gets stuck in a local optimum state. If it still gets stuck in a local optimum state, set a virtual sub - target point again, drive and control the wheelchair to move to the virtual sub - target point, and repeat S22 until the wheelchair escapes from the local optimum state.

[0040] Preferably, the method for determining whether it gets stuck in a local optimum state includes:

[0041] Set the radius of the local optimum trap circle as r, record the time taken for the wheelchair to travel a path of length r as t, and the travel distance of the wheelchair during this time t as d.

[0042] If d < r, the wheelchair gets stuck in a local optimum state; otherwise, the wheelchair does not get stuck in a local optimum state.

[0043] (III) Advantageous Effects

[0044] Compared with the prior art, the intelligent wheelchair drive control system based on voiceprint eye movement control provided by the present invention has the following advantageous effects:

[0045] 1) The sound wave generator emits detection sound waves to the user's eyes. The sound wave sensor array receives the echo signals reflected by the user's eyes after receiving the detection sound waves. The signal processing and analysis module performs signal processing and feature extraction on the echo signals to determine the user's eye movement trajectory. The control instruction generation module generates corresponding control instructions based on the user's eye movement trajectory. By using the voiceprint eye movement control technology to generate corresponding control instructions based on the user's eye movement trajectory, patients with amyotrophic lateral sclerosis can effectively control the wheelchair drive through eye movement, which is convenient for patients with amyotrophic lateral sclerosis to use.

[0046] 2) The environmental image acquisition module acquires environmental images of the surrounding environment, the environmental map construction module generates a three-dimensional environmental map based on the environmental image sequence, the target detection module identifies the category information of the surrounding environmental objects, the semantic map construction module integrates the category information of the surrounding environmental objects into the three-dimensional environmental map, and generates a three-dimensional semantic map. The positioning module locates the starting point and the target point based on the three-dimensional semantic map, and generates a three-dimensional environmental map by using the SLAM algorithm based on the environmental image sequence, and integrates the category information of the surrounding environmental objects into the three-dimensional environmental map by performing target detection on the surrounding environmental objects, and generates a three-dimensional semantic map, and realizes accurate positioning of the starting point and the target point based on the three-dimensional semantic map;

[0047] 3) The driving path planning module performs preliminary driving path planning based on the starting point and the target point. The obstacle information acquisition module obtains the obstacle information of the surrounding environment. The target obstacle determination module screens the obstacles in the surrounding environment and obtains the target obstacles. The obstacle avoidance driving module drives and controls the wheelchair to avoid the target obstacles and drive to the target point. Therefore, based on the preliminary driving path, the artificial potential field method can be used to drive and control the wheelchair to avoid the target obstacles and drive to the target point, effectively improving the obstacle avoidance accuracy, ensuring that the wheelchair can safely drive to the target point, and ensuring the safety of use for ALS patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a system schematic diagram of the present invention;

[0050] Figure 2 It is a schematic diagram of the process of locating the starting point and the target point based on the three-dimensional semantic map in the present invention;

[0051] Figure 3 It is a schematic diagram of the process of driving and controlling the wheelchair to avoid the target obstacle and travel to the target point in the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] An intelligent wheelchair drive control system based on voiceprint eye movement control, such as Figure 1 As shown, it includes a control unit, which sends detection sound waves to the user's eyes through an acoustic wave generator, and uses an acoustic wave sensor array to receive echo signals reflected by the user's eyes from the detection sound waves. The control unit performs signal processing and feature extraction on the echo signals through a signal processing and analysis module to determine the user's eye movement trajectory, and uses a control instruction generation module to generate corresponding control instructions based on the user's eye movement trajectory.

[0054] ① The signal processing and analysis module processes the echo signal and extracts features to determine the user's eye movement trajectory, including:

[0055] Perform high-frequency filtering and signal amplification processing on the echo signal to convert the echo signal into an echo profile;

[0056] Extract the voiceprint frequency features reflecting the distance from the eyes in the echo profile, and use the voiceprint frequency features to determine the user's eye movement trajectory;

[0057] Among them, the deep convolutional neural network model ResNet18 is used to extract the voiceprint frequency characteristics reflecting the distance from the eyes in the echo profile.

[0058] ② The control instruction generation module generates corresponding control instructions based on the user's eye movement trajectory, including:

[0059] The user's gaze direction is determined according to the user's eye movement trajectory, and the target object or target area in the user's sight is determined, and the position of the target object or target area is converted into a corresponding control instruction.

[0060] In the above technical solution, the sound wave generator sends detection sound waves to the user's eyes, the sound wave sensor array receives the echo signal reflected by the user's eyes, the signal processing and analysis module performs signal processing and feature extraction on the echo signal to determine the user's eye movement trajectory, and the control instruction generation module generates corresponding control instructions based on the user's eye movement trajectory. The voiceprint eye movement control technology is used to generate corresponding control instructions based on the user's eye movement trajectory, so that ALS patients can effectively control the wheelchair drive through eye movements, which is convenient for ALS patients to use.

[0061] In the technical solution of this application, based on the voiceprint eye movement control technology, ALS patients can also realize voiceprint eye movement control of smart homes. Lightweight IoT communication protocols such as MQTT are used to achieve a stable communication connection between the smart wheelchair drive control system and smart home devices. This communication protocol can support fast data transmission and response. Develop responsive mobile applications or web interfaces, and ALS patients can remotely control smart home devices through eye movement control commands. The system supports preset scenarios and automated tasks, such as timed switching of lights, temperature adjustment, etc.

[0062] like Figure 1 and Figure 2 As shown, the control unit collects environmental images of the surrounding environment through the environmental image acquisition module, and uses the environmental map construction module to generate a three-dimensional environmental map based on the environmental image sequence. The control unit identifies the category information of the surrounding environment objects through the target detection module, and uses the semantic map construction module to integrate the category information of the surrounding environment objects into the three-dimensional environmental map to generate a three-dimensional semantic map. The control unit locates the starting point and the target point based on the three-dimensional semantic map through the positioning module.

[0063] ① The environmental image acquisition module acquires environmental images of the surrounding environment, and the environmental map construction module generates a three-dimensional environmental map based on the environmental image sequence, including:

[0064] The RGB-D camera carried on the wheelchair is used to collect environmental images of the surrounding environment to form an environmental image sequence;

[0065] A 3D environment map is generated based on the environment image sequence using the SLAM algorithm.

[0066] ②The target detection module identifies the category information of objects in the surrounding environment, including:

[0067] Perform semantic segmentation on key frames in the environmental image sequence, and perform target detection on surrounding environmental objects in the key frames to determine the category information of surrounding environmental objects;

[0068] Among them, the region-based convolutional neural network mask model Mask R-CNN is used to perform semantic segmentation on the key frames in the environmental image sequence, and to detect the surrounding objects in the key frames.

[0069] ③ The positioning module locates the starting point and the target point based on the three-dimensional semantic map, including:

[0070] The RGB-D camera carried on the wheelchair is used to collect real-time environment images of the surrounding environment, perform semantic segmentation on the real-time environment images, and perform target detection on the surrounding environment objects in the real-time environment images to determine the category information of the surrounding environment objects;

[0071] Based on the 3D semantic map, the wheelchair's position is roughly estimated using the category information of surrounding objects.

[0072] Obtain the distance between the wheelchair and surrounding objects, and accurately locate the wheelchair based on the rough position estimate to determine the starting point;

[0073] Determine the target point position according to the control instruction;

[0074] Among them, the time-of-flight method ToF is used to obtain the distance between the wheelchair and the surrounding objects.

[0075] In the above technical solution, the environmental image acquisition module acquires environmental images of the surrounding environment, the environmental map construction module generates a three-dimensional environmental map based on the environmental image sequence, the target detection module identifies the category information of the surrounding environment objects, the semantic map construction module integrates the category information of the surrounding environment objects into the three-dimensional environmental map to generate a three-dimensional semantic map, the positioning module locates the starting point and the target point based on the three-dimensional semantic map, generates a three-dimensional environmental map based on the environmental image sequence using the SLAM algorithm, and integrates the category information of the surrounding environment objects into the three-dimensional environmental map by performing target detection on the surrounding environment objects to generate a three-dimensional semantic map, and realizes accurate positioning of the starting point and the target point based on the three-dimensional semantic map.

[0076] like Figure 1 and Figure 3 As shown, the control unit performs preliminary driving path planning based on the starting point and the target point through the driving path planning module, and uses the obstacle information acquisition module to obtain obstacle information of the surrounding environment. The control unit screens the obstacles in the surrounding environment through the target obstacle determination module, obtains the target obstacle, and uses the obstacle avoidance driving module to drive and control the wheelchair to avoid the target obstacle and drive to the target point.

[0077] ① The obstacle information acquisition module acquires obstacle information in the surrounding environment, and the target obstacle determination module screens obstacles in the surrounding environment and acquires target obstacles, including:

[0078] Integrate the data collected by LiDAR and RGB-D camera to obtain obstacle information in the surrounding environment;

[0079] It is determined whether the distance between the obstacle and the preliminary driving path is less than a preset distance threshold according to the obstacle information of the surrounding environment. If it is less than the preset distance threshold, the obstacle is regarded as a target obstacle.

[0080] ② The obstacle avoidance drive module drives and controls the wheelchair to avoid the target obstacle and drive to the target point, including:

[0081] Based on the preliminary driving path, the artificial potential field method is used to drive and control the wheelchair to avoid the target obstacles and drive to the target point, including:

[0082] S1. Drive and control the wheelchair to travel along the preliminary travel path, and determine whether the wheelchair is trapped in a local optimal state during the travel;

[0083] S2. If the wheelchair is trapped in a local optimal state, introduce a repulsive force deflection model to deflect the repulsive force to reduce the angle between the gravitational force and the repulsive force; otherwise, continue to drive and control the wheelchair to travel along the preliminary travel path;

[0084] S3. Repeat S1 - S2 until the wheelchair travels to the target point.

[0085] Specifically, after introducing the repulsive force deflection model to deflect the repulsive force when the wheelchair is trapped in a local optimal state in S2, it includes:

[0086] S21. Determine whether the deflected wheelchair is trapped in a local optimal state. If it is still trapped in a local optimal state, set a virtual sub - target point, drive and control the wheelchair to move to the virtual sub - target point, and enter S22; otherwise, continue to drive and control the wheelchair to travel along the preliminary travel path;

[0087] S22. Determine whether the moved wheelchair is trapped in a local optimal state. If it is still trapped in a local optimal state, set a virtual sub - target point again, drive and control the wheelchair to move to the virtual sub - target point, and repeat S22 until the wheelchair escapes from the local optimal state.

[0088] In the technical solution of this application, the method for judging whether it is trapped in a local optimal state includes:

[0089] Set the radius of the local optimal trap circle as r, record the time taken for the wheelchair to travel a path of length r as t, and the travel distance of the wheelchair within this time t after time t as d;

[0090] If d < r, the wheelchair is trapped in a local optimal state; otherwise, the wheelchair is not trapped in a local optimal state.

[0091] In the above - mentioned technical solution, the travel path planning module conducts preliminary travel path planning based on the starting point and the target point, the obstacle information acquisition module acquires the obstacle information of the surrounding environment, the target obstacle determination module screens the obstacles in the surrounding environment to obtain the target obstacles, and the obstacle avoidance driving module drives and controls the wheelchair to avoid the target obstacles and travel to the target point. Thus, it can drive and control the wheelchair to avoid the target obstacles and travel to the target point based on the preliminary travel path by using the artificial potential field method, effectively improving the obstacle avoidance accuracy, ensuring that the wheelchair can safely travel to the target point, and guaranteeing the use safety of patients with amyotrophic lateral sclerosis.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent wheelchair driving control system based on voiceprint eye movement control, characterized in that: The control unit includes a control unit, which sends detection sound waves to the user's eyes through a sound wave generator, and uses a sound wave sensor array to receive echo signals reflected by the detection sound waves from the user's eyes, and the control unit performs signal processing and feature extraction on the echo signals through a signal processing and analysis module to determine the user's eye movement trajectory, and uses a control instruction generation module to generate corresponding control instructions based on the user's eye movement trajectory; The control unit collects environmental images of the surrounding environment through the environmental image acquisition module, and generates a three-dimensional environmental map based on the environmental image sequence using the environmental map construction module; the control unit identifies the category information of the surrounding environmental objects through the target detection module, and integrates the category information of the surrounding environmental objects into the three-dimensional environmental map using the semantic map construction module to generate a three-dimensional semantic map; the control unit locates the starting point and the target point based on the three-dimensional semantic map through the positioning module; The control unit performs preliminary driving path planning based on the starting point and the target point through the driving path planning module, and obtains obstacle information of the surrounding environment through the obstacle information acquisition module. The control unit screens the obstacles in the surrounding environment through the target obstacle determination module, obtains the target obstacle, and uses the obstacle avoidance driving module to drive and control the wheelchair to avoid the target obstacle and drive to the target point.

2. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 1 is characterized in that: The signal processing and analysis module performs signal processing and feature extraction on the echo signal to determine the user's eye movement trajectory, including: Perform high-frequency filtering and signal amplification processing on the echo signal to convert the echo signal into an echo profile; Extract the voiceprint frequency features reflecting the distance from the eyes in the echo profile, and use the voiceprint frequency features to determine the user's eye movement trajectory; Among them, the deep convolutional neural network model ResNet18 is used to extract the voiceprint frequency characteristics reflecting the distance from the eyes in the echo profile.

3. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 2 is characterized in that: The control instruction generation module generates corresponding control instructions based on the user's eye movement trajectory, including: The user's gaze direction is determined according to the user's eye movement trajectory, and the target object or target area in the user's sight is determined, and the position of the target object or target area is converted into a corresponding control instruction.

4. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 1 is characterized in that: The environment image acquisition module acquires environment images of the surrounding environment, and the environment map construction module generates a three-dimensional environment map based on the environment image sequence, including: The RGB-D camera carried on the wheelchair is used to collect environmental images of the surrounding environment to form an environmental image sequence; A 3D environment map is generated based on the environment image sequence using the SLAM algorithm.

5. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 4 is characterized in that: The target detection module identifies the category information of the surrounding environment objects, including: Perform semantic segmentation on key frames in the environmental image sequence, and perform target detection on surrounding environmental objects in the key frames to determine the category information of surrounding environmental objects; Among them, the region-based convolutional neural network mask model Mask R-CNN is used to perform semantic segmentation on the key frames in the environmental image sequence, and to detect the surrounding objects in the key frames.

6. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 5 is characterized in that: The positioning module locates the starting point and the target point based on the three-dimensional semantic map, including: Collect real-time environmental images of the surrounding environment using an RGB-D camera carried on a wheelchair, perform semantic segmentation on the real-time environmental images, and perform object detection on the surrounding environmental objects in the real-time environmental images to determine the category information of the surrounding environmental objects; Based on the three-dimensional semantic map, use the category information of the surrounding environmental objects to roughly estimate the position of the wheelchair; Obtain the distance between the wheelchair and the surrounding environmental objects, and combine the rough position estimate to accurately locate the position of the wheelchair to determine the starting point position; Determine the target point position according to the control instruction; Among them, the time-of-flight method ToF is used to obtain the distance between the wheelchair and the surrounding environmental objects.

7. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 1 is characterized in that: The obstacle information acquisition module acquires the obstacle information of the surrounding environment, and the target obstacle determination module screens the obstacles in the surrounding environment to obtain the target obstacles, including: Integrate the acquisition data of the lidar and the RGB-D camera to obtain the obstacle information of the surrounding environment; According to the obstacle information of the surrounding environment, judge whether the distance between the obstacle and the preliminary driving path is less than the preset distance threshold. If it is less than the preset distance threshold, then regard the obstacle as the target obstacle.

8. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 7 is characterized in that: The obstacle avoidance driving module drives and controls the wheelchair to avoid the target obstacle and drive to the target point, including: Based on the preliminary driving path, use the artificial potential field method to drive and control the wheelchair to avoid the target obstacle and drive to the target point. Specifically, it includes: S1. Drive and control the wheelchair to drive along the preliminary driving path, and judge whether the wheelchair falls into the local optimal state during the driving process; S2. If the wheelchair falls into the local optimal state, then introduce a repulsive force deflection model to deflect the repulsive force to reduce the angle between the gravitational force and the repulsive force. Otherwise, continue to drive and control the wheelchair to drive along the preliminary driving path; S3. Repeat S1~S2 until the wheelchair drives to the target point.

9. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 8 is characterized in that: After introducing a repulsive force deflection model to deflect the repulsive force when the wheelchair falls into the local optimal state in S2, it includes: S21. Judge whether the deflected wheelchair falls into the local optimal state. If it still falls into the local optimal state, then set a virtual sub-target point, drive and control the wheelchair to move to the virtual sub-target point, and enter S22. Otherwise, continue to drive and control the wheelchair to drive along the preliminary driving path; S22. Judge whether the moved wheelchair falls into the local optimal state. If it still falls into the local optimal state, then set a virtual sub-target point again, drive and control the wheelchair to move to the virtual sub-target point, and repeat S22 until the wheelchair escapes from the local optimal state.

10. The intelligent wheelchair driving control system based on voiceprint eye movement control according to claim 8 or 9, characterized in that: The method for judging whether it falls into the local optimal state includes: Set the radius of the local optimal trap circle as r, record the time t used for the wheelchair to drive a path of length r, and the driving distance d of the wheelchair during this time after time t; If d < r, then the wheelchair falls into the local optimal state, otherwise the wheelchair does not fall into the local optimal state.

Citation Information

Patent Citations

  • Sight tracing method and disabled assisting system using the same

    CN101344919A

  • Human-computer interaction system and method of intelligent wheelchair based on head posture

    CN102048621A

  • Brain electric and eye electric signal decision fusion method for identifying control intention

    CN102521505A

  • Wheelchair system on basis of Bluetooth myoelectric collectors and method for controlling wheelchair system

    CN104622649A

  • Automatic obstacle avoidance method of multi-sensor based brain-controlled wheelchair

    CN110584898A