Intelligent auxiliary blind guiding system and method based on audio-visual fusion and tactile feedback

Abstract

The invention discloses an intelligent auxiliary blind guiding system and method based on audio-visual fusion and tactile feedback, and relates to the technical field of intelligent assistion.The intelligent auxiliary blind guiding system comprises a machine blind guiding dog and a blind guiding stick, and the machine blind guiding dog comprises a machine blind guiding dog body and a four-foot driving device; a visual module, a voice interaction module, a data processing module and a control module are arranged on the machine blind guiding dog body, a tactile feedback module is arranged at a handle at the top end of the blind guiding stick, a surrounding environment map is constructed according to surrounding environment information, and a planned path and a moving strategy are generated in combination with a user voice instruction and a user push-pull force; a walking prompt instruction and a driving instruction are generated, the machine blind guiding dog is driven to carry out auxiliary blind guiding guidance, and touch vibration feedback is carried out according to the walking prompt instruction. According to the invention, path planning, autonomous navigation and real-time obstacle avoidance in various complex environments can be realized, and travel assistance with high intelligence, automation and reliability is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent assistance technology, and particularly to an intelligent assistance blind guiding system and method based on audio-visual fusion and tactile feedback. Background Art

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] In recent years, with the continuous progress of technology, more and more intelligent assistance devices have been used to assist visually impaired people in solving problems in life. As a long-term used assistance tool, traditional blind guiding dogs for the blind play an important role, but there are also some drawbacks that cannot be ignored: First, the training cost is high and the cycle is long. Blind guiding dogs need to undergo 2-3 years of professional training before they can be put into use, and the training cost of each guiding dog is relatively high. Dependent on the physical conditions of animals, the working cycle is limited. Second, the life management and working environment are restricted. Guiding dogs need regular feeding, health management and psychological training, and guiding dogs can only be used in limited environments and are not applicable to all scenarios. In addition, traditional guiding dogs have poor environmental adaptability. In complex urban environments, busy traffic or dynamically changing places, the working efficiency and reaction ability of guiding dogs are limited, and there are certain risks.

[0004] With the progress of technology, existing intelligent assistance devices such as intelligent blind guiding canes and voice assistants have also begun to be used by visually impaired people. However, most of these devices rely on the manual operation of users and cannot adapt to complex dynamic environments, with great limitations. Existing intelligent blind guiding canes can only provide obstacle perception and simple navigation guidance, cannot achieve automatic path planning and real-time obstacle avoidance, cannot achieve adaptive auxiliary adjustment in different scenarios, and cannot ensure the safety of users. Summary of the Invention

[0005] To solve the above deficiencies of the prior art, the present invention provides an intelligent assistance blind guiding system and method based on audio-visual fusion and tactile feedback. By integrating a variety of sensor technologies, voice interaction technologies and tactile feedback mechanisms, it can adapt to path planning, autonomous navigation and real-time obstacle avoidance in various complex environments, and can provide highly intelligent, automated and reliable travel assistance functions for visually impaired people, ensuring the travel safety of users and avoiding the problems that traditional auxiliary blind guiding devices need manual operation and cannot adapt to complex dynamic environments and cannot perform adaptive auxiliary adjustment.

[0006] In the first aspect, the present invention provides an intelligent assistance blind guiding system based on audio-visual fusion and tactile feedback.

[0007] An intelligent assisted blind navigation system based on audiovisual fusion and tactile feedback, comprising a robotic guide dog and a guide stick. The robotic guide dog includes a robotic guide dog main body and a quadruped drive device. The following are provided on the robotic guide dog main body:

[0008] A vision module for real-time perception of surrounding environment information;

[0009] A voice interaction module for voice interaction with the user, receiving user voice commands and providing voice navigation prompts according to walking prompt commands;

[0010] A data processing module for constructing a surrounding environment map based on the received surrounding environment information and generating a planned path and a movement strategy in combination with user voice commands; the movement strategy includes the current required walking speed;

[0011] A control module for generating walking prompt commands and drive commands for driving the quadruped drive device in combination with the planned path and movement strategy, user voice commands and user push-pull forces;

[0012] A tactile feedback module is provided at the top handle of the guide stick. The tactile feedback module includes a vibration device and a mechanical induction element for sensing the user's push-pull force. The vibration device is used for tactile vibration feedback according to the walking prompt command.

[0013] In a second aspect, the present invention provides an intelligent assisted blind navigation method based on audiovisual fusion and tactile feedback, which is implemented based on the intelligent assisted blind navigation system proposed in the first aspect, and includes:

[0014] Real-time perception of surrounding environment information, receiving user voice commands, and sensing the user's push-pull force on the guide stick;

[0015] Constructing a surrounding environment map based on the received surrounding environment information and generating a planned path and a movement strategy in combination with user voice commands; the movement strategy includes the current required walking speed;

[0016] Generating walking prompt commands and drive commands in combination with the planned path and movement strategy, user voice commands and user push-pull forces, performing tactile vibration feedback and voice feedback according to the walking prompt commands, and driving the robotic guide dog for assisted blind navigation guidance according to the drive commands.

[0017] In a third aspect, the present invention further provides an electronic device, including: a memory for storing executable instructions; a processor for implementing the above-mentioned intelligent assisted blind navigation method when executing the executable instructions stored in the memory.

[0018] Fourthly, the present invention further provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-mentioned intelligent assisted blind guiding method based on audiovisual fusion and tactile feedback.

[0019] Fifthly, the present invention further provides a computer program product, which includes executable instructions stored in a computer-readable storage medium. When a processor of an electronic device reads and executes the executable instructions, the above-mentioned intelligent assisted blind guiding method based on audiovisual fusion and tactile feedback is implemented.

[0020] The above one or more technical solutions have the following beneficial effects:

[0021] 1. The present invention provides an intelligent assisted blind guiding system and method based on audiovisual fusion and tactile feedback. By integrating various sensor technologies, voice interaction technologies, and tactile feedback mechanisms, it can adapt to path planning, autonomous navigation, and real-time obstacle avoidance in various complex environments, and can provide highly intelligent, automated, and reliable travel assistance functions for visually impaired people, ensuring the travel safety of users and avoiding the problems of traditional assisted blind guiding devices that require manual operation, cannot adapt to complex dynamic environments, and cannot be adaptively adjusted.

[0022] 2. The intelligent assisted blind guiding system and method based on audiovisual fusion and tactile feedback proposed by the present invention innovatively proposes an audiovisual-tactile fusion method. Through the deep integration of multi-sensors and audiovisual-tactile modules, the system can perceive environmental changes in real time, dynamically adjust the navigation strategy, and provide multi-sensory navigation guidance for users through voice and tactile feedback. This fusion mechanism not only improves the intelligent level of the system, but also enhances the safety and user experience of users, solves the problems of long training cycle and poor environmental adaptability of traditional guide dogs, and provides an efficient, intelligent, and safe travel assistance solution for visually impaired people. In addition, the voice interaction function enables visually impaired people to interact with the robot through voice, reducing the dependence on other auxiliary devices; the tactile feedback mechanism further enhances the interactivity and safety of the system, enabling users to perceive environmental changes through touch; the design of the emergency stop and obstacle avoidance functions also further enhances the safety performance of the system and ensures the safety of users.

[0023] The advantages of the additional aspects of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings of the specification, which form a part of the present invention, are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not unduly limit the present invention.

[0025] Figure 1 It is a schematic diagram of the machine guide dog in the embodiment of the present invention;

[0026] Figure 2 It is a schematic diagram of the module design of the intelligent assisted guiding system in the embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of the guiding stick in the embodiment of the present invention;

[0028] Figure 4 It is a flowchart of the visual, auditory, and tactile three - aspect fusion mechanism in the embodiment of the present invention;

[0029] Figure 5 It is a flowchart of path planning and decision - making in the embodiment of the present invention.

[0030] Among them, 1. guiding stick; 2. folding joint; 3. lidar; 4. camera; 5. machine guide dog body; 6. quadruped drive device; 7. handle; 8. battery; 9. warning light; 10. traction connection device; 11. vibration device; 12. mechanical induction element; 13. blind stick rod. Detailed implementation manners

[0031] It should be noted that the following detailed description is exemplary only for describing the specific implementation manners, aiming to provide a further explanation of the present invention, and is not intended to limit the exemplary embodiments of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or their combinations.

[0032] Embodiment 1

[0033] This embodiment provides an intelligent assisted guiding system based on audiovisual fusion and tactile feedback. By integrating various sensor technologies, voice interaction technologies, and tactile feedback mechanisms, it can adapt to path planning, autonomous navigation, and real - time obstacle avoidance in various complex environments, and can provide highly intelligent, automated, and reliable travel assistance functions for visually impaired people, ensuring the travel safety of users and avoiding the problems of traditional assisted guiding devices that require manual operation, cannot adapt to complex dynamic environments, and cannot adaptively adjust for assistance, which may affect the safety of users.

[0034] Such as Figure 1As shown in the figure, the intelligent assisted guiding system proposed in this embodiment includes a robotic guide dog and a guiding cane 1. The end of the guiding cane 1 is connected to the autonomously mobile robotic guide dog. The guiding cane 1 can achieve universal rotation in space relative to the robotic guide dog. The robotic guide dog includes a robotic guide dog main body 5 (which can also be referred to as the chassis of the robotic navigation dog) and a quadruped drive device 6. The quadruped drive device 6 includes four legs for supporting the entire robotic guide dog main body and a drive module for driving the four legs to run, enabling the robotic guide dog to walk smoothly on various grounds and achieve obstacle avoidance and steering functions, while the robotic guide dog main body 5 is used to drive the entire system to operate. Among them, the robotic guide dog main body is made of high-strength aluminum alloy material. Through precision machining and structural optimization design, it is ensured to have sufficient strength and stability to carry the entire system. A quadruped drive device is installed below the main body, with drive wheels serving as the feet of the robotic guide dog. Each drive wheel is equipped with a high-performance DC motor. The motor adopts advanced vector control technology, which can accurately control the rotational speed and torque output. The drive wheel tires are made of special anti-slip and wear-resistant rubber material, and the surface is designed with unique patterns to enhance the grip and passability on different grounds (such as slippery tile floors, rough asphalt roads, soft grasslands, etc.). In addition, an advanced shock absorption system is integrated on the main body, including suspension springs and hydraulic shock absorbers, which can effectively filter out road bumps, provide guarantee for the smooth operation of the robot, and ensure that the measurement accuracy of sensors and the stability of the system will not be affected due to excessive vibration when driving on complex terrains.

[0035] Further, as Figure 2 shown, a vision module, a voice interaction module, a data processing module, and a control module are provided on the robotic guide dog main body, and a tactile feedback module is provided at the top handle of the guiding cane. Each module set in the system will be introduced in more detail through the following content.

[0036] (1) Vision module, used to perceive the surrounding environment information in real time.

[0037] The vision module includes a lidar (LiDAR), an ultrasonic sensor, a camera, and an inertial measurement unit (IMU), which are used to perceive the surrounding environment information in real time, such as static and dynamic obstacles, traffic signs, pedestrian postures, road boundaries, the speed and acceleration of the robotic guide dog itself, etc. According to this surrounding environment information, spatial positioning can be carried out, and a high-precision map of the surrounding environment can be modeled to ensure that users can walk safely and efficiently while avoiding collisions with obstacles.

[0038] Among them, the lidar 3 is installed on an adjustable turntable at the top of the main body of the robotic guide dog. Through precise mechanical structures and angle sensors, it can achieve 360-degree omnidirectional scanning. The lidar accurately measures the distance and contour information of surrounding objects by emitting laser beams and can work stably even in environments with low light or certain interference, providing key basic data for environmental perception and map construction.

[0039] The ultrasonic sensors are evenly distributed around the perimeter of the main body of the robotic guide dog to supplement the detection of small obstacles, low objects, or blind spots in the lidar scan in the short-distance range. The working frequency is optimized to avoid interference with the signals of other sensors and timely and accurately detect potential dangers around.

[0040] The camera 4 uses an industrial-grade camera with high resolution and wide dynamic range, which is installed at specific positions in the front and on both sides of the main body of the robotic guide dog. It has optical image stabilization and autofocus functions to clearly capture environmental images. The lens is made of special optical glass material with good light transmittance and anti-fouling properties, adapting to different lighting conditions (such as direct strong light, backlight, low light, etc.), providing rich visual data for deep learning algorithms to identify traffic signs, pedestrian postures, road boundaries, and other information.

[0041] The inertial measurement unit is tightly integrated at the core position inside the main body of the robotic guide dog. It real-time monitors the changes in the attitude, acceleration, and angular velocity of the robotic guide dog through high-precision accelerometers and gyroscopes. The sensor chip adopts advanced MEMS technology, featuring low noise and high sensitivity, ensuring accurate measurement of the motion state of the robotic guide dog during complex movements and providing key motion information for navigation and positioning.

[0042] (2) The voice interaction module is used to conduct voice interaction with the user, receive the user's voice commands, and provide voice navigation prompts according to the walking prompt commands.

[0043] The voice interaction module is used to conduct voice interaction with the user. It receives the voice commands input by the user through voice, such as voice navigation commands, and transfers the user's voice commands to the control module. The control module then adjusts the traveling state of the robot (i.e., the robotic guide dog) according to the commands. At the same time, the control module will generate walking prompt commands based on the actual navigation situation and provide real-time navigation prompts, as well as obstacle warnings, path adjustment information, etc., such as "Go forward", "Stop", "Turn left", "Turn right", "There is an obstacle 50 meters ahead, please detour" or "You have reached the destination", etc. In case of emergency, it will prompt the user about the current environmental conditions through voice, such as "The obstacle ahead is too close, please stop", so that the user can make corresponding adjustments.

[0044] Furthermore, the voice interaction module selects a high-sensitivity microphone array. By using beamforming technology, it enhances the reception effect of voice signals, effectively suppresses environmental noise interference, and can accurately recognize user voice commands in a noisy outdoor environment. Among them, the microphone array adopts a special acoustic structure design and signal processing algorithm, which can focus and enhance voice signals from different directions, improving the robustness of voice recognition. In addition, a speaker is provided in the voice interaction module. It uses a high-quality audio chip and a high-power amplifier to output clear, loud, and natural voice prompts. The volume can be automatically adjusted according to environmental noise to ensure that users can clearly hear navigation information and warnings in different scenarios. Moreover, the voice interaction module also supports multiple language and voice timbre selections, and users can make personalized settings according to their own needs.

[0045] Preferably, the voice interaction module can also be set at the handle of the blind cane 1 to obtain user voice commands and broadcast navigation prompts more clearly, enabling users to hear navigation information and warning information more clearly.

[0046] (3) The data processing module is used to construct a map of the surrounding environment based on the received surrounding environment information, and combine it with the user's voice commands to generate a planned path and a movement strategy. The movement strategy includes the current required walking speed.

[0047] In this embodiment, the data processing module is equipped with a large-capacity high-speed memory and a professional graphics processing unit (GPU), such as a high-performance GPU chip of NVIDIA, for accelerating operations; the storage device uses a solid-state drive (SSD), which has fast data read and write speeds and high reliability, meeting the fast storage and reading requirements of a large amount of map data, training models, and environmental information, and ensuring the timeliness of data processing and the system response speed; in addition, a dedicated operating system and software platform are also installed, such as the Robot Operating System ROS based on Linux, as well as related development tools and library files, facilitating algorithm development and system debugging.

[0048] The data processing module is used to process data from sensors, analyze the environmental situation, and generate a path planning scheme, with fast data processing capabilities to support real-time navigation and obstacle avoidance decisions; at the same time, the data processing module extracts information such as the number of pedestrians, pedestrian speed, and obstacle distance based on the current surrounding environment information provided by the vision module, and calculates the optimal movement strategy, that is, the recommended average safe walking speed that the user needs to adjust.

[0049] Specifically, use the multi-sensors in the above vision module to collect surrounding environment information, including the point cloud data of the lidar, the image data of the camera, the distance data of the ultrasonic sensor, and the motion data of the inertial measurement unit. Transmit the collected data to the data processing module, and perform real-time processing and fusion of these data by the data processing module for path planning, such asFigure 4 As shown in the figure, it includes:

[0050] First, use cameras, lidar, ultrasonic sensors, etc. to obtain real-time surrounding environment information. According to the obtained surrounding environment information, detect and identify the categories and states of the surrounding terrain, static and dynamic obstacles (such as pedestrians, vehicles, etc.), and the self-motion state of the machine guide dog in real time. Specifically, filter and segment the point cloud data collected by the lidar to remove noise points and invalid data, and divide the point cloud data into different object and terrain regions through a clustering algorithm; combine the texture and color information in the camera image to perform image recognition and semantic annotation on the segmented regions, such as identifying roads, buildings, pedestrians, vehicles, obstacles, etc.; use the motion data of the inertial measurement unit to accurately estimate and compensate the motion state of the robot to ensure the accuracy and stability of environmental perception.

[0051] Then, based on the processed data above, use deep learning algorithms to perform environmental analysis and construct a high-precision map model. The map model adopts a grid-based representation method, and the environmental information at the corresponding position, such as terrain type, obstacle distribution, passing probability, etc., is stored in each grid cell.

[0052] Finally, according to the constructed map model and the destination information specified based on the user's voice command, use an improved A* algorithm combined with deep reinforcement learning technology to perform environmental analysis and optimal path planning, and search for the optimal path from the current position to the destination on the map.

[0053] In this embodiment, the existing A* algorithm is improved by improving the evaluation function. Specifically, compared with the traditional evaluation function, this embodiment introduces a dynamic weight adjustment mechanism and a local replanning mechanism, and integrates user preferences to achieve better path planning. Among them, the evaluation function of the improved A* algorithm is:

[0054] f(n) = [g(n) + λ·T(n)] + [w1·h(n) + w2·D(n) + w3·C(n)];

[0055] Among them, g(n) represents the historical movement cost from the starting point to node n; T(n) represents the terrain traversability cost coefficient. The ground type is identified through lidar point cloud segmentation, and then combined with the attitude feedback data of the inertial measurement unit (IMU) for fusion calculation, which is used to quantify the difficulty of passing through terrains (such as grasslands, steps, slopes, etc.), and the quantified value is used as the terrain traversability cost coefficient; h(n) is the traditional heuristic function, which uses the Euclidean distance from node n to the target point; D(n) represents the dynamic obstacle density factor, which is based on the real-time data of ultrasonic sensors and cameras, and statistically calculates the obstacle distribution density within a preset radius around node n, and uses it as the dynamic obstacle density factor; C(n) represents the user preference coefficient, which is generated according to preset personalized requirements (such as quietness, flatness, avoidance areas, etc.) and calculated through semantic map matching; λ, w1, w2, and w3 are weight parameters, and the dynamic optimization of the weight parameters is performed through the deep reinforcement learning module to balance the priorities of terrain, obstacles, and user preferences.

[0056] Based on the above-set evaluation function, when the sensor detects environmental changes (such as sudden obstacles, terrain changes), the deep reinforcement learning module updates parameters such as λ and w2 in real time, and preferentially reduces the passing weights of high-risk areas, thereby achieving dynamic adjustment of the weights; when D(n) of a certain node in the current path exceeds the safety threshold, such as in a dense pedestrian area, local path re-search and planning can be triggered to avoid global repeated calculations; embed C(n) into the heuristic function. For example, when the user selects a "quiet route", the passing cost of noisy areas (which can be recognized through camera semantics) is automatically increased.

[0057] By adopting the above improved A* algorithm, during the search process, it is possible to dynamically select the travel route according to the current surrounding environment and user needs, that is, comprehensively consider the kinematic model of the robot, environmental obstacle information, and user preference settings, preferentially select paths with high safety and few obstacles, and at the same time consider user preferences, such as avoiding busy traffic areas and selecting quiet routes. Based on the above considerations, combined with the heuristic function, evaluate the costs of different paths, and preferentially select paths with lower costs, so as to achieve an optimal path planning that is more in line with the actual situation and user needs, further improve the path planning effect, and can effectively adapt to complex dynamic environments.

[0058] Furthermore, as Figure 5 shown, use dynamic programming techniques to record and optimize the searched paths to avoid repeated searches and falling into local optimal solutions. In addition, according to the actual situation, such as the real-time position of the robot and environmental changes, the planned path is adjusted in real time. When an obstacle or complex environmental change is detected, the robot can automatically select a new path to ensure avoiding obstacles, thereby ensuring that the robot can reach the destination safely and efficiently.

[0059] As another implementation, while building a map of the surrounding environment and performing path planning, this embodiment also formulates a movement strategy for the safe movement of the user according to the actual situation. The movement strategy includes the current required walking speed, that is, according to the environmental visual information captured by the camera, combined with the data collected by the lidar and ultrasonic sensors, the environmental sparsity and the maximum and minimum safe walking speeds of the user are calculated by the data processing module, etc.

[0060] Specifically, according to the obtained surrounding environment information, the surrounding terrain, the categories and states of static and dynamic obstacles (such as pedestrians, vehicles, etc.), and the self-movement state of the machine guide dog are detected and recognized in real time. On this basis, according to the real-time number of pedestrians, the pedestrian speed, and the distance between the machine guide dog and the nearest obstacle in the detected current scene, the environmental sparsity is calculated, and its calculation formula is:

[0061]

[0062] where N is the number of pedestrians detected in the current environment, V avg is the average speed of pedestrians in the current environment, which detects pedestrians in the environment through the camera and lidar, and calculates the speed v i of each pedestrian, and then the average speed of pedestrians in the crowd can be calculated through : d min is the distance between the machine guide dog and the nearest obstacle, and the distance d between the machine guide dog and the nearest obstacle can be detected by the lidar or ultrasonic sensor min ; k1, k2, and k3 are weight coefficients used to adjust the influence of the number of pedestrians, pedestrian speed, and obstacle distance on the environmental sparsity.

[0063] After that, according to the environmental sparsity, combined with the preset user reference walking speed, the current required walking speed, that is, the recommended walking speed for the user, is generated. Among them, the minimum safe speed V min and the maximum safe speed V max for the recommended user walking are calculated as:

[0064] V min = V base ·(1 - α·(1 - A));

[0065] V max = V base ·(1 + β·A);

[0066] where V base is the reference walking speed of the user, and α and β are adjustment coefficients used to control the range of the safe speed.

[0067] In addition, according to the self-movement state of the machine guide dog, the current real-time speed V current, and its calculation formula is:

[0068] V current = f × s × n;

[0069] Where, f is the step frequency, that is, the number of steps the robotic guide dog takes per second, s is the step length, that is, the distance of each step of the robotic guide dog; n is the gait coefficient, indicating the proportion of effective steps within a unit of time.

[0070] The above weight coefficients, adjustment coefficients, gait coefficients, etc. can all be set according to specific situations and user preferences. For example, set the weight of the number of pedestrians k1 = 0.3, the weight of the pedestrian speed k2 = 0.1, the weight of the obstacle distance k3 = 0.8, the adjustment coefficient α of the minimum safe speed = 0.15, the adjustment coefficient β of the maximum safe speed = 0.25, and the gait coefficient n = 0.5.

[0071] (4) The control module is used to generate a walking prompt instruction and a driving instruction for driving the quadruped driving device according to the planned path and the movement strategy, in combination with the user's voice command and the user's push-pull force.

[0072] The control module coordinates the work of each module, compares the path planning result with the user's current position, provides user guidance in combination with the voice interaction module and the tactile feedback module, and real-time feedbacks the traveling direction and the position of obstacles to the user through voice reminders and tactile feedback. At the same time, it executes emergency stop and obstacle avoidance decisions. When detecting an obstacle in front or a change in the complex environment, the intelligent robotic guide dog can automatically stop or adjust the path to avoid collisions and ensure that the robot can intelligently navigate to the destination.

[0073] In this embodiment, the control module selects a high-performance industrial-grade processor, such as a multi-core ARM processor or an Intel high-performance processor, which has powerful computing power and multi-task processing capabilities, can quickly process a large amount of data from sensors, and efficiently coordinates the work of each module; its hardware circuit design adopts multi-layer wiring and electromagnetic shielding technology to ensure the stability and anti-interference of signal transmission and ensure reliable operation in a complex electromagnetic environment. The control unit is also equipped with a large-capacity high-speed cache and memory, as well as high-speed communication interfaces, such as Ethernet interfaces, USB interfaces, and CAN bus interfaces, etc., for fast data transmission and communication with other modules.

[0074] (5) The tactile feedback module includes a vibration device and a mechanical induction element for sensing the user's push-pull force, and the vibration device is used for tactile vibration feedback according to the walking prompt instruction.

[0075] Such as Figure 3As shown in the figure, the guiding cane 1 is a traction guiding cane, which includes a cane rod 13, a handle 7 and a folding joint 2 are provided on the cane rod. The cane rod can be folded through the folding joint 2 for easy storage; a battery 8 and a warning light 9 are safely installed in the handle 7, and the warning light is powered by the battery for warning; a traction connection device 10 is provided at the tail of the cane rod, and it is connected to the main body of the robotic guide dog through the traction connection device 10; a tactile feedback module is also provided on the cane rod, and the tactile feedback module includes a vibration device 11 and a mechanical induction element 12, and tactile interaction with the user is carried out by sensing the pushing and pulling force of the user and vibration feedback.

[0076] In this embodiment, the tactile feedback module uses a strong and durable high-strength engineering plastic to build the cane traction device, ensuring stable structural performance in various usage scenarios and providing users with a reliable gripping experience. The design of the cane conforms to the ergonomic principle, and its surface is specially treated with anti-slip, which can not only effectively prevent the user from slipping during the gripping process, but also provide a comfortable touch for the user. At the same time, a high-precision mechanical induction element is integrated inside the cane. This element uses advanced piezoresistive sensing technology and can accurately sense the changes in the pushing and pulling forces applied by the user on the cane: when detecting the user's pushing force, according to the degree of change in the pushing force and combined with the preset acceleration rule, adjust the rotation speed of the drive motor in the drive command to adjust the smooth acceleration of the robotic guide dog; when detecting the user's pulling force, according to the degree of change in the pulling force and combined with the preset acceleration rule, adjust the rotation speed of the drive motor in the drive command to adjust the smooth deceleration of the robotic guide dog.

[0077] Specifically, when the user pushes the cane forward, the mechanical induction element will quickly capture the pressure change generated by this action and convert it into an electrical signal and transmit it to the control module. After receiving the electrical signal, the control module adjusts the rotation speed of the drive motor of the robotic guide dog according to the preset acceleration rule, so as to achieve the smooth acceleration of the robotic guide dog; similarly, when the user pulls the cane backward, the control module controls the drive motor to reduce the speed according to the signal fed back by the mechanical induction element, so that the robotic guide dog gradually decelerates, realizing the precise and flexible control of the traveling speed by the user. Among them, the preset acceleration rule actually reflects the relationship between the pushing and pulling forces applied by the user and the speed regulation, and it can be expressed by the formula:

[0078] Δv=k·sign(F)·|F| n ;

[0079] where F represents the pushing and pulling force applied by the user on the guiding cane, Δv represents the speed adjustment amount of the robotic guide dog; k is a proportionality coefficient, which can be adjusted according to the user's preference and system design; sign(F) is a sign function, indicating the direction of the pushing and pulling force, which is positive when pushing forward and negative when pulling backward; n is an exponent used to control the non-linear relationship between the pushing and pulling force and the speed adjustment.

[0080] In addition, the vibration device supports a multi-mode vibration feedback mechanism, which can deliver different tactile information to the user according to different environmental conditions and navigation requirements (prompting the user to accelerate, decelerate or stop through changes in vibration frequency and intensity). The dynamic adjustment function of the tactile module is based on the environmental sparsity and real-time speed of the user calculated above. The environmental sparsity is a function used to quantify the density of obstacles and pedestrians in the current environment, reflecting the complexity and congestion of the environment. The function calculates the distribution of obstacles and the number and speed of pedestrians in the environment in real time through sensor data (such as lidar, camera, etc.), and outputs a sparsity index. The higher the sparsity index, the emptier the environment and the fewer obstacles there are, while the lower the sparsity index, the more complex the environment, the more obstacles there are or the greater the pedestrian density. The environment sparsity function is used to dynamically adjust the vibration mode of the tactile feedback module, prompting the user to adjust the walking speed or take obstacle avoidance measures. When the environment is complex or the user needs to slow down, the vibration motor switches to a medium-frequency, medium-intensity and relatively long vibration mode, prompting the user to slow down; when the environment is sparse or the user needs to speed up, the vibration motor switches to a low-frequency, low-intensity and relatively long vibration mode, prompting the user to speed up; when the user's actual walking speed is within the safe speed range, the vibration motor remains stationary, thereby informing the user that the current walking status is normal.

[0081] Specifically, the vibration device installed on the cane uses a high-performance eccentric rotating mass (ERM) vibration motor, which has the characteristics of fast response speed, rich vibration patterns and precise adjustment. The vibration motor is closely connected to the robot's environmental perception module (i.e., visual module), path planning module (i.e., data processing module) and decision-making module (i.e., control module) through a specially designed control circuit. The control circuit uses an advanced microcontroller unit (MCU), which can accurately control the vibration frequency, intensity, duration and vibration pattern of the vibration motor according to the information transmitted by different modules.

[0082] In this embodiment, the walking prompt instruction includes a voice prompt instruction and a tactile feedback prompt instruction. The control module generates the voice prompt instruction and the tactile feedback prompt instruction according to the current planned path and movement strategy, combined with the obstacles identified in the current path, wherein:

[0083] (5.1) generating a tactile feedback instruction according to the currently detected obstacle, and the vibration device performs vibration feedback of set frequency, intensity, and time according to the tactile feedback prompt instruction, including:

[0084] When the environmental perception module detects an obstacle 1m ahead, the control circuit will quickly adjust the parameters of the vibration motor to make it work in a vibration mode with high frequency (5Hz), high intensity (80% of the maximum output power of the motor), and relatively long duration (10 seconds). Through strong vibrations, the user is promptly reminded that there is danger ahead and they need to stop moving immediately;

[0085] (5.2) Compare the current required walking speed with the user's real-time speed, and generate corresponding tactile feedback prompt instructions according to the comparison result. The vibration device performs vibration feedback according to the tactile feedback prompt instructions, including:

[0086] When the current speed V of the blind person current is higher than V max the vibration motor switches to a vibration mode with medium frequency (2Hz), medium intensity (50% of the maximum output power of the motor), and relatively long duration (5 seconds) to prompt the user to slow down;

[0087] When the current speed V of the blind person current is lower than V min the vibration motor switches to a vibration mode with low frequency (1Hz), low intensity (30% of the maximum output power of the motor), and relatively long duration (5 seconds) to prompt the user that they can accelerate;

[0088] When the actual walking speed of the user is within the interval of [V min , V max , the vibration motor remains stationary to inform the user that the current walking state is normal.

[0089] In addition, the vibration device also supports personalized settings. The user can flexibly adjust the intensity, frequency, and mode of vibration through the operation interface connected to the voice interaction module according to their own sensitivity and usage habits; the tactile feedback module also has a safety enhancement function. When it detects that the obstacle ahead is too close, the tactile module will use high-frequency combined vibrations to prompt the user to stop immediately. When the robot detects an emergency, the vibration device will, under the action of the control circuit, perform high-frequency vibrations at the highest frequency (10Hz) and the maximum intensity (100% of the maximum output power of the motor), and at the same time combine with the emergency voice prompt issued by the voice interaction module to issue an emergency warning to the user in a two-pronged manner to ensure that the user can make corresponding responses in the first time, greatly improving the safety of the system.

[0090] The system proposed in this embodiment can achieve:

[0091] Integration of vision and touch: The vision module captures dynamic information in the environment through cameras and radar sensors. The data processing module calculates the safe walking speed of the user and transmits this information to the touch module through the control module. The touch module adjusts the vibration frequency according to the safe walking speed of the user to prompt the user to adjust the walking speed. For example, when it detects that the speed of the crowd increases, it means that the environment is relatively more complex. At this time, if the comparison result shows that the current speed of the blind person is greater than the maximum safe speed, to ensure the safe and efficient walking of the user, the touch module will increase the vibration frequency according to the received comparison result information to prompt the user to slow down. When it is judged through the algorithm that the environment is relatively sparse, and if the comparison result shows that the current speed of the blind person is less than the minimum safe speed, the touch module will reduce the vibration frequency to prompt the user to speed up. In this way, the user can intuitively perceive the environmental changes through the tactile feedback and adjust their own walking speed in a timely manner.

[0092] Integration of hearing and touch: The voice interaction module provides real-time voice instructions to the user according to the path planning result, such as "turn left", "turn right", or "there are more pedestrians ahead, please slow down" and "there is an obstacle ahead". At the same time, the touch module synchronizes with the voice instructions through vibration feedback to enhance the user's perception ability. For example, when the voice prompts "there is an obstacle ahead", the touch module gives feedback with the corresponding vibration frequency and intensity according to the distance and danger level of the obstacle. When the obstacle is relatively close, it uses high-frequency and high-intensity vibration, enabling the user to obtain information from both the auditory and tactile aspects and respond to environmental changes more accurately.

[0093] Comprehensive feedback of vision, hearing, and touch: In a complex environment, the vision module, hearing module, and touch module work together to ensure that the user can receive navigation information through multiple senses. For example, when it detects an obstacle ahead, the vision module identifies the position of the obstacle, the voice module issues a warning, and the touch module prompts the user to stop or detour through high-frequency vibration. Taking a shopping mall as an example, the vision module uses cameras and lidar to real-time identify the positions of obstacles such as pedestrians and shelves ahead and transmits the information to other modules. The voice module issues warnings based on this information, such as "there are pedestrians ahead, please be careful to avoid" and "the shelf ahead is blocking the way, please detour". The touch module, according to the distance and danger level of the obstacle, uses high-frequency vibration to prompt the user to stop or detour. When the user approaches the shelf, the high-frequency vibration of the touch module reminds the user to stop in time, and at the same time, the voice prompt further clarifies the dangerous situation, enabling the user to make a quick response and effectively avoid collisions to ensure travel safety.

[0094] Through the above integration of vision and touch, integration of hearing and touch, and comprehensive feedback of vision, hearing, and touch, efficient, intelligent, and safe travel assistance can be achieved.

[0095] Embodiment 2

[0096] This embodiment provides an intelligent assisted blind guiding method based on audiovisual integration and tactile feedback, which is implemented based on the intelligent assisted blind guiding system proposed in Embodiment 1, and includes:

[0097] Perceive the surrounding environment information in real time, receive the user's voice commands, and sense the pushing and pulling forces of the user on the blind cane;

[0098] According to the received surrounding environment information, construct a surrounding environment map, and combine with the user's voice commands to generate a planned path and a movement strategy; this movement strategy includes the current required walking speed;

[0099] According to the planned path and movement strategy, combine with the user's voice commands and the user's pushing and pulling forces to generate walking prompt commands and driving commands, perform tactile vibration feedback and voice feedback according to the walking prompt commands, and drive the robotic guide dog to perform assisted blind guiding according to the driving commands.

[0100] Embodiment 3

[0101] This embodiment provides an electronic device, including: a memory for storing executable instructions; a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0102] Embodiment 4

[0103] This embodiment also provides a computer-readable storage medium storing executable instructions, which when executed by a processor, will cause the processor to execute the above method provided in this embodiment.

[0104] Embodiment 5

[0105] This embodiment provides a computer program product, which includes executable instructions, and the executable instructions are a kind of computer instructions; the executable instructions are stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and the processor executes the executable instructions, the electronic device is caused to execute the above method provided in this embodiment.

[0106] The steps involved in Embodiments 2 to 5 above correspond to those in Embodiment 1, and the specific implementation manners can be referred to the relevant description part of Embodiment 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to execute any method in the present invention.

[0107] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device. Thus, they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0108] The above are only the preferred embodiments of the present invention. Although the specific implementation manners of the present invention have been described in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. An intelligent assisted blind guiding system based on audio-visual fusion and tactile feedback, characterized in that, It includes a robotic guide dog and a guide cane. The robotic guide dog includes a robotic guide dog main body and a quadruped drive device. The following are provided on the robotic guide dog main body: A vision module for real-time perception of surrounding environment information; A voice interaction module for voice interaction with the user, receiving user voice commands and providing voice navigation prompts according to walking prompt commands; A data processing module for constructing a surrounding environment map based on the received surrounding environment information, and generating a planned path and a movement strategy in combination with user voice commands; the movement strategy includes the current required walking speed; A control module for generating walking prompt commands and drive commands for driving the quadruped drive device in combination with the planned path and movement strategy, user voice commands, and user pushing and pulling forces; A tactile feedback module is provided at the top handle of the guide cane. The tactile feedback module includes a vibration device and a mechanical induction element for sensing the user's pushing and pulling force. The vibration device is used for tactile vibration feedback according to the walking prompt command.

2. The intelligent assisted blind guiding system based on audiovisual integration and tactile feedback according to claim 1, characterized in that Generating a planned path based on the received surrounding environment information and in combination with user voice commands includes: Real-time detection and identification of the surrounding terrain, the types and states of static and dynamic obstacles, and the self-movement state of the robotic guide dog based on the obtained surrounding environment information; Performing environmental analysis and modeling a high-precision map based on the identified data; Based on the constructed map model and the destination information specified by the user voice command, using an improved A* algorithm combined with a deep reinforcement learning algorithm for environmental analysis and optimal path planning, and searching for the optimal path from the current position to the destination on the map; among them, the improved A* algorithm is an evaluation function that introduces a dynamic weight adjustment mechanism and a local replanning mechanism and integrates user preferences, and dynamically selects the optimal path according to the evaluation value; Making real-time adjustments to the planned path according to the real-time surrounding environment and user needs, and dynamically selecting and updating the travel route.

3. The intelligent assisted blind guiding system based on audiovisual fusion and tactile feedback according to claim 2, wherein The generation of the movement strategy includes: Calculating the environmental sparsity according to the detected number of pedestrians, pedestrian speed, and the distance between the robotic guide dog and the nearest obstacle in the current scene; at the same time, calculating the current real-time speed of the blind user according to the self-movement state of the robotic guide dog; Generating the current required walking speed, that is, the user-recommended walking speed, according to the environmental sparsity in combination with the preset user benchmark walking speed; the user-recommended walking speed includes the minimum safe speed and the maximum safe speed for the user to walk.

4. The intelligent assisted blind guiding system based on audiovisual fusion and tactile feedback according to claim 1, wherein The walking prompt commands include voice prompt commands and tactile feedback prompt commands. The control module generates voice prompt commands and tactile feedback prompt commands in combination with the current planned path and movement strategy and the obstacles identified in the current path, including: Generating voice prompt commands and tactile feedback commands according to the currently detected obstacles. The voice interaction module provides obstacle voice prompts according to the voice prompt commands, and the vibration device provides vibration feedback with a set frequency, intensity, and time according to the tactile feedback prompt commands; Compare the current required walking speed with the user's current real-time speed, generate tactile feedback prompt instructions with different vibration levels according to the comparison result, and the vibration device performs vibration feedback of the corresponding vibration level according to the tactile feedback prompt instructions, where different vibration levels include different vibration frequencies, intensities, and durations.

5. The intelligent assisted blind guiding system based on audiovisual fusion and tactile feedback according to claim 4, characterized in that, The vibration device performs vibration feedback of the corresponding vibration level according to the tactile feedback prompt instructions, including: When the user's current real-time speed is lower than the minimum safe speed, prompt to accelerate with low frequency, low intensity, and long duration vibration; When the user's current real-time speed is higher than the maximum safe speed, prompt to decelerate with medium frequency, medium intensity, and long duration vibration; When the user's current real-time speed is within the safe speed range, the vibration stops.

6. The intelligent assisted blind guiding system based on audiovisual fusion and tactile feedback according to claim 1, wherein, In the control module, according to the planned path and movement strategy, combined with the user's pushing and pulling forces, adjust the driving instruction to adjust the traveling speed of the robotic guide dog, including: When detecting the user's pushing force, according to the degree of change of the pushing force, combined with the preset acceleration rule, adjust the rotation speed of the driving motor in the driving instruction to adjust the smooth acceleration of the robotic guide dog; When detecting the user's pulling force, according to the degree of change of the pulling force, combined with the preset acceleration rule, adjust the rotation speed of the driving motor in the driving instruction to adjust the smooth deceleration of the robotic guide dog; wherein, the preset acceleration rule is used to reflect the relationship between the pushing and pulling forces applied by the user and the speed regulation.

7. An intelligent assisted blind guiding method based on audiovisual fusion and tactile feedback, characterized in that, Implemented based on the intelligent assisted guiding system for the blind based on audio-visual fusion and tactile feedback according to any one of claims 1-6, including: Perceive the surrounding environment information in real time, receive the user's voice instructions, and sense the pushing and pulling forces of the user on the guide stick; According to the received surrounding environment information, construct a map of the surrounding environment, and combined with the user's voice instructions, generate a planned path and a movement strategy; this movement strategy includes the current required walking speed; According to the planned path and movement strategy, combined with the user's voice instructions and the user's pushing and pulling forces, generate walking prompt instructions and driving instructions, perform tactile vibration feedback and voice feedback according to the walking prompt instructions, and drive the robotic guide dog to perform assisted guiding according to the driving instructions.

8. An electronic device, characterized in that, Including: A memory for storing executable instructions; A processor, when executing the executable instructions stored in the memory, implements the intelligent assisted guiding method for the blind based on audio-visual fusion and tactile feedback according to claim 7.

9. A computer-readable storage medium, characterized in that, Stores executable instructions, which are used to cause the processor to implement the intelligent assisted guiding method for the blind based on audio-visual fusion and tactile feedback according to claim 7 when executing the executable instructions.

10. A computer program product, characterized in that, The computer program product includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the intelligent assisted guiding method for the blind based on audio-visual fusion and tactile feedback according to claim 7.

Images