Intelligent auxiliary blind guiding system and method based on audio-visual fusion and tactile feedback

Abstract

The application discloses an intelligent auxiliary blind guiding system and method based on audio-visual fusion and tactile feedback, and relates to the technical field of intelligent auxiliary technology.The system comprises a machine guide dog and a guide stick.The machine guide dog comprises a machine guide dog main body and a four-foot driving device.A vision module, a voice interaction module, a data processing module and a control module are arranged on the machine guide dog main body.A tactile feedback module is arranged at the top handle of the guide stick.A surrounding environment map is constructed according to surrounding environment information, and a planning path and a moving strategy are generated in combination with a user voice instruction and a user push-pull force, so as to generate walking prompt instructions and driving instructions, drive the machine guide dog to guide and assist, and perform tactile vibration feedback according to the walking prompt instructions.The application can realize path planning, autonomous navigation and real-time obstacle avoidance in various complex environments, and realize highly intelligent, automatic and reliable travel assistance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent assistance, and in particular to an intelligent auxiliary guiding system and method based on audio-visual fusion and tactile feedback. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In recent years, with the continuous progress of technology, more and more intelligent assistance devices are used to assist visually impaired people to solve problems in life. Although the traditional guide dog for the blind as a long-term used auxiliary tool has played an important role, it also has some shortcomings that cannot be ignored: first, the training cost is high and the period is long, the guide dog for the blind needs to be trained for 2-3 years before it can be put into use, and the training cost of each guide dog is high, which depends on the physical condition of the animal and has a limited working period; second, the life management and working environment are limited, the guide dog needs regular feeding, health management and psychological training, and the guide dog can only be used in limited environment and is not suitable for all scenes; in addition, the traditional guide dog has poor environmental adaptability, and in complex urban environment, busy traffic or dynamic changing places, the working efficiency and reaction ability of the guide dog are limited, and there is a certain risk.

[0004] With the progress of science and technology, existing intelligent assistance devices such as intelligent guide stick and voice assistant have also been used by visually impaired people, but most of these devices rely on manual operation of the user and cannot adapt to complex dynamic environment, which has great limitations; the existing intelligent guide stick can only provide obstacle perception and simple navigation guidance, cannot realize automatic path planning and real-time obstacle avoidance, cannot realize adaptive assistance adjustment in different scenes, and cannot guarantee the safety of the user. SUMMARY

[0005] To solve the above problems of the prior art, the present application provides an intelligent auxiliary guiding system and method based on audio-visual fusion and tactile feedback, which can adapt to path planning, autonomous navigation and real-time obstacle avoidance in various complex environments by integrating various sensor technologies, voice interaction technologies and tactile feedback mechanisms, can provide highly intelligent, automated and reliable travel assistance functions for visually impaired people, guarantee the safety of the user in travel, and avoid the problems that the traditional auxiliary guiding device needs manual operation and cannot adapt to complex dynamic environment and cannot adaptively assist adjustment.

[0006] In a first aspect, the present application provides an intelligent auxiliary guiding system based on audio-visual fusion and tactile feedback.

[0007] The application discloses an intelligent auxiliary blind guiding system based on audio-visual fusion and tactile feedback, which comprises a machine guide dog and a guide stick.

[0008] A vision module is used for perceiving surrounding environment information in real time.

[0009] A voice interaction module is used for voice interaction with a user, receiving a user voice instruction and feeding back a voice navigation prompt according to a walking prompt instruction.

[0010] A data processing module is used for constructing a surrounding environment map according to the received surrounding environment information, and generating a planning path and a moving strategy in combination with the user voice instruction; the moving strategy comprises a current required walking speed.

[0011] A control module is used for generating a walking prompt instruction and a driving instruction for driving the four-foot driving device according to the planning path and the moving strategy in combination with the user voice instruction and the user push-pull force.

[0012] A tactile feedback module is arranged at a top handle of the guide stick, and the tactile feedback module comprises a vibration device and a mechanical sensing element for sensing the user push-pull force; the vibration device is used for tactile vibration feedback according to the walking prompt instruction.

[0013] In the second aspect, the application provides an intelligent auxiliary blind guiding method based on audio-visual fusion and tactile feedback, which is realized based on the intelligent auxiliary blind guiding system based on audio-visual fusion and tactile feedback in the first aspect, and comprises the following steps:

[0014] Surrounding environment information is perceived in real time, a user voice instruction is received, and a user push-pull force on the guide stick is sensed.

[0015] A surrounding environment map is constructed according to the received surrounding environment information, and a planning path and a moving strategy are generated in combination with the user voice instruction; the moving strategy comprises a current required walking speed.

[0016] A walking prompt instruction and a driving instruction are generated according to the planning path and the moving strategy in combination with the user voice instruction and the user push-pull force, tactile vibration feedback and voice feedback are performed according to the walking prompt instruction, and the machine guide dog is driven according to the driving instruction to guide the auxiliary blind guiding.

[0017] In the third aspect, the application further provides an electronic device, which comprises a memory for storing executable instructions and a processor for executing the executable instructions stored in the memory to realize the intelligent auxiliary blind guiding method based on audio-visual fusion and tactile feedback.

[0018] In a fourth aspect, the present application also provides a computer readable storage medium storing executable instructions for causing a processor to implement the above-mentioned intelligent auxiliary blind guiding method based on audio-visual fusion and tactile feedback when the processor executes the executable instructions.

[0019] In a fifth aspect, the present application also provides a computer program product comprising executable instructions stored in a computer readable storage medium, wherein a processor of an electronic device reads the executable instructions from the computer readable storage medium and executes the executable instructions to implement the above-mentioned intelligent auxiliary blind guiding method based on audio-visual fusion and tactile feedback.

[0020] The above one or more technical solutions have the following beneficial effects.

[0021] 1. The present application provides an intelligent auxiliary blind guiding system and method based on audio-visual fusion and tactile feedback, which can adapt to path planning, autonomous navigation and real-time obstacle avoidance in various complex environments by integrating various sensor technologies, voice interaction technologies and tactile feedback mechanisms, can provide highly intelligent, automated and reliable travel assistance functions for visually impaired people, ensure user travel safety, and avoid the problems that traditional auxiliary blind guiding devices need to be manually operated and cannot adapt to complex dynamic environments and cannot be adaptively adjusted.

[0022] 2. The intelligent auxiliary blind guiding system and method based on audio-visual fusion and tactile feedback proposed in the present application innovatively proposes an audio-visual tactile fusion method, which integrates multiple sensors and audio-visual tactile modules for deep fusion, so that the system can perceive environmental changes in real time, dynamically adjust navigation strategies, and provide multi-sensory navigation guidance for users through voice and tactile feedback. This fusion mechanism not only improves the intelligent level of the system, but also enhances the safety and user experience of the user, solves the problems of long training cycle and poor environmental adaptability of traditional guide dogs, and provides an efficient, intelligent and safe travel assistance solution for visually impaired people. In addition, the voice interaction function enables visually impaired people to interact with the robot through voice, reducing the dependence on other auxiliary equipment; the tactile feedback mechanism further enhances the interactivity and safety of the system, enabling users to perceive environmental changes through tactile sensation; the design of the emergency stop and obstacle avoidance function further enhances the safety performance of the system, ensuring user safety.

[0023] Advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings constituting a part of this disclosure serve to provide further understanding of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute undue limitations on the present application.

[0025] Figure 1 A schematic diagram of a machine guide dog in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a module design of an intelligent auxiliary guide dog system in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a guide dog in an embodiment of the present application;

[0028] Figure 4 A flowchart of a visual, auditory and tactile fusion mechanism in an embodiment of the present application;

[0029] Figure 5 A flowchart of path planning and decision-making in an embodiment of the present application.

[0030] Among them, 1, guide stick; 2, folding joint; 3, laser radar; 4, camera; 5, machine guide dog main body; 6, four-foot driving device; 7, handle; 8, battery; 9, warning light; 10, traction connecting device; 11, vibration device; 12, mechanical sensing element; 13, stick rod. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed description is exemplary only, is only for the purpose of describing specific embodiments, and is intended to provide further description of the present application, and is not intended to limit the exemplary embodiments according to the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in the present specification, they refer to the presence of a feature, step, operation, device, component and / or their combination.

[0032] Embodiment one

[0033] The present embodiment provides an intelligent auxiliary guide dog system based on audio-visual fusion and tactile feedback, which can adapt to path planning, autonomous navigation and real-time obstacle avoidance in various complex environments by integrating various sensor technologies, voice interaction technologies and tactile feedback mechanisms, can provide highly intelligent, automated and reliable travel assistance functions for visually impaired persons, ensure user travel safety, and avoid the problems that traditional auxiliary guide dog devices need to be manually operated, cannot adapt to complex dynamic environments, and cannot be self-adapted to assist adjustment to affect user safety.

[0034] As Figure 1As shown, the intelligent auxiliary blind guiding system proposed in the embodiment includes a machine guide dog and a guide stick 1, the guide stick 1 is connected to the machine guide dog which can move autonomously, the guide stick 1 can realize universal rotation in space relative to the machine guide dog, the machine guide dog includes a machine guide dog body 5 (also referred to as a machine navigation dog chassis) and a four-legged driving device 6, wherein the four-legged driving device 6 includes four legs for supporting the entire machine guide dog body and a driving module for driving the four legs to run, so that the machine guide dog can walk stably on various grounds and realize obstacle avoidance and turning functions, and the machine guide dog body 5 is used to drive the entire system to run. Among them, the machine guide dog body is made of high-strength aluminum alloy material, which is precisely machined and structurally optimized to ensure sufficient strength and stability to carry the entire system; the four-legged driving device is installed below the body, the driving wheel is used as the foot of the machine guide dog, each driving wheel is equipped with a high-performance DC motor, the motor adopts advanced vector control technology, which can accurately control the speed and torque output, and the driving wheel tire is made of special anti-skid and wear-resistant rubber material, and the surface is designed with unique patterns to enhance the grip and passability on different grounds (such as wet and slippery ceramic tile ground, rough asphalt pavement, soft grassland, etc.). In addition, the body is also integrated with an advanced damping system, including suspension springs and hydraulic shock absorbers, which can effectively filter road bumps and provide protection for the stable operation of the robot, ensuring that the measurement accuracy of the sensor and the stability of the system will not be affected by excessive vibration when driving on complex terrain.

[0035] Further, as shown in Figure 2 The machine guide dog body is provided with a vision module, a voice interaction module, a data processing module and a control module, and the top handle of the guide stick is provided with a tactile feedback module. The modules provided in the system will be introduced in more detail through the following contents.

[0036] (1) The vision module is used to perceive the surrounding environment information in real time.

[0037] The vision module includes a laser radar (LiDAR), an ultrasonic sensor, a camera and an inertial measurement unit (IMU), which is used to perceive the surrounding environment information in real time, such as static and dynamic obstacles, traffic signs, pedestrian posture, road boundaries, the speed and acceleration of the machine guide dog itself, etc. According to the surrounding environment information, the space can be positioned, and a high-precision map of the surrounding environment can be built to ensure that the user can walk safely and efficiently, and at the same time avoid collision with obstacles.

[0038] The laser radar 3 is installed on the adjustable holder on the top of the machine guide dog body. Through the precise mechanical structure and angle sensor, 360-degree omnidirectional scanning is realized. The laser radar can accurately measure the distance and contour information of the surrounding objects by emitting laser beams, and can work stably even in the environment with relatively dark light or certain interference, thereby providing key basic data for environment perception and map construction.

[0039] The ultrasonic sensor is evenly distributed around the machine guide dog body. In the short distance range, it can supplement the detection of small obstacles, low objects or blind areas of laser radar scanning, optimize the working frequency, avoid interference with other sensor signals, and timely and accurately detect potential dangers around.

[0040] The camera 4 adopts an industrial-grade camera with high resolution and wide dynamic range, which is installed at specific positions in front of and on both sides of the machine guide dog body. It has optical image stabilization and automatic focusing functions, and can clearly capture environmental images. The lens is made of special optical glass material, which has good light transmission and anti-fouling properties, and can adapt to different lighting conditions (such as strong direct light, backlight, weak light, etc.). It provides rich visual data for deep learning algorithms, and is used to identify traffic signs, pedestrian posture, road boundaries, etc.

[0041] The inertial measurement unit is tightly integrated into the core position of the machine guide dog body. It can monitor the attitude, acceleration and angular velocity changes of the machine guide dog in real time through high-precision accelerometers and gyroscopes. The sensor chip uses advanced MEMS technology, has low noise and high sensitivity, and can accurately measure the motion state of the machine guide dog during complex motion, providing key motion information for navigation and positioning.

[0042] (2) The voice interaction module is used for voice interaction with the user, receives the user's voice instructions, and feeds back the voice navigation prompts according to the walking prompt instructions.

[0043] The voice interaction module is used for voice interaction with the user, which receives the voice instructions input by the user through voice, such as voice navigation instructions, and transmits the user's voice instructions to the control module. The control module adjusts the advancing state of the robot (i.e. the machine guide dog) according to the instructions. At the same time, the control module generates walking prompt instructions according to the actual navigation situation, provides real-time navigation prompts, obstacle warning, path adjustment information, etc. through voice feedback, such as "go forward", "stop", "turn left", "turn right", "there is an obstacle 50 meters ahead, please detour" or "you have arrived at the destination", etc. In emergency situations, the user is prompted through voice about the current environmental conditions, such as "the obstacle in front is too close, please stop", so that the user can make corresponding adjustments.

[0044] Further, the voice interaction module selects a high-sensitivity microphone array, enhances the voice signal receiving effect through beamforming technology, effectively suppresses environmental noise interference, and can accurately identify user voice instructions in a noisy outdoor environment. Among them, the microphone array adopts a special acoustic structure design and signal processing algorithm, which can focus and enhance voice signals in different directions, improving the robustness of voice recognition. In addition, the voice interaction module is provided with a loudspeaker, which uses a high-quality audio chip and a high-power power amplifier to output clear, loud and natural voice prompts. The volume can be automatically adjusted according to the environmental noise to ensure that the user can clearly hear the navigation information and warnings in different scenarios. In addition, the voice interaction module also supports multiple languages and voice tone selection, and users can make personalized settings according to their needs.

[0045] Preferably, the voice interaction module can also be provided at the handle of the blind walking stick 1 to more clearly obtain user voice instructions and broadcast navigation prompts, so that the user can more clearly hear the navigation information and warning information.

[0046] (3) Data processing module, for constructing the surrounding environment map according to the received surrounding environment information, and generating a planned path and a moving strategy in combination with the user voice instruction, the moving strategy including a current required walking speed.

[0047] In this embodiment, the data processing module is equipped with a large-capacity high-speed memory and a professional graphics processing unit (GPU), such as a high-performance GPU chip of NVIDIA, for accelerating computation; the storage device adopts a solid state drive (SSD) with fast data read / write speed and high reliability, meeting the fast storage and reading requirements of a large amount of map data, training models and environmental information, and ensuring the timeliness of data processing and system response speed; in addition, a special operating system and software platform are also installed, such as a robot operating system ROS based on Linux, and related development tools and library files, facilitating algorithm development and system debugging.

[0048] The data processing module is used to process data from sensors, analyze environmental conditions and generate path planning schemes, and has fast data processing capability to support real-time navigation and obstacle avoidance decision-making; at the same time, the data processing module extracts information such as the number of pedestrians, pedestrian speed and obstacle distance according to the current surrounding environment information provided by the vision module, and calculates the optimal moving strategy, i.e. the average safe walking speed required by the user to adjust.

[0049] Specifically, the surrounding environment information is collected by the multi-sensor in the above-mentioned vision module, including the point cloud data of the laser radar, the image data of the camera, the distance data of the ultrasonic sensor, and the motion data of the inertial measurement unit, and the collected data is transmitted to the data processing module. The data processing module processes and fuses these data in real time to plan a path, such asFigure 4 As shown, comprising:

[0050] First, real-time surrounding environment information is acquired by using a camera, a laser radar, an ultrasonic sensor, etc., and according to the acquired surrounding environment information, the surrounding terrain, the categories and states of static and dynamic obstacles (such as pedestrians, vehicles, etc.), the motion state of the machine guide dog itself, etc. are detected and recognized in real time. Specifically, the point cloud data collected by the laser radar is filtered and segmented to remove noise points and invalid data, and the point cloud data is divided into different objects and terrain areas by a clustering algorithm; the segmented areas are image-recognized and semantically labeled in combination with the texture and color information in the camera image, such as recognizing roads, buildings, pedestrians, vehicles, obstacles, etc.; the motion data of the inertial measurement unit is used to accurately estimate and compensate the motion state of the robot, to ensure the accuracy and stability of the environment perception.

[0051] Then, based on the above-processed data, a deep learning algorithm is used for environment analysis to construct a high-precision map model. The map model adopts a grid-based representation method, and each grid cell stores the environmental information of the corresponding position, such as terrain type, obstacle distribution, traffic probability, etc.

[0052] Finally, according to the constructed map model and the destination information specified based on the user's voice instruction, an improved A* algorithm combined with deep reinforcement learning technology is used for environment analysis and optimal path planning to search for the optimal path from the current position to the destination on the map.

[0053] In this embodiment, the existing A* algorithm is improved by improving the evaluation function. Specifically, compared with the traditional evaluation function, the dynamic weight adjustment mechanism and the local re-planning mechanism are introduced in this embodiment, and the user preference is fused to realize better path planning, wherein the evaluation function of the improved A* algorithm is:

[0054] f(n)=[g(n)+λ·T(n)]+[w1·h(n)+w2·D(n)+w3·C(n)];

[0055] wherein g(n) represents the historical movement cost from the starting point to node n; T(n) represents the terrain passing cost coefficient, which is calculated by fusing the ground type identified by laser point cloud segmentation and the attitude feedback data of the inertial measurement unit (IMU), for quantifying the passing difficulty of the terrain (such as grassland, steps, slopes, etc.), and taking the quantized value as the terrain passing cost coefficient; h(n) is a traditional heuristic function, which adopts the Euclidean distance from node n to the target point; D(n) represents the dynamic obstacle density factor, which is calculated based on the real-time data of the ultrasonic sensor and the camera, and the obstacle distribution density within a preset radius around node n is taken as the dynamic obstacle density factor; C(n) represents the user preference coefficient, which is generated according to the preset personalized requirements (such as quietness, flatness, avoidance area, etc.) and calculated through semantic map matching; and λ, w1, w2 and w3 are weight parameters, which are dynamically optimized by the deep reinforcement learning module to balance the priority of terrain, obstacle and user preference.

[0056] Based on the above setting of the evaluation function, when the sensor detects an environmental change (such as a sudden obstacle or terrain change), the deep reinforcement learning module updates the parameters λ, w2, etc. in real time to preferentially reduce the passing weight of high-risk areas, thereby achieving dynamic adjustment of the weight; if D(n) of a certain node in the current path exceeds the safety threshold, such as a dense pedestrian area, local path re-search and planning can be triggered to avoid global repeated calculation; and C(n) is embedded in the heuristic function, for example, when the user selects a "quiet route", the passing cost of a noisy area (which can be identified by a camera) is automatically increased.

[0057] By using the improved A* algorithm described above, the robot can dynamically select a travel route according to the current surrounding environment and user requirements during the search process, i.e., by comprehensively considering the kinematic model of the robot, environmental obstacle information and user preference settings, the robot can preferentially select a path with high safety and few obstacles, while considering the user's preferences, such as avoiding busy traffic areas and selecting a quiet route, etc. Based on the above considerations, the heuristic function is used to evaluate the cost of different paths, and a path with lower cost is preferentially selected. In this way, the optimal path planning that is more in line with the actual situation and user requirements can be achieved, and the path planning effect is further improved, which can effectively adapt to complex dynamic environments.

[0058] Further, as shown in Figure 5 In addition, according to the actual situation, such as the real-time position of the robot and environmental changes, the planned path is adjusted in real time, and when an obstacle or complex environmental change is detected, the robot can automatically select a new path to ensure that the obstacle is avoided, thereby ensuring that the robot can safely and efficiently reach the destination.

[0059] As another implementation, while constructing the surrounding environment map and performing path planning, the embodiment also formulates a movement strategy for the user to move safely according to the actual situation, which includes the current required walking speed, i.e. according to the environmental visual information captured by the camera, in combination with the data collected by the laser radar and ultrasonic sensor, the environmental sparsity and the maximum and minimum safe speed of the user walking are calculated by the data processing module.

[0060] Specifically, according to the acquired surrounding environment information, the surrounding terrain, the category and state of static and dynamic obstacles (such as pedestrians, vehicles, etc.), and the motion state of the machine guide dog are detected and recognized in real time, and on this basis, according to the real-time number of pedestrians in the current scene, the speed of the pedestrians, the distance between the machine guide dog and the nearest obstacle, the environmental sparsity is calculated, and the calculation formula is:

[0061]

[0062] Among them, N is the number of pedestrians detected in the current environment, V avg is the average speed of the pedestrians in the current environment, which detects the pedestrians in the environment through the camera and laser radar, and calculates the speed v i of each pedestrian, and then the average speed of the pedestrians in the crowd can be calculated through d min is the distance between the machine guide dog and the nearest obstacle, which can be detected by the laser radar or ultrasonic sensor. min ; k1, k2, k3 are weight coefficients for adjusting the influence of the number of pedestrians, the speed of pedestrians and the distance of obstacles on the environmental sparsity.

[0063] Then, according to the environmental sparsity, in combination with the preset user reference walking speed, the current required walking speed, i.e. the user recommended walking speed, is generated. Among them, the minimum safe speed V min and the maximum safe speed V max recommended for the user to walk are:

[0064] V min =V base ·(1-α·(1-A));

[0065] V max =V base ·(1+β·A);

[0066] Among them, V base is the user's reference walking speed, and α and β are adjustment coefficients for controlling the range of safe speed.

[0067] In addition, according to the motion state of the machine guide dog itself, the current real-time speed V currentThe calculation formula is:

[0068] V current = f x s x n;

[0069] Wherein, f is the step frequency, that is, the number of steps taken by the machine guide dog per second, s is the step length, that is, the distance of each step of the machine guide dog; n is the gait coefficient, which represents the proportion of effective steps in unit time.

[0070] The above weight coefficient, adjustment coefficient, gait coefficient and the like can be set according to specific circumstances and user preferences, such as setting the weight of the number of pedestrians k1 = 0.3, the weight of the speed of the pedestrians k2 = 0.1, the weight of the distance of the obstacles k3 = 0.8, the adjustment coefficient of the minimum safe speed a = 0.15, the adjustment coefficient of the maximum safe speed b = 0.25, and the gait coefficient n = 0.5.

[0071] (4) A control module for generating walking prompt instructions and driving instructions for driving the four-legged driving device according to the planned path and the movement strategy in combination with the user voice instructions and the user push-pull force.

[0072] The control module coordinates the work of each module, compares the path planning result with the current position of the user, provides user guidance in combination with the voice interaction module and the tactile feedback module, and provides real-time feedback of the advancing direction and the obstacle position to the user through voice reminders and tactile feedback, while executing emergency stop and obstacle avoidance decisions. When detecting the front obstacle or complex environmental changes, the intelligent machine guide dog can automatically stop or adjust the path to avoid collision and ensure that the robot can intelligently navigate to the destination.

[0073] In the embodiment, the control module selects a high-performance industrial processor, such as a multi-core ARM processor or an Intel high-performance processor, which has powerful computing power and multi-task processing capability, can quickly process a large amount of data from the sensor, and efficiently coordinates the work of each module; the hardware circuit design adopts multi-layer wiring and electromagnetic shielding technology to ensure the stability and anti-interference of signal transmission, and ensure reliable operation in complex electromagnetic environment. The control unit is also equipped with a large-capacity cache and memory, as well as high-speed communication interfaces such as Ethernet interface, USB interface and CAN bus interface, etc., for fast data transmission and communication with other modules.

[0074] (5) A tactile feedback module, the tactile feedback module comprising a vibration device and a mechanical sensing element for sensing the user's push-pull force, the vibration device being used for tactile vibration feedback according to the walking prompt instructions.

[0075] For example, Figure 3As shown, the guide stick 1 is a traction stick, including a stick rod 13, a handle 7 and a folding joint 2 are arranged on the stick rod, the stick rod can be folded through the folding joint 2, which is convenient for storage; the handle 7 has a battery 8 and a warning light 9, the battery supplies power to the warning light, and the warning light warns; the tail of the stick rod is provided with a traction connecting device 10, which is connected with the main body of the machine guide dog through the traction connecting device 10; the stick rod is also provided with a tactile feedback module, which includes a vibration device 11 and a mechanical sensing element 12, which interacts with the user through tactile feedback of sensing user push-pull force and vibration.

[0076] In this embodiment, the tactile feedback module is made of a solid and durable high-strength engineering plastic to make the guide stick traction device, which can maintain stable structural performance in various use scenarios and provide users with reliable holding experience. The design of the guide stick conforms to the principle of human engineering, and its surface is treated with special anti-slip treatment, which can effectively prevent the user from slipping during holding, and also provides a comfortable touch for the user. At the same time, the guide stick is integrated with high-precision mechanical sensing elements, which use advanced piezoresistive sensing technology to accurately sense the changes of the user's push and pull force on the guide stick. When detecting the user's push force, according to the degree of change of the push force, combined with the preset acceleration rule, the speed of the drive motor in the drive instruction is adjusted to adjust the smooth acceleration of the machine guide dog; when detecting the user's pull force, according to the degree of change of the pull force, combined with the preset acceleration rule, the speed of the drive motor in the drive instruction is adjusted to adjust the smooth deceleration of the machine guide dog.

[0077] Specifically, when the user pushes the guide stick forward, the mechanical sensing element will quickly capture the pressure change caused by this action and convert it into an electrical signal to the control module. After receiving the electrical signal, the control module adjusts the speed of the machine guide dog drive motor according to the preset acceleration rule, so as to realize the smooth acceleration of the machine guide dog; similarly, when the user pulls the guide stick backward, the control module controls the drive motor to reduce the speed according to the signal feedback by the mechanical sensing element, so as to gradually decelerate the machine guide dog, realizing the accurate and flexible control of the user on the speed of travel. The preset acceleration rule actually reflects the relationship between the push-pull force applied by the user and the speed control, which can be expressed by the formula:

[0078] Δv=k·sign(F)·|F| n ;

[0079] Where F represents the push-pull force applied by the user on the guide stick, Δv represents the speed adjustment of the machine guide dog; k is a proportional coefficient, which can be adjusted according to the user's preference and system design; sign(F) is a sign function, which represents the direction of the push-pull force, and is positive when pushing forward and negative when pulling backward; n is an index, which is used to control the non-linear relationship between the push-pull force and the speed adjustment.

[0080] In addition, the vibration device supports a multi-mode vibration feedback mechanism, which can deliver different tactile information to the user (prompting the user to accelerate, decelerate, or stop through changes in vibration frequency and intensity) according to different environmental conditions and navigation needs. The dynamic adjustment function of the tactile module is based on the above-mentioned calculated environmental sparsity and real-time user speed, where the environmental sparsity is a function for quantifying the density of obstacles and pedestrians in the current environment, reflecting the complexity and congestion of the environment. The function calculates the distribution of obstacles and the number and speed of pedestrians in the environment in real time through sensor data (such as lidar, camera, etc.), outputs a sparsity index, and the higher the sparsity index, the more open and less obstacles the environment, and the lower the sparsity index, the more complex the environment, the more obstacles or the greater the pedestrian density. The environmental sparsity function is used to dynamically adjust the vibration mode of the tactile feedback module, prompting the user to adjust the walking speed or take obstacle avoidance measures. When the environment is complex or the user needs to slow down, the vibration motor switches to a medium-frequency, medium-intensity, and relatively long-time vibration mode, prompting the user to slow down; when the environment is sparse or the user needs to speed up, the vibration motor switches to a low-frequency, low-intensity, and relatively long-time vibration mode, prompting the user to speed up; when the user's actual walking speed is within the safe speed interval, the vibration motor remains stationary, informing the user that the current walking state is normal.

[0081] Specifically, the vibration device installed on the blind stick selects a high-performance eccentric rotating mass (ERM) vibration motor, which has the characteristics of fast response speed, rich vibration modes, and accurate adjustment. The vibration motor is closely connected to the robot's environmental perception module (i.e., the vision module), path planning module (i.e., data processing module), and decision module (i.e., control module) through a specially designed control circuit. The control circuit uses an advanced microcontroller unit (MCU) that can accurately control the vibration frequency, intensity, duration, and vibration mode of the vibration motor based on information from different modules.

[0082] In this embodiment, the walking prompt instruction includes a voice prompt instruction and a tactile feedback prompt instruction. The control module generates a voice prompt instruction and a tactile feedback prompt instruction based on the current planned path and movement strategy, combined with the obstacles identified in the current path, where:

[0083] (5.1) According to the currently detected obstacles, generate a tactile feedback instruction, and the vibration device sets the frequency, intensity, and time of vibration feedback according to the tactile feedback prompt instruction, including:

[0084] When the environmental perception module detects an obstacle 1m ahead, the control circuit quickly adjusts the parameters of the vibration motor to work in a high-frequency (5Hz), high-intensity (80% of the maximum output power of the motor) and relatively long-time (10 seconds) vibration mode, reminding the user in time of the danger ahead and the need to stop immediately through strong vibration;

[0085] (5.2) Compare the current required walking speed with the real-time speed of the user, and generate corresponding tactile feedback prompt instructions according to the comparison result. The vibration device vibrates according to the tactile feedback prompt instructions, including:

[0086] When the current speed V current of the blind person is higher than V max , the vibration motor switches to a medium-frequency (2Hz), medium-intensity (50% of the maximum output power of the motor) and relatively long-time (5 seconds) vibration mode, prompting the user to slow down;

[0087] When the current speed V current of the blind person is lower than V min , the vibration motor switches to a low-frequency (1Hz), low-intensity (30% of the maximum output power of the motor) and relatively long-time (5 seconds) vibration mode, prompting the user to speed up;

[0088] When the actual walking speed of the user is in the interval [V min , V max ], the vibration motor remains stationary, indicating that the current walking state is normal.

[0089] In addition, the vibration device also supports personalized settings. The user can adjust the intensity, frequency and mode of vibration through the operation interface connected to the voice interaction module according to their own sensitivity and usage habits. The tactile feedback module also has a safety enhancement function. When it detects that the obstacle ahead is too close, the tactile module will vibrate in high-frequency combination to prompt the user to stop immediately. When the robot detects an emergency, the vibration device will vibrate at the highest frequency (10Hz) and maximum intensity (100% of the maximum output power of the motor) under the action of the control circuit, and combine with the emergency voice prompt from the voice interaction module to issue an emergency warning to the user, ensuring that the user can respond in the first time, greatly improving the safety of the system.

[0090] The system proposed in this embodiment can achieve:

[0091] The fusion of vision and touch: the visual module captures dynamic information in the environment through cameras and radar sensors, the data processing module calculates the safe walking speed of the user and transmits this information to the tactile module through the control module, and the tactile module adjusts the vibration frequency according to the safe walking speed of the user, prompting the user to adjust the walking speed. For example, when detecting that the crowd speed is increasing, it means that the environment is relatively more complex. At this time, if the comparison result is that the current speed of the blind person is greater than the maximum safe speed, in order to ensure the safety and efficiency of the user's walking, the tactile module will increase the vibration frequency according to the received comparison result information, prompting the user to slow down. When the algorithm judges that the environment is relatively sparse, if the comparison result is that the current speed of the blind person is less than the minimum safe speed, the tactile module will reduce the vibration frequency, prompting the user to speed up. In this way, the user can intuitively perceive the changes in the environment through tactile feedback and adjust their walking speed in a timely manner.

[0092] The fusion of hearing and touch: the voice interaction module provides real-time voice instructions to the user according to the path planning results, such as "turn left", "turn right", or "there are many pedestrians in front, please slow down", "there is an obstacle in front". At the same time, the tactile module provides feedback through vibration in synchronization with the voice instructions, enhancing the user's perception ability. For example, when the voice prompt is "there is an obstacle in front", the tactile module provides feedback with corresponding vibration frequency and intensity according to the distance and danger level of the obstacle, such as high-frequency and high-intensity vibration when the obstacle is close, allowing the user to obtain information from both hearing and touch, and more accurately respond to environmental changes.

[0093] Comprehensive feedback of audio-visual-touch: in complex environments, the visual module, the hearing module, and the tactile module work together to ensure that the user can receive navigation information through multiple senses. For example, when detecting an obstacle in front, the visual module identifies the position of the obstacle, the voice module issues a warning, and the tactile module prompts the user to stop or detour through high-frequency vibration. Taking a shopping mall as an example, the visual module identifies the positions of pedestrians, shelves, and other obstacles in front in real time through cameras and laser radars, and transmits the information to other modules; the voice module issues warnings based on this information, such as "be careful to avoid pedestrians in front", "detour around the shelf in front"; the tactile module prompts the user to stop or detour through high-frequency vibration according to the distance and danger level of the obstacle; when the user approaches the shelf, the tactile module prompts the user to stop in time through high-frequency vibration, and the voice prompt further clarifies the danger situation, allowing the user to respond quickly and effectively avoid collisions, ensuring safe travel.

[0094] Through the above fusion of vision and touch, fusion of hearing and touch, and comprehensive feedback of audio-visual-touch, efficient, intelligent, and safe travel assistance can be achieved.

[0095] Embodiment Two

[0096] The embodiment provides an intelligent auxiliary blind guiding method based on audio-visual fusion and tactile feedback, and is realized based on the intelligent auxiliary blind guiding system based on audio-visual fusion and tactile feedback provided in embodiment one, and comprises the following steps.

[0097] Real-time sensing of surrounding environment information, receiving of user voice instructions, and sensing of user pushing and pulling force on the blind guiding stick;

[0098] According to the received surrounding environment information, a surrounding environment map is constructed, and a planning path and a moving strategy are generated in combination with the user voice instructions; the moving strategy comprises a current required walking speed;

[0099] According to the planning path and the moving strategy, in combination with the user voice instructions and the user pushing and pulling force, walking prompt instructions and driving instructions are generated, tactile vibration feedback and voice feedback are performed according to the walking prompt instructions, and the machine guide dog is driven to perform auxiliary blind guiding according to the driving instructions.

[0100] Embodiment three

[0101] The embodiment provides an electronic device, comprising a memory for storing executable instructions, and a processor for executing the executable instructions stored in the memory to realize the above method provided in the embodiment.

[0102] Embodiment four

[0103] The embodiment also provides a computer readable storage medium storing executable instructions, which, when executed by a processor, causes the processor to execute the above method provided in the embodiment.

[0104] Embodiment five

[0105] The embodiment provides a computer program product, which comprises executable instructions, and the executable instructions are computer instructions; the executable instructions are stored in a computer readable storage medium. When the processor of the electronic device reads the executable instructions from the computer readable storage medium, the processor executes the executable instructions, so that the electronic device executes the above method provided in the embodiment.

[0106] The steps involved in the above embodiments two to five correspond to embodiment one, and the specific embodiments can be referred to the related description part of embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to execute any method in the present application.

[0107] Those skilled in the art should understand that the modules or steps of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0108] The above only describes the preferred embodiments of the present application, and the specific embodiments of the present application are described in conjunction with the drawings, but are not limited to the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. An intelligent assistive navigation system based on audio-visual fusion and haptic feedback, characterized in that, The machine guide dog includes a machine guide dog body and a four-legged driving device, and the machine guide dog body is provided with: a visual module for real-time perception of surrounding environment information; a voice interaction module for voice interaction with the user, receiving user voice instructions and feeding back voice navigation prompts according to walking prompt instructions; a data processing module for constructing a surrounding environment map according to the received surrounding environment information, and generating a planned path and a movement strategy in combination with user voice instructions; the movement strategy includes a current required walking speed; a control module for generating walking prompt instructions and driving instructions for driving the four-legged driving device according to the planned path and the movement strategy in combination with user voice instructions and user push-pull force; a haptic feedback module is arranged at a top handle of the guide stick, and the haptic feedback module includes a vibration device and a mechanical sensing element for sensing user push-pull force, and the vibration device is used for haptic vibration feedback according to the walking prompt instructions; The generation of the movement strategy includes: calculating the environmental sparsity according to the detected real-time number of pedestrians, the pedestrian speed, and the distance between the machine guide dog and the nearest obstacle in the current scene, and the calculation formula is: ； wherein, is the number of pedestrians detected in the current environment, is the average speed of pedestrians in the current environment, the pedestrians in the environment are detected by the camera and laser radar, and the speed of each pedestrian is calculated , and then the average speed of pedestrians in the crowd is calculated by is the distance between the machine guide dog and the nearest obstacle, which is detected by laser radar or ultrasonic sensor; 、 、 is a weight coefficient, used to adjust the influence of the number of pedestrians, the speed of pedestrians and the distance of obstacles on the sparsity of the environment;​ At the same time, the real-time speed of the blind user is calculated according to the motion state of the machine guide dog itself; According to the environmental sparsity, in combination with a preset user reference walking speed, a current required walking speed, i.e., a user suggested walking speed, is generated; wherein the user suggested walking speed comprises a minimum safe speed and a maximum safe speed for suggesting the user to walk . . ； ； wherein, is a reference walking speed of the user; and is an adjustment coefficient for controlling the range of the safety speed.

2. The intelligent assistive blind guidance system based on audio-visual fusion with tactile feedback as claimed in claim 1 wherein, constructing a surrounding environment map according to the received surrounding environment information, and generating a planned path in combination with user voice instructions, including: real-time detection and identification of the surrounding terrain, the category and state of static and dynamic obstacles, and the motion state of the machine guide dog itself according to the acquired surrounding environment information; environmental analysis and modeling of a high-precision map according to the identified data; adopting an improved A* algorithm combined with a deep reinforcement learning algorithm to perform environmental analysis and optimal path planning according to the constructed map model and the destination information specified based on user voice instructions, and searching for an optimal path from the current position to the destination on the map; wherein the improved A* algorithm is an evaluation function that introduces a dynamic weight adjustment mechanism and a local re-planning mechanism and fuses user preferences, and dynamically selects an optimal path according to the evaluation value; real-time adjustment of the planned path according to the real-time surrounding environment and user demand, and dynamic selection and update of the travel route.

3. The intelligent assistive blind guidance system based on audio-visual fusion with tactile feedback as claimed in claim 1 wherein, The walking prompt instructions include voice prompt instructions and haptic feedback prompt instructions, and the control module generates the voice prompt instructions and the haptic feedback prompt instructions according to the current planned path and the movement strategy in combination with the identified obstacles in the current path, including: generating voice prompt instructions and haptic feedback instructions according to the currently detected obstacles, and the voice interaction module performs obstacle voice prompting according to the voice prompt instructions, and the vibration device performs vibration feedback with a set frequency, intensity, and time according to the haptic feedback prompt instructions; comparing the current required walking speed with the real-time speed of the user, and generating haptic feedback prompt instructions of different vibration levels according to the comparison result, and the vibration device performs vibration feedback of the corresponding vibration level according to the haptic feedback prompt instructions, wherein the different vibration levels include different vibration frequencies, intensities, and times.

4. The intelligent assistive blind guidance system based on audio-visual fusion and haptic feedback as claimed in claim 3, wherein, The vibration device performs vibration feedback of the corresponding vibration level according to the haptic feedback prompt instructions, including: When the user's current real-time speed is lower than the minimum safe speed, a low-frequency, low-intensity, long-term vibration prompts the user to speed up; When the user's current real-time speed exceeds the maximum safe speed, a medium-frequency, medium-intensity, long-lasting vibration prompts the user to slow down. When the user's current real-time speed is within the safe speed range, the vibration stops.

5. The intelligent assistive blind guidance system based on audio-visual fusion with tactile feedback as claimed in claim 1, wherein, In the control module, based on the planned path and movement strategy, combined with the user's push and pull force, the drive instructions are adjusted to adjust the travel speed of the robot guide dog, including: When the user's thrust is detected, the speed of the drive motor in the drive instruction is adjusted according to the degree of thrust change and the preset acceleration rules to adjust the robot guide dog to accelerate smoothly; When the user's pulling force is detected, the speed of the drive motor in the drive instruction is adjusted according to the degree of change in the pulling force and the preset acceleration rules to adjust the robot guide dog to decelerate smoothly; among them, the preset acceleration rules are used to reflect the relationship between the push and pull force applied by the user and the speed control.

6. An intelligent assisted blind guiding method based on audio-visual fusion and tactile feedback, characterized in that, The intelligent blind-aiding system based on audio-visual fusion and tactile feedback according to any one of claims 1 to 5 is implemented, comprising: Real-time perception of surrounding environment information, receiving user voice commands, and sensing the user's push and pull force on the guide stick; Based on the received surrounding environment information, a map of the surrounding environment is constructed, and combined with the user's voice instructions, a planned path and movement strategy is generated; the movement strategy includes the current required walking speed; Based on the planned path and movement strategy, combined with the user's voice commands and the user's push and pull force, walking prompt commands and driving commands are generated. Tactile vibration feedback and voice feedback are provided according to the walking prompt commands, and the machine guide dog is driven according to the driving commands to provide auxiliary guidance.

7. An electronic device, comprising: include: a memory for storing executable instructions; The processor is configured to implement the intelligent assisted blind guiding method based on audio-visual fusion and tactile feedback as described in claim 6 when executing the executable instructions stored in the memory.

8. A computer-readable storage medium, characterized in that, Executable instructions are stored, which are used to cause the processor to execute the executable instructions to implement the intelligent assisted blind guiding method based on audio-visual fusion and tactile feedback as described in claim 6.

Images