Vision-based route navigation obstacle avoidance system

Abstract

The invention discloses a vision-based route navigation obstacle avoidance system, and belongs to the field of travel assistance for visually impaired people, and the vision-based route navigation obstacle avoidance system is composed of three core parts, namely intelligent hardware equipment, a visual impaired assistance APP and a cloud service platform. The intelligent hardware equipment comprises a vision field pilot cap and an intelligent tactile stick; all the components realize real-time and accurate data sharing and smooth interaction through stable and efficient wireless communication technologies such as Bluetooth and Wi-Fi, and cooperative work of the system is ensured; the components are in stable and efficient wireless communication technologies such as Bluetooth, Wi-Fi and the like; according to the system, multiple core functions such as route navigation, obstacle avoidance, commodity recognition, emotion recognition and social interaction are highly integrated in a unified hardware platform and software system, a user does not need to carry multiple complex devices, operation is more convenient and smoother, inconvenience caused by switching among multiple devices is effectively avoided, and user experience is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of travel assistance for visually impaired people, and particularly to a vision-based route navigation and obstacle avoidance system. Background Art

[0002] In today's society, the visually impaired group faces many severe challenges during travel. For example, in the complex urban street environment, ordinary blind paths may be occupied or damaged, making it difficult for the visually impaired to rely on them for safe travel; in indoor places such as shopping malls and office buildings, there is a lack of effective guiding signs, resulting in the visually impaired being prone to getting lost. Although existing navigation and obstacle avoidance technologies have provided certain help to the visually impaired, there are still many limitations. From the aspect of function integration, most devices only have a single navigation or obstacle avoidance function and cannot meet the diverse travel needs of the visually impaired; in terms of user interaction, the operation interface is not friendly enough, and the voice prompts are not accurate and personalized enough, bringing inconvenience to the use of the visually impaired; for personalized needs, such as the special requirements of users with different visual impairment degrees and targeted assistance in different scenarios, existing technologies often cannot fully take them into account.

[0003] Therefore, a vision-based route navigation and obstacle avoidance system is needed to comprehensively improve the travel experience of the visually impaired, ensure safety during travel, greatly improve travel convenience, and fully meet their diverse needs in daily life such as socializing and shopping. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention provides a vision-based route navigation and obstacle avoidance system, which solves the problems raised in the above background art.

[0005] Technical Solution: To solve the above technical problems, according to one aspect of the present invention, more specifically, a vision-based route navigation and obstacle avoidance system is composed of three core parts: intelligent hardware devices, visually impaired assistance APP, and cloud service platforms; the intelligent hardware devices include a vision navigation cap and an intelligent blind stick.

[0006] Each component realizes real-time, accurate data sharing and smooth interaction through stable and efficient wireless communication technologies such as Bluetooth and Wi-Fi, ensuring the collaborative work of the system;

[0007] Image Recognition and Obstacle Detection: By applying cutting-edge computer vision algorithms and combining high-precision cameras and depth sensors, it can capture the images of the road ahead in real time and at high speed. It can not only accurately identify the position, type and size of obstacles, but also track and monitor the dynamic changes of obstacles. At the same time, the multi-scale feature fusion technology is introduced to improve the recognition accuracy of obstacles at different distances and of different sizes. The recognition results are timely and accurately informed to the user through clear and natural voice feedback and vibration feedback of different frequencies and intensities, helping the user make reasonable avoidance decisions in advance and ensuring safety in the walking process in all aspects.

[0008] Intelligent Navigation and Path Planning: Based on the fusion of multi-source positioning information such as high-precision maps, GPS positioning, and inertial navigation technology, the system can calculate the user's position in real time and accurately, and quickly plan the optimal path. During the path planning process, dynamic factors such as real-time traffic conditions, road construction information, and pedestrian flow are fully considered, as well as the user's historical walking preferences and habits. Through reinforcement learning algorithms, the path planning strategy is continuously optimized to provide dynamic and highly personalized path recommendations for visually impaired people, ensuring the efficiency and comfort of travel.

[0009] Multi-modal Perception and Environment Adaptation: By comprehensively applying a variety of advanced sensing technologies such as ultrasonic, infrared sensors, and lidar, a full-range and multi-level environment perception system is constructed. For the characteristics of different scenarios such as indoor, outdoor, and complex road conditions, an adaptive fusion algorithm is adopted to intelligently adjust the weights of each sensor and the data fusion method to achieve accurate environment perception and navigation. At the same time, deep learning algorithms are used to train a large amount of data in different scenarios, enabling the system to continuously optimize its perception and recognition capabilities, and significantly improving the accuracy and reliability of navigation in various complex environments.

[0010] Other Functions

[0011] Commodity and Price Recognition: By using deep learning technology and combining image recognition and optical character recognition (OCR) technology, the system can quickly and accurately automatically identify the commodities around the user. It can not only identify basic information such as the name, price, and brand of the commodity, but also identify and interpret the detailed specifications, ingredients, usage instructions, etc. of the commodity. Through voice broadcast and voice interaction functions, it provides comprehensive and detailed shopping assistance for visually impaired people, enabling them to make choices independently during the shopping process.

[0012] Emotion Recognition and Social Interaction: By adopting multi-modal fusion technologies such as advanced facial expression recognition, speech emotion analysis, and physiological signal monitoring, the system accurately analyzes the user's emotional state. According to different emotional states, the system can provide personalized emotional guidance, such as playing soothing music and providing encouraging words. Meanwhile, with the help of social networking platforms and instant messaging technologies, the system has powerful social interaction functions, helping visually impaired people establish close and convenient connections with family members, friends, volunteers, etc., promoting social interaction and enriching their social life.

[0013] Acquaintance Recognition and Emergency Response: Through high-precision face recognition and near-field communication technologies, the system can quickly and accurately identify the user's acquaintances (family members, friends, volunteers). When detecting the approach of an acquaintance, the system provides personalized voice or vibration feedback, enhancing the user's sense of security and social experience. In case of an emergency, when a visually impaired person is in a dangerous state or actively sends a distress signal, the system immediately automatically sends a distress notice containing detailed information such as real-time location and on-site environmental sounds to family members and nearby volunteers, ensuring that effective assistance can be obtained in a timely manner.

[0014] Visually Impaired Assistant APP Intelligent Navigation and Safety Reminder: The APP is deeply linked with intelligent hardware devices. It can not only synchronize navigation information in real time but also conduct real-time assessment and analysis of the safety of the surrounding environment based on sensor data. When detecting potential dangers ahead, such as road construction and approaching vehicles, the APP will issue various forms of reminders in a timely manner, such as voice alarms, vibration prompts, and flash reminders, ensuring that users can detect and take corresponding measures in a timely manner to ensure travel safety.

[0015] Data Security and Privacy Protection: By adopting advanced end-to-end encryption technology, all transmitted data, including the personal data, travel trajectories, health information, and help-seeking information of visually impaired people, is encrypted with high strength to prevent data from being stolen or tampered with during transmission and storage. Meanwhile, the system strictly follows relevant domestic and international data privacy protection regulations and sets up a perfect user privacy management mechanism. Users can independently control the scope and permissions of the use of their personal data, fully protecting the privacy and security of users.

[0016] Cloud Computing and Big Data Analysis: With the help of a powerful cloud computing platform and advanced big data technologies, the APP can share information of the visually impaired group, family members, and volunteers in real time and efficiently. Through in-depth mining and analysis of a large amount of data, the system can continuously optimize the route planning algorithm, improve the accuracy of obstacle recognition, and improve the intelligent recommendation function. At the same time, according to the user's usage habits and feedback information, it provides more personalized services and suggestions, continuously enhancing the performance of the system and the user experience.

[0017] Intelligent early warning and help-seeking function: When visually impaired people encounter difficulties or need help, they can trigger the help-seeking signal through various convenient methods, such as voice commands, button operations, gesture recognition, etc. After receiving the help-seeking signal, the system will immediately automatically send a detailed early warning notice to nearby volunteers and family members, and provide real-time and accurate positioning information. At the same time, the on-site environment monitoring function is activated, and information such as the sound and image at the scene is transmitted to the rescue personnel in real time so that they can better understand the situation and take effective rescue measures. In addition, the system will also intelligently analyze the rescue needs based on historical help-seeking data and rescue experience, provide reasonable rescue suggestions and resource allocation plans for the rescue personnel, and improve the rescue efficiency.

[0018] The beneficial effects of a vision-based route navigation and obstacle avoidance system of the present invention are as follows:

[0019] (1) High integration and integrated design: This system highly integrates multiple core functions such as route navigation, obstacle avoidance, commodity recognition, emotion recognition, and social interaction in a unified hardware platform and software system. Users do not need to carry multiple complex devices, and the operation is more convenient and smooth, effectively avoiding the inconvenience caused by switching between multiple devices, and greatly improving the user experience.

[0020] Combining multiple perception technologies such as vision, hearing, and touch, a multi-modal perception system is constructed, and advanced intelligent algorithms such as deep learning and reinforcement learning are used for data fusion and analysis. This innovative design can significantly improve the system's perception ability and adaptability to complex environments, improve the accuracy of obstacle recognition and the safety of path planning, and provide more reliable travel assistance for visually impaired people.

[0021] (2) Smart interconnection and social interaction ecosystem: Through the cloud platform, real-time information sharing and efficient interaction among visually impaired people, family members, and volunteers are realized, and a smart interconnection ecosystem full of care and support is constructed. It not only effectively improves the safety of visually impaired people's travel, but also provides them with more social opportunities, promotes social interaction and emotional communication, and helps to improve the quality of life and happiness of visually impaired people.

[0022] (3) Enhanced emergency help-seeking function: It has accurate real-time positioning and intelligent early warning functions. When users encounter danger, they can quickly and conveniently send a help-seeking signal. The system will automatically and timely notify family members and nearby volunteers, and provide comprehensive and detailed on-site information to ensure that visually impaired people can receive timely and effective help in the shortest time, and maximize the protection of their lives.

[0023] (4) Personalized Service and Continuous Optimization: Through in-depth analysis and learning of user data, highly personalized services can be provided according to the special needs and usage habits of each visually impaired person, such as personalized voice prompts, customized navigation paths, etc. At the same time, with the help of cloud computing and big data technologies, the system can continuously collect user feedback and new data, and continuously optimize the functions and performance of the system to provide more high-quality and considerate services for visually impaired people. Brief Description of the Drawings

[0024] The present invention will be further described in detail below with reference to the drawings and specific implementation methods.

[0025] Figure 1 It is a structural schematic diagram of the present invention. Specific Embodiments

[0026] The present invention will be described in detail below with reference to the drawings and embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0027] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0028] Refer to Figure 1 Example 1

[0029] Image Recognition and Obstacle Detection:

[0030] Hardware Basis: The high-precision camera on the Vision Navigation Cap collects the image data of the road ahead at a rate of 30 frames per second or more, and the depth sensor synchronously obtains the distance information.

[0031] Algorithm Process: Use object detection algorithms based on convolutional neural networks (CNNs), such as Faster R-CNN or YOLO series. Input the collected image data into a model trained with a large number of obstacle images. The model extracts and analyzes the image features to identify the category (such as pedestrians, vehicles, roadblocks, etc.), position coordinates, and approximate size of the obstacles. For dynamic obstacles, techniques such as optical flow method are used to track their movement trajectories. When an obstacle is recognized, the system broadcasts the obstacle information through the voice module at different speech rates and intonations according to the set rules based on the distance and relative position between the obstacle and the user. At the same time, the vibration module of the intelligent blind stick or the Vision Navigation Cap vibrates with different frequencies and intensities to remind the user to take evasive measures.

[0032] Intelligent Navigation and Path Planning:

[0033] Location Technology Integration: The system obtains the approximate location of the user through GPS positioning, and combines inertial navigation technology (built-in gyroscope and accelerometer) to accurately measure the position change within a short period of time to make up for the possible short-term loss or inaccuracy of GPS signals. At the same time, Wi-Fi positioning technology is used to further improve the positioning accuracy by scanning the surrounding Wi-Fi hotspot information and matching it with the database.

[0034] Path Planning Algorithm: Based on the Dijkstra algorithm, combined with real-time traffic information (obtained from the cloud service platform) and high-precision map data. The map data contains detailed information such as road types, traffic directions, slopes, etc. When planning the path, the algorithm sets the user's current location as the starting point and the destination as the ending point, considering factors such as road congestion, whether it is suitable for visually impaired people to pass (such as the presence of blind paths, the number of steps, etc.), calculates multiple feasible paths, and filters out the optimal path according to the user's preference for paths in historical travel data (more inclined to quiet roads or shorter-distance roads). During the navigation process, real-time navigation instructions such as turning and going straight are provided to the user through the voice module, and when there are traffic condition changes (road construction ahead, vehicle congestion on the road), the path is re-planned in a timely manner.

[0035] Multi-modal Perception and Environment Adaptation:

[0036] Sensor Fusion: In addition to cameras and depth sensors, ultrasonic sensors are used to detect obstacles at close range, with an effective detection range of 0.1 - 5 meters and an accuracy of up to centimeter level. Infrared sensors are used to detect heat source objects in front, such as humans, vehicle engines, etc., to assist in judging the environmental situation. Data fusion algorithms such as the Kalman filter algorithm are used to fuse data from different sensors to improve the accuracy and reliability of environmental perception.

[0037] Deep Learning Optimization: An environment perception model based on a recurrent neural network (RNN) or long short-term memory network (LSTM) is constructed, with the fused sensor data as the input. The model is trained with a large amount of data in different scenarios (indoors, outdoors, sunny days, rainy days), and can automatically learn the data feature patterns in different environments, so as to accurately classify and adapt to the current environment. For example, in an indoor environment, by analyzing the change in Wi-Fi signal strength and ultrasonic sensor data, the room layout and furniture positions are judged; in rainy days, according to the blur degree of camera images and the interference of raindrops on sensor signals, the obstacle detection threshold and algorithm parameters are adjusted to ensure the accuracy of navigation.

[0038] Embodiment 2

[0039] Commodity and Price Recognition:

[0040] Image Acquisition and Processing: The user aims the camera on the Vision Navigation Cap at the commodity, and the camera automatically adjusts the focus to obtain a clear image of the commodity. Image enhancement techniques, such as histogram equalization and contrast stretching, are used to improve the image quality for subsequent recognition.

[0041] Deep Learning Recognition: An image classification model based on a convolutional neural network is used to conduct a preliminary classification of the commodity to determine the general category to which the commodity belongs (such as food, daily necessities, etc.). Then, optical character recognition (OCR) technology, such as the CRNN model combining a convolutional neural network and a recurrent neural network, is used to recognize the text information on the commodity packaging and extract key information such as the commodity name, price, brand, and specifications. For longer texts such as ingredients and usage instructions, semantic analysis and understanding are performed through natural language processing technology, and the results are organized and clearly announced to the user through the voice module. To improve the recognition accuracy, the system continuously collects new commodity image data and regularly updates and trains the model.

[0042] Emotion Recognition and Social Interaction:

[0043] Multi-modal Data Acquisition: Facial expression images of the user are collected through the camera on the Vision Navigation Cap, with 15 - 30 frames collected per second; the user's voice signal is collected using the built-in microphone, and at the same time, physiological signals such as the user's heart rate and skin conductance response are monitored through wearable devices (such as smart bracelets and ankle rings).

[0044] Fusion Analysis Algorithm: Multi-modal fusion technology is adopted. The facial expression image data is input into an expression recognition model based on a convolutional neural network, the voice signal is input into a voice emotion analysis model based on a recurrent neural network, and the physiological signals are input into machine learning models such as a support vector machine (SVM) for classification analysis. Then, through a decision-level fusion algorithm, such as the weighted voting method, the analysis results of the three models are integrated to obtain the user's current emotional state (such as happy, sad, anxious, etc.). According to the emotional state, the system selects corresponding strategies from a pre-set emotional guidance strategy library, such as playing cheerful music, sending encouraging voice messages, etc. In terms of social interaction, it is connected to the social network platform through the visually impaired assistance APP, and the user can send messages, answer voice calls, etc. through voice commands to achieve interactive communication with family members, friends, and volunteers.

[0045] Acquaintance Recognition and Emergency Response:

[0046] Face Recognition Technology: The camera of the vision-leading cap captures the surrounding face images. Using a face recognition algorithm based on deep learning, such as the FaceNet model based on convolutional neural network, the captured face images are compared with the facial feature library of acquaintances pre-stored in the system. The features in the feature library are obtained by preprocessing and feature extraction of acquaintance photos. The similarity threshold is set to 0.8 - 0.9. When the similarity exceeds the threshold, it is determined as an acquaintance, and the user is prompted through the voice module.

[0047] Emergency Response Mechanism: When the user presses the emergency help button on the intelligent blind stick or the visually impaired assistance APP, or sends a help signal through a voice command, the system immediately obtains the user's real-time location information (obtained through the fusion of GPS, inertial navigation, and Wi-Fi positioning), and sends the location information, the surrounding environment image (captured by the camera), and the user's basic information (such as name, health status, etc.) to the cloud service platform through the network. The cloud service platform sends help notifications to the family members and nearby volunteers according to the list of emergency contacts set by the user. The volunteers can receive the notifications through the visually impaired assistance APP and view the user's location and relevant information, and go to provide help in a timely manner. At the same time, the system activates the recording function to record the on-site sounds for subsequent situation understanding.

[0048] Example Three

[0049] Function Realization of the Visually Impaired Assistance APP

[0050] Intelligent Navigation and Safety Reminder: The APP is connected to the intelligent hardware device in real time through Bluetooth, and receives data such as location and environmental perception from the hardware device. On the navigation interface, the navigation route and real-time traffic conditions are displayed in a simple and clear voice interaction manner. In terms of the safety reminder function, according to the obstacle detection data and environmental risk assessment results transmitted by the hardware device (such as a vehicle approaching quickly nearby, a big pit on the road ahead, etc.), potential safety hazards are timely informed to the user through voice alarms, vibration reminders, and displaying danger prompts in large fonts and high-contrast icons on the APP interface.

[0051] Data Security and Privacy Protection: During the data transmission process, the SSL / TLS encryption protocol is adopted to encrypt and transmit data such as the user's personal information, travel trajectory, and help information, ensuring the security of data during network transmission. In terms of data storage, the cloud service platform adopts encryption storage technology to store user data in a secure database, and sets strict access permission control. Only authorized system modules and the user himself can access the relevant data. At the same time, the APP provides a user privacy setting interface, and the user can independently choose whether to share certain data and set the data retention period, etc.

[0052] Cloud Computing and Big Data Analysis: The APP uploads users' travel data, usage habit data, product identification data, etc. to the cloud service platform. The cloud service platform uses cloud computing technology to store and compute massive amounts of data. Through big data analysis techniques such as clustering analysis and association rule mining, it analyzes information such as users' travel patterns, preferred shopping places, and types of obstacles commonly encountered. Based on the analysis results, it optimizes the route planning algorithm to provide users with a navigation path more in line with their habits; improves the product identification model to enhance the identification accuracy; and at the same time, according to users' needs, pushes personalized service information to users, such as surrounding activity places suitable for visually impaired people, preferential shopping information, etc.

[0053] Intelligent Early Warning and Help Function: The APP monitors the user's status in real time. When it detects situations that may indicate difficulties, such as the user remaining stationary for a long time (exceeding the set time, such as 15 minutes) or walking at an abnormally slow speed, it automatically issues an early warning prompt to ask the user if they need help. After the user confirms the request for help, the system sends a help message to the family members and volunteers according to the emergency response mechanism. In addition, the user can also actively send a help signal through voice commands or by operating the help button on the APP interface. After receiving the signal, the system quickly executes the help process to ensure that the user can obtain assistance in a timely manner.

[0054] The above-described embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

Claims

1. A vision-based route navigation and obstacle avoidance system, comprising an intelligent hardware device, a visually impaired assistance APP, and a cloud service platform, characterized in that: The smart hardware device, the visually impaired assistance APP and the cloud service platform achieve data sharing and interaction through wireless communication technology; the smart hardware device includes a vision navigation cap and a smart blind stick.

2. The vision-based route navigation and obstacle avoidance system according to claim 1, characterized in that: The Vision Pilot Cap is equipped with a high-precision camera, depth sensor, ultrasonic sensor, infrared sensor and laser radar. It uses computer vision algorithms and multi-scale feature fusion technology, combined with high-precision cameras and depth sensors for image recognition and obstacle detection, and provides feedback to users in the form of voice, vibrations of different frequencies and intensities, and flashes. Based on the fusion of multi-source positioning information such as high-precision maps, GPS positioning, and inertial navigation technology, the system uses reinforcement learning algorithms to achieve intelligent navigation and route planning, and provides dynamic and highly personalized route recommendations based on real-time traffic conditions, road construction information, pedestrian flow, and user historical walking preferences; It uses a variety of sensing technologies such as ultrasonic waves, infrared sensors, and lidar, adopts an adaptive fusion algorithm to achieve multimodal perception and environmental adaptation, and intelligently adjusts sensor weights and data fusion methods for different scenarios.

3. The vision-based route navigation and obstacle avoidance system according to claim 2, characterized in that: The system uses deep learning technology combined with image recognition and optical character recognition technology to identify products and prices. It can automatically identify product names, prices, brands, specifications, ingredients, instructions for use and other information and provide voice broadcast to assist shopping.

4. The vision-based route navigation and obstacle avoidance system according to claim 3, characterized in that: The system achieves emotion recognition and social interaction through multimodal fusion technologies such as facial expression recognition, voice emotion analysis, and physiological signal monitoring. It can accurately analyze user emotions, provide personalized emotional guidance, and realize social interaction functions with the help of social networking platforms and instant messaging technologies.

5. The vision-based route navigation and obstacle avoidance system according to claim 4, characterized in that: Through high-precision facial recognition and near-field communication technology, acquaintance recognition and emergency response can be achieved. It can quickly and accurately identify acquaintances, provide personalized feedback when acquaintances are detected approaching, and automatically send help notifications containing detailed information such as real-time positioning and on-site environmental sounds to family members and nearby volunteers in an emergency.

6. A vision-based route navigation and obstacle avoidance system according to claim 5, characterized in that: the visually impaired assistance APP is deeply linked with the intelligent hardware device to perform real-time navigation, the safety of the surrounding environment is evaluated in real time according to the sensor data, and safety reminders are issued in various forms such as voice alarms, vibration reminders, and flash reminders; advanced end-to-end encryption technology is used to protect data security and user privacy, relevant data privacy protection laws and regulations are followed, and a complete user privacy management mechanism is set up; Use cloud computing platforms and big data technology to achieve real-time information sharing among visually impaired groups, their families, and volunteers. Optimize route planning, obstacle identification, and intelligent recommendation functions through in-depth mining and analysis of large amounts of data, and provide personalized services based on user habits and feedback. It has intelligent early warning and help-seeking functions, supports triggering help signals through voice commands, button operations, gesture recognition, etc., automatically sends detailed early warning notifications to nearby volunteers and family members, provides real-time and accurate positioning, starts on-site environmental monitoring and transmits relevant information, and can also provide rescue suggestions and resource allocation plans based on historical data and rescue experience.

7. The vision-based route navigation and obstacle avoidance system according to claim 6, characterized in that: The system has personalized service functions, providing personalized voice prompts, customized navigation paths and targeted auxiliary function settings according to the special needs and usage habits of the visually impaired.

8. The vision-based route navigation and obstacle avoidance system according to claim 6, characterized in that: The system continuously collects user feedback and new data through cloud computing and big data technologies, and automatically optimizes and upgrades the system's functions and performance.

Images