Visually impaired person navigation processing method and device based on smart phone
Through the navigation processing method of smartphones, real-time environmental measurement and object recognition are used to use cameras to solve the problems of the convenience and maintenance cost of existing smart devices to assist visually impaired people, and realize independent travel and safety of visually impaired people promote.
Patent Information
- Application Number
- CN202410916855.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-08
AI Technical Summary
Existing smart devices assist visually impaired people with high weight, inconvenience, high maintenance costs, and long learning and adaptation time, making it difficult to meet the independent travel and social participation needs of visually impaired people.
The navigation processing method based on smartphones enables navigation function through voice commands, uses the camera to perform real-time environmental measurement and object recognition, identify roads, obstacles and potholes, and provides navigation guidance through voice broadcasts to achieve convenient navigation and environmental perception.
It provides a convenient and low maintenance cost navigation solution, improves the independence and safety of visually impaired people, simplifies operations, reduces dependence on help from others, and enhances autonomy and social participation.
Smart Images

Figure CN118913243A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of smart terminals, and in particular to a navigation processing method and device for visually impaired persons based on smart phones, a smart terminal and a storage medium. Background Art
[0002] Visually impaired people have a strong need for independence, freedom and participation in social life. In the existing technology, some smart devices assist visually impaired people, such as GPS navigation devices for the visually impaired: this type of device can help the visually impaired give directions to guide dogs, assist in determining the destination through voice settings, and the device prompts the direction through vibration. Peepo navigator: It includes a handheld voice-controlled GPS and a device carried by a guide dog, which reminds the visually impaired where to go through vibration.
[0003] However, existing smart devices for assisting the visually impaired are heavy or not convenient enough, and are not adequate for some situations. At the same time, some high-end assistive devices are expensive, have high maintenance costs, and require a long time to learn and adapt to the effective use of the equipment, which is not convenient for the visually impaired to use.
[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention
[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a navigation processing method, device, intelligent terminal and storage medium for visually impaired people based on smart phones are provided. The present invention is based on the implementation of smart phones, which is more convenient for visually impaired people to use, has low maintenance costs, and is simple, convenient and easy to understand.
[0006] The technical solution adopted by the present invention to solve the problem is as follows:
[0007] A navigation processing method for visually impaired people based on a smart phone, comprising:
[0008] Get voice commands to start the navigation function and set the navigation destination;
[0009] Plan the navigation route according to the set navigation destination and start the real-time navigation voice broadcast;
[0010] During real-time navigation, the camera is controlled to intelligently measure the distance and height of objects around the navigation route, identify objects and classify them, and realize the recognition of roads, objects, obstacles, and potholes;
[0011] Obtain information within a predetermined range from the camera and perform analysis, calculation, and screening; determine whether there are obstacles, potholes, and / or dangerous areas, and use voice broadcasts to prompt and assist users to pay attention and walk.
[0012] In the navigation processing method for visually impaired people based on a smartphone, the predetermined range is 60 / 80 degrees from the camera viewing angle and 10 meters in front.
[0013] The navigation processing method for visually impaired persons based on a smart phone, wherein, during the real-time navigation process, the step of controlling the camera to intelligently measure the distance and height of objects around the navigation route, identifying and classifying objects, and realizing the identification of roads, objects, obstacles, and potholes also includes:
[0014] When a voice command is received to describe the surrounding environment, the camera is controlled to intelligently measure and identify the environment and objects within the predetermined range around the camera, and a real-time voice broadcast is performed on the object category and its orientation, color, shape, and size within the predetermined range around the camera.
[0015] The method for processing navigation for visually impaired persons based on a smart phone further comprises the steps of:
[0016] When the user's voice question about the type of item is obtained, the camera is controlled to identify the items within the viewing angle to search and feedback the results;
[0017] Based on the feedback results, it is determined whether it is necessary to move the viewing angle of the smartphone camera to control the continued search until successful recognition is obtained and feedback and confirmation are given, and then the control is used to provide direction guidance prompts.
[0018] The navigation processing method for visually impaired persons based on a smart phone, wherein, during the real-time navigation process, the steps of controlling the camera to intelligently measure the distance and height of objects around the navigation route, identifying and classifying objects, and realizing the identification of roads, objects, obstacles, and potholes include:
[0019] When performing real-time navigation, the camera function is automatically activated to capture real-time images around the navigation route;
[0020] The image data obtained by the camera is used to intelligently measure the distance and height of objects using computer vision technology;
[0021] Perform object detection and recognition on the captured images, identify various objects in the images, and classify them; the identified objects will be classified into roads, ordinary objects, obstacles, and potholes;
[0022] Comprehensively analyze the results of recognition and classification, make corresponding navigation decisions based on road conditions and object location information, provide navigation guidance, and remind people to avoid risk areas such as obstacles and potholes.
[0023] The method for processing navigation for visually impaired persons based on a smart phone, wherein the steps of obtaining information within a predetermined range from the camera and performing analysis, calculation, and screening; determining whether there are obstacles, potholes, and / or dangerous areas, and providing voice broadcasts to prompt and assist the user to pay attention and walk include:
[0024] Obtain information within a certain range that the camera can detect, including distance data, height data, shape data, and analyze it;
[0025] Computationally analyze the acquired information to determine the characteristics and location of surrounding objects and terrain;
[0026] After calculation, the information that is important or may affect the user's navigation is screened out, including obstacles, potholes and dangerous places;
[0027] Based on the screened information, determine whether there are obstacles, potholes or other dangerous factors in the surrounding environment and conduct risk assessment;
[0028] When obstacles, potholes or other dangerous situations are confirmed, users will be reminded through voice broadcasts and specific precautions and suggestions will be provided.
[0029] The method for processing navigation for visually impaired persons based on a smart phone, wherein the steps of obtaining information within a predetermined range from the camera and performing analysis, calculation, and screening; determining whether there are obstacles, potholes, and / or dangerous areas, and providing voice broadcasts to prompt and assist the user to pay attention and walk, further include:
[0030] For navigation destinations you have already visited, turn on the habit learning mode to memorize the route;
[0031] The next time you navigate to the same destination, the familiar route mode will be automatically turned on for navigation reminders.
[0032] A navigation processing device for visually impaired people based on a smart phone, wherein the device comprises:
[0033] Voice navigation is turned on, get voice commands to turn on the navigation function and set the navigation destination;
[0034] The navigation broadcast module is used to plan the navigation route according to the set navigation destination and start the real-time navigation voice broadcast;
[0035] The surrounding object recognition module is used in real-time navigation to control the camera to intelligently measure the distance and height of objects around the navigation route, identify objects and classify them, and realize the recognition of roads, objects, obstacles, and potholes;
[0036] The prompt module is used to obtain information within a predetermined range from the camera and perform analysis, calculation, and screening; determine whether there are obstacles, potholes, and / or dangerous areas, and provide voice notifications to assist users in paying attention and walking;
[0037] The surrounding environment detection and broadcasting module is used to control the camera to intelligently measure and identify the environment and objects within the predetermined range around the camera when a voice command for describing the surrounding environment is obtained, and to perform real-time voice broadcasting of the object type and its orientation, color, shape, and size within the predetermined range around the camera;
[0038] The search prompt module is used to control the camera to identify objects within the field of view and feedback the results when the user's voice question about the type of object is obtained; based on the feedback results, it is determined whether it is necessary to move the field of view of the smartphone camera to control the continued search until the recognition is successful and feedback and confirmation are given, and then the direction guidance prompt is given.
[0039] An intelligent terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.
[0040] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described above.
[0041] Beneficial effects of the invention: The invention provides a navigation processing method, device, intelligent terminal and storage medium for visually impaired people based on smart phones. The invention develops and applies a system for visually impaired people based on smart phones, and utilizes the entire system of smart phones combined with data network big data / cloud, plus different third-party algorithms and applications, to integrate and implement development and application on mobile phones. The smart phones are given the following new functions:
[0042] Navigation function: Navigation in the visually impaired mode and assisting the visually impaired to go out;
[0043] Surrounding environment reporting function: Provides users with reports on surrounding environment in visually impaired mode, usually suitable for indoor or outdoor sitting scenes;
[0044] Object-finding function: helps users find objects / stuff in visually impaired mode;
[0045] Help / alarm function: Help / alarm in emergency;
[0046] Route memorizing function: remember the route when you are familiar with it and get used to the learning mode.
[0047] The present invention is based on the implementation of a smart phone, is more convenient for visually impaired people to use, has low maintenance cost, and is simple, convenient and easy to understand to operate. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 It is a flowchart of a navigation processing method for visually impaired persons based on a smart phone provided in an embodiment of the present invention.
[0050] Figure 2 It is a flowchart of a navigation processing method for visually impaired people based on a smart phone provided in a specific application embodiment of the present invention.
[0051] Figure 3 It is a principle block diagram of an embodiment of a navigation processing device for visually impaired persons based on a smart phone provided by the present invention.
[0052] Figure 4 It is a block diagram of the internal structure principle of the intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0054] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0055] The desires that a visually impaired person may have for traveling include:
[0056] Independence: Visually impaired people want to be able to carry out daily activities independently, just like sighted people, without relying on others.
[0057] Social participation: They desire to participate in social activities, including work, education, entertainment and socializing, which requires them to be able to travel freely.
[0058] Exploring new places: Like everyone else, visually impaired people want to explore new places and experience different environments and cultures.
[0059] Technological assistance: They hope to have more advanced assistive technologies, such as smart navigation devices, to help them travel more easily.
[0060] Affordability: Visually impaired people also want to be able to afford the assistive devices and services they need to get around.
[0061] Based on the above requirements and in combination with the rapid development of smart phones today, the present invention proposes a navigation processing method for visually impaired people based on smart phones.
[0062] like Figure 1 As shown, a navigation processing method for visually impaired persons based on a smart phone in Embodiment 1 of the present invention comprises the following steps:
[0063] Step S100, obtaining a voice command to start the navigation function and set the navigation destination;
[0064] In the embodiment of the present invention, a smart phone is taken as an example for description. The smart phone is very convenient for users to carry.
[0065] In this step, when the user says the voice command: "Start the navigation function", the system receives and recognizes the user's command. The system turns on the navigation function, displays the map interface, and waits for the user's next command. If the user says the voice command: "Set the destination to XXX", the system receives and recognizes the destination information.
[0066] The benefit of this step is that voice commands can save users from manual operation, making it more convenient for the visually impaired to walk and improving their walking safety. Users can directly input destination information by voice without manual input, saving time and improving user experience.
[0067] Step S200, planning a navigation route according to the set navigation destination, and starting real-time navigation voice broadcast;
[0068] In the embodiment of this step, route planning is performed based on the navigation destination set by the user through map data and traffic information. After determining the best route, the present invention controls the start of the real-time navigation voice broadcast function. During the navigation process, the present invention will give voice prompts based on the current position and direction to guide the user along the preset route. The real-time navigation voice broadcast will provide early warnings to help users reach their destination more conveniently and safely.
[0069] The present invention can update the route and remind the user in time according to different situations through real-time navigation voice broadcast. The voice prompt can provide more intuitive guidance.
[0070] Step S300: During real-time navigation, the camera is controlled to intelligently measure the distance and height of objects around the navigation route, identify and classify the objects, and realize the identification of roads, objects, obstacles, and potholes;
[0071] In this step embodiment, during the real-time navigation process, the camera of the smartphone is controlled to perform intelligent measurement to detect the distance and height of objects around the navigation route. Through image recognition technology, surrounding objects are classified, such as roads, buildings, vehicles, pedestrians, etc., and any potential obstacles or potholes can be identified. Through real-time monitoring and recognition, the system can provide more comprehensive and accurate scene information for the visually impaired, helping users to effectively respond to changes in navigation road conditions and ensure walking safety.
[0072] In this way, through intelligent measurement and object recognition, users can be informed of possible obstacles or dangerous situations on the road in a timely manner to improve safety. And by real-time monitoring of objects around the navigation route, it can provide users with more accurate information and assist users in making more informed decisions. Intelligent recognition technology can help users understand the surrounding environment more clearly and improve the navigation experience.
[0073] For example, Xiao Wang, a visually impaired person, is using navigation to go to a shopping supermarket. The system turns on the camera's intelligent measurement function and identifies objects around the road. On the road, the present invention uses the camera to identify a protruding stone and a pedestrian crossing the road. The present invention will promptly remind Xiao Wang to avoid them through voice prompts to ensure walking safety. In addition, the present invention can also identify potholes on the road and remind Xiao Wang to walk slowly to avoid falling. Such intelligent monitoring and identification functions add a layer of safety protection to Xiao Wang, a visually impaired person, during his walking process.
[0074] Furthermore, the step S300 specifically includes:
[0075] S301, when performing real-time navigation, automatically start the camera function to capture real-time images around the navigation route;
[0076] During real-time navigation, this step will automatically activate the smartphone camera function to capture real-time images around the navigation route to obtain the latest road and surrounding environment information.
[0077] S302, using computer vision technology to intelligently measure the distance and height of an object using image data obtained by a camera;
[0078] In the embodiment of this step, computer vision technology is used to intelligently measure the image data obtained by the camera, calculate the distance and height of the object, and provide basic data for subsequent recognition and classification.
[0079] S303, performing object detection and recognition on the captured image, identifying various objects in the image, and classifying them; the identified objects will be classified into roads, ordinary objects, obstacles, and potholes;
[0080] In this step, object detection and recognition are performed on the captured image, various objects in the image are identified and classified, and the identified objects are classified into roads, ordinary objects, obstacles and potholes, so as to fully understand the environmental conditions around the navigation path.
[0081] S304, comprehensively analyzing the results of recognition and classification, making corresponding navigation decisions according to road conditions and object location information, providing navigation guidance, and reminding to avoid risk areas such as obstacles and potholes.
[0082] In the embodiment of this step, the results of comprehensive analysis of recognition and classification will make corresponding navigation decisions based on road conditions, object location information and intelligent measurement data, provide navigation guidance and promptly remind the visually impaired to avoid risk areas such as obstacles and potholes to ensure safe driving.
[0083] It can be seen that in this step, through intelligent measurement, object recognition and classification analysis, real-time environmental information on the road can be provided to the visually impaired in a timely manner, effectively helping the visually impaired to prevent potential dangers and obstacles and improve walking safety. The present invention comprehensively analyzes the recognition data and makes intelligent navigation decisions based on the road conditions, which can provide more accurate and personalized navigation guidance, allowing the visually impaired to reach their destination more smoothly and optimize the navigation experience.
[0084] For example, Xiao Li, a visually impaired person, is using navigation to go to his friend's house. The present invention will activate the camera in real time to capture images around the route. During the walking process, the present invention can control the smartphone to intelligently measure the distance and height of buildings on both sides of the road through computer vision technology, and perform object detection and recognition at the same time, identifying trees, pedestrians and road obstacles such as roadblocks or potholes on the roadside. After comprehensive analysis, detailed navigation guidance will be provided, including reminders to slow down, avoid obstacles, and remind to pay attention to potholes, so as to help Xiao Li reach his destination safely and smoothly. This intelligent navigation system can effectively improve the walking experience of the visually impaired and ensure the walking safety of the visually impaired.
[0085] Step S400, obtaining information within a predetermined range from the camera and performing analysis, calculation, and screening; determining whether there are obstacles, potholes, and / or dangerous areas, and providing voice notifications to assist the user in paying attention and walking.
[0086] In an embodiment of the present invention, a camera scanning function can be provided for the visually impaired: Specifically, the visually impaired can scan the surrounding environment through the camera carried by the smartphone. The camera scanning distance and range are set to obtain information within a specific distance. The mobile phone application is used to identify, analyze and calculate the information, including detecting obstacles, potholes and potential dangerous places. Based on the identification results, screening and judgment are performed to confirm whether there are obstacles, potholes or dangerous places.
[0087] Then, a voice announcement will be made; specifically, according to the recognition result, the voice announcement system can provide warning information to the visually impaired, including "there are obstacles ahead", "please pay attention to the potholes ahead", "go slowly, there are dangerous places", etc. The present invention can provide accurate and clear voice announcements to help the visually impaired pay attention to the surrounding environment and take corresponding actions.
[0088] As can be seen from the above, the present invention can help the visually impaired quickly obtain information about the surrounding environment through camera scanning and voice prompts, avoid collisions with obstacles or falling into potholes, and thus improve walking safety. And after using this technology, the visually impaired can travel more independently, reduce the need for help from others, and enhance their autonomy. And by improving the independence and safety of the visually impaired, they can be encouraged to participate more in social activities and actively integrate into society.
[0089] For example, Xiao Ming is a visually impaired person. When he travels, he can use the camera scanning function of his smartphone and identify the surrounding environment through the mobile phone application. When there are obstacles or potholes, the system will automatically issue a voice prompt, such as "There is an obstacle ahead, please be careful to detour." After hearing the prompt, Xiao Ming can adjust the direction of walking to avoid collision or falling. This system makes Xiao Ming feel more at ease when he goes out, increasing his independence and sense of security.
[0090] Furthermore, in the navigation processing method for visually impaired persons based on a smart phone according to an embodiment of the present invention, step S400 specifically includes:
[0091] S401, obtaining information within a certain range that can be detected by the camera, including distance data, height data, and shape data, and performing analysis;
[0092] In this step, the visually impaired person uses a mobile phone to obtain data information such as distance, height and shape within a certain range through a camera, and analyzes the data. The predetermined range is preferably 60 / 80 degrees from the camera angle and 10 meters in front.
[0093] S402, determining the characteristics and positions of surrounding objects and terrain by calculating and analyzing the acquired information;
[0094] In the embodiment of this step, the characteristics and positions of the surrounding objects and terrain are determined by calculating and analyzing the acquired information.
[0095] S403, after calculation, screening out information that is important or may affect user navigation, including obstacles, potholes, and dangerous places;
[0096] In the embodiment of this step, information that is important or may affect the user's navigation is screened out, including obstacles, potholes and dangerous places.
[0097] S404, judging whether there are obstacles, potholes or other dangerous factors in the surrounding environment based on the screened information, and conducting risk assessment;
[0098] In the embodiment of this step, a risk assessment is performed based on the screened information to determine whether there are obstacles, potholes or other dangerous factors in the surrounding environment.
[0099] S405. When it is confirmed that there are obstacles, potholes or other dangerous situations, the user will be reminded through voice broadcast and specific precautions and suggestions will be provided.
[0100] In the embodiment of the present invention, when the existence of obstacles, potholes or other dangerous situations is confirmed, the visually impaired are reminded through voice broadcasting, and specific precautions and suggestions are provided.
[0101] In this way, the present invention obtains and analyzes the surrounding environment information through the mobile phone, which can help the visually impaired to find obstacles, potholes or other dangerous situations in time, reducing the risk of accidents. The present invention can also enable the visually impaired to independently obtain information about the surrounding environment, and receive warnings and suggestions in time, reduce dependence on others, and enhance autonomy. And through voice broadcasting to remind and guide, it can make the visually impaired more convenient and comfortable during the walking process, improving the overall user experience.
[0102] For example, Xiao Wang, a visually impaired person, uses the camera function of his mobile phone to obtain information about the surrounding environment and analyzes the acquired data through an application. When Xiao Wang approaches a pothole, the smartphone system of the embodiment of the present invention will automatically recognize and issue a voice prompt: "There is a pothole ahead, please be careful of your feet." After hearing the prompt, Xiao Wang will be more alert when walking to avoid falling and getting injured. This technology of the present invention makes Xiao Wang, a visually impaired person, feel safer and more assured when going out, improving his walking experience.
[0103] In the embodiment of the present invention, further, it also includes the steps of:
[0104] When a voice command is received to describe the surrounding environment, the camera is controlled to intelligently measure and identify the environment and objects within the predetermined range around the camera, and a real-time voice broadcast is performed on the object category and its orientation, color, shape, and size within the predetermined range around the camera.
[0105] In this step, after receiving the specified voice command for describing the surrounding environment, the present invention uses the smartphone camera to intelligently measure and identify the surrounding environment, and describes the detected object category, orientation, color, shape, size and other information through real-time voice broadcast.
[0106] Specifically, the present invention can receive commands input by the user through voice, such as "describe the surrounding environment" or "tell me what is around". The present invention will then control the camera to measure and identify the surrounding environment and objects according to the command. Through image processing technology, the object category can be identified, and the attributes such as orientation, color, shape and size can be measured. Then the identified object information will be converted into voice broadcast in real time and conveyed to the user. The user can understand the surrounding environment through the description heard, and improve the cognition and understanding of the surrounding environment.
[0107] The benefit of this is that it can improve the visually impaired's ability to perceive the surrounding environment: by controlling the camera through a smartphone, the camera can measure and identify, as well as the real-time voice broadcast function, the visually impaired can more intuitively understand the surrounding environment and enhance their ability to perceive the surrounding things. It can also help the visually impaired to obtain information about the surrounding environment independently, reduce dependence on others, and enhance autonomy and independence.
[0108] It can also provide real-time voice broadcasts so that the visually impaired can obtain information more conveniently without relying on other auxiliary tools, thus improving the user experience.
[0109] For example, Xiaohong is a visually impaired person. She uses the intelligent system of the present invention equipped on her mobile phone to identify the surrounding environment. When Xiaohong says the voice command "describe the surrounding environment", the camera will be activated for intelligent measurement and identification, and the category, orientation, color, shape and size of the objects in the surrounding environment will be described through real-time voice broadcast, such as "there is a rectangular table in front, the tabletop is red, and the length and width are 1 meter × 0.5 meters respectively". In this way, Xiaohong can understand the surrounding environment more accurately, improving her perception and independence of the environment. It provides better perception and independence for the visually impaired.
[0110] Furthermore, the navigation processing method for visually impaired persons based on a smart phone of the present invention further comprises the steps of:
[0111] When the user's voice question about the type of object is received, the camera is controlled to identify the objects within the field of view to search and feedback the results; based on the feedback results, it is determined whether the field of view of the smartphone camera needs to be moved to control the continued search, until the recognition is successful and feedback and confirmation are given, and then the direction guidance prompts are given.
[0112] In the embodiment of this step, the user can ask questions to search, and the present invention can obtain the user's voice questions. Specifically, the present invention can obtain the user's questions about the type of items through voice input, such as "find where the water cup near me is." Then, the camera is controlled to recognize and identify. Specifically, the camera can be controlled to recognize items within the viewing angle according to the user's questions, and try to find the items the user needs. The embodiment of the present invention will determine whether the target item is found based on the recognition results. If not found, it will decide whether it is necessary to adjust the camera's viewing angle to continue searching until the recognition is successful and feedback is given to the user for confirmation. When the system confirms that the target item has been found, it will convey the location of the target item to the user through voice prompts to help the user accurately locate it.
[0113] The advantage of the present invention is that the function enables the visually impaired to locate the target object by voice questioning, reducing the need for help from others and improving their autonomy and independence. And by controlling the camera to identify and locate the object, the user can quickly and conveniently find the required object, saving time and energy. The object-finding function of the present invention provides a smart way for the visually impaired to find objects, improves the user experience, and makes them more convenient and comfortable in their daily life.
[0114] For example, Mr. Wang is a visually impaired person. If he loses his keys at home, he needs to find them. Therefore, he can use the method of the present invention to ask a voice question: "Please find where the keys are near me". The method of the embodiment of the present invention controls the camera to identify and search for objects within the field of view. If the key is found, it will feedback confirmation information and prompt the location of the key, such as "Your key is on the table". If it is not found, the system of the present invention will adjust the viewing angle of the camera to continue searching until it successfully identifies and feedbacks the confirmed location, and then give specific directional guidance prompts, such as "Please move one step to the right front, the key is under your feet". In this way, through the new object-finding function realized by the method of the present invention, Mr. Wang can find his lost keys quickly and easily, which improves his independence and quality of life.
[0115] In the embodiment of the present invention, further, it also includes the steps of:
[0116] For navigation destinations that you have already visited, turn on the habit learning mode to remember the route and become familiar with it; when you navigate to the same destination next time, the familiar route mode will be automatically turned on to provide navigation reminders.
[0117] In an embodiment of the present invention, the habit learning mode of the navigation destinations that have been traveled, when implemented specifically, the present invention will record the walking route, characteristics and surrounding environment information of the destination that the user has navigated to before, to form a habit learning mode, so that the user can be more familiar with, reminded and guided the next time the user navigates to the same destination.
[0118] If the user navigates to the same destination next time, the present invention will automatically turn on the familiar route mode. Specifically: when the user chooses to navigate to a destination that has been visited before, the system of the embodiment of the present invention will automatically turn on the familiar route mode, and will remind and guide the user according to the previously learned route and environmental features when providing navigation, helping the user to reach the destination faster and more accurately.
[0119] The advantage of this is that, through the habit learning mode and the familiar route mode, the system of the present invention can provide more accurate navigation guidance based on the user's walking habits and known routes, helping the user to reach the destination accurately. It can also remember the user's habitual route, automatically remind and guide the user, reduce the risk of the user getting lost, and improve the user's navigation experience and convenience. The habit learning mode and the familiar route mode of the present invention can help the visually impaired navigate more independently, reduce dependence on external help, and improve their ability to take care of themselves.
[0120] For example, Xiao Ming is a visually impaired person who often needs to go to a nearby coffee shop to buy coffee. Every time Xiao Ming navigates to the coffee shop, the system of the present invention will record Xiao Ming's walking route, characteristics and surrounding environment information, and form a habitual learning mode. The next time Xiao Ming chooses to navigate to the coffee shop again, the system of the embodiment of the present invention will automatically turn on the familiar road mode, and during the navigation process, it will remind and guide according to the previously learned route and environmental characteristics, such as "There is a zebra crossing ahead, please pay attention to cross the road", "There is a flight of stairs at the door of the coffee shop, please be careful when going up and down". In this way, Xiao Ming can reach his destination faster and more conveniently, which improves his navigation experience and independence.
[0121] Based on the above embodiments, the present invention further describes in detail a navigation processing method for visually impaired persons based on a smart phone through a specific application embodiment of the present invention:
[0122] like Figure 2 As shown, a navigation processing method for visually impaired people based on a smart phone in this specific application embodiment includes the following steps:
[0123] Step S01, wake up, then enter S10 navigation function, S20 environment description function or S30 find object function respectively;
[0124] Step S10, start the navigation function and proceed to step S11;
[0125] The navigation function can be activated by voice.
[0126] Step S11, confirm the destination, and proceed to step S12;
[0127] Destination confirmation can be performed by voice.
[0128] Step S12, navigation starts, and proceeds to step S13;
[0129] Step S13, control the camera to identify objects / measure distance, and proceed to step S14;
[0130] Step S14, the system calculates / analyzes / screens, and proceeds to step S15;
[0131] Step S15, voice broadcast;
[0132] Step S20, start the environment description function, and proceed to step S21;
[0133] Step S21, the camera recognizes the object and proceeds to step S22;
[0134] Step S22, the system calculates / analyzes and proceeds to step S23;
[0135] Step S23: voice broadcast.
[0136] Step S30, start the object-finding function and proceed to step S31;
[0137] Step S31, the camera recognizes / measures the object, and then proceeds to step S32;
[0138] Step S32, the system calculates / analyzes and proceeds to step S33;
[0139] Step S33: Item confirmation. If the item is not confirmed, the process returns to step S31. If the item is confirmed, the process proceeds to step S34.
[0140] Step S34: location broadcast.
[0141] In a specific application embodiment of the present invention, the functions implemented include:
[0142] The first function (refer to steps S10-S15): Navigation in the visually impaired mode and assisting the visually impaired to go out. The visually impaired can open the map (which can be other APPs or developed maps) through the mobile phone voice assistant or human-computer interaction and select the destination (default walking) to start navigation. At the same time, the mobile phone camera function is turned on to act as the user's eyes (for example: through a specific application, the mobile phone camera can intelligently measure the distance and height of objects, and use computer vision technology or HUSKYLENS cameras to identify objects and classify them to achieve road, object, obstacle, pothole, etc.) The camera transmits the information obtained within a certain range (for example: the camera angle of view is 60 / 80 degrees, 10 meters ahead) to the mobile phone system for analysis, calculation, screening, judgment, and then voice broadcast through the mobile phone speaker or headset or Bluetooth protocol to prompt and assist the user to pay attention and walk.
[0143] The second function (refer to steps S20-S23): Provide users with a report of the surrounding environment in the visually impaired mode, which is generally suitable for indoor or outdoor meditation scenes. This function is implemented by describing the environment and things within a certain range around the user (within a radius of 5 meters, and the camera's viewing angle direction is adjusted by the user) according to the user's wake-up command requirements, such as: the type of objects within the 5-meter range of the camera's viewing angle and their orientation, color, shape, size, etc.
[0144] The third function (refer to steps S30-S34): Find items for users in the visually impaired mode. According to the type of item in the user's question, the system instructs the camera to identify the item within the viewing angle and feedback the result. For example, if the user asks where the key is, the mobile phone starts to search and identify through the camera and feedback. The user determines whether to move the viewing angle of the mobile phone camera according to the result in order to continue searching, until the item is identified and feedback and confirmation are given, and then the direction guidance is given.
[0145] In addition, the present invention can also realize the fourth function: emergency alarm / help, emergency contacts 1, 2, alarm, and the fifth function: familiar route memorization, memory, and learning mode.
[0146] During the first three functions mentioned above, human-computer interaction (wearable device interaction, interaction through voice) is achieved through a smartphone and is on standby at any time, with the camera and microphone performing both video and audio recording.
[0147] When the present invention is implemented, it can use computer vision technology to identify objects, laser radar sensor force distance measurement, stereoscopic perspective technology distance measurement, image processing algorithms, machine learning models to identify objects and measure distance, artificial intelligence algorithms, intelligent obstacle avoidance, etc. Human-computer interaction can use: gesture recognition, multimodal interaction.
[0148] It can be seen that the present invention is based on integrating new functions into mobile phones and developing a navigation processing method program for visually impaired people based on smart phones. The present invention utilizes the existing hardware configuration of mobile phones and the conditions of real-time access to the network / cloud / big database / integrated software algorithms to develop and apply the navigation function of mobile phones for visually impaired people, and add barrier-free navigation functions for visually impaired people. The present invention is easy to implement and has many functions, which can meet the needs of most visually impaired people.
[0149] Exemplary Devices
[0150] like Figure 3 As shown, an embodiment of the present invention provides a navigation processing device for visually impaired people based on a smart phone, the device comprising:
[0151] Voice navigation is turned on 310, and voice commands are obtained to turn on the navigation function and set the navigation destination;
[0152] The navigation broadcast module 320 is used to plan a navigation route according to the set navigation destination and start real-time navigation voice broadcast;
[0153] The surrounding object recognition module 330 is used to control the camera to intelligently measure the distance and height of objects around the navigation route during real-time navigation, identify objects and classify them, and realize the recognition of roads, objects, obstacles, and potholes;
[0154] The prompt module 340 is used to obtain information within a predetermined range from the camera and perform analysis, calculation, and screening; determine whether there are obstacles, potholes, and / or dangerous areas, and provide voice notifications to assist the user to pay attention and walk;
[0155] The surrounding environment detection and broadcasting module 350 is used to control the camera to intelligently measure and identify the environment and objects within the predetermined range around the camera when a voice command for describing the surrounding environment is obtained, and to perform real-time voice broadcasting of the object type and its orientation, color, shape, and size within the predetermined range around the camera;
[0156] The search prompt module 360 is used to control the camera to identify objects within the field of view and feedback the results when the user's voice question about the type of object is obtained; based on the feedback results, it is determined whether it is necessary to move the field of view of the smartphone camera to control the continued search until the recognition is successful and feedback and confirmation are given, and then control the direction guidance prompts, as described above.
[0157] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 4As shown. The smart terminal includes a processor, a memory, a network interface, a display screen, and a database connected through a system bus. Among them, the processor of the smart terminal is used to provide computing and control capabilities. The memory of the smart terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the smart terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a navigation processing method for visually impaired people based on a smart phone is implemented. The database of the smart terminal is used to store a navigation processing program for visually impaired people based on a smart phone.
[0158] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0159] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:
[0160] Get voice commands to start the navigation function and set the navigation destination;
[0161] Plan the navigation route according to the set navigation destination and start the real-time navigation voice broadcast;
[0162] During real-time navigation, the camera is controlled to intelligently measure the distance and height of objects around the navigation route, identify objects and classify them, and realize the recognition of roads, objects, obstacles, and potholes;
[0163] Obtain information within a predetermined range from the camera and perform analysis, calculation, and screening; determine whether there are obstacles, potholes, and / or dangerous areas, and use voice broadcasts to prompt and assist users to pay attention and walk, as described above.
[0164] The predetermined range is 60 / 80 degrees from the camera viewing angle and 10 meters in front.
[0165] In the real-time navigation process, the steps of controlling the camera to intelligently measure the distance and height of objects around the navigation route, identifying and classifying the objects, and realizing the identification of roads, objects, obstacles, and potholes also include:
[0166] When a voice command is received to describe the surrounding environment, the camera is controlled to intelligently measure and identify the environment and objects within the predetermined range around the camera, and a real-time voice broadcast is performed on the object category and its orientation, color, shape, and size within the predetermined range around the camera.
[0167] Among them, it also includes the steps of:
[0168] When the user's voice question about the type of item is obtained, the camera is controlled to identify the items within the viewing angle to search and feedback the results;
[0169] Based on the feedback results, it is determined whether it is necessary to move the viewing angle of the smartphone camera to control the continued search until successful recognition is obtained and feedback and confirmation are given, and then the control is used to provide direction guidance prompts.
[0170] Among them, in the real-time navigation process, the steps of controlling the camera to intelligently measure the distance and height of objects around the navigation route, identifying and classifying objects, and realizing the identification of roads, objects, obstacles, and potholes include:
[0171] When performing real-time navigation, the camera function is automatically activated to capture real-time images around the navigation route;
[0172] The image data obtained by the camera is used to intelligently measure the distance and height of objects using computer vision technology;
[0173] Perform object detection and recognition on the captured images, identify various objects in the images, and classify them; the identified objects will be classified into roads, ordinary objects, obstacles, and potholes;
[0174] Comprehensively analyze the results of recognition and classification, make corresponding navigation decisions based on road conditions and object location information, provide navigation guidance, and remind people to avoid risk areas such as obstacles and potholes.
[0175] The steps of obtaining information within a predetermined range from the camera and performing analysis, calculation, and screening; determining whether there are obstacles, potholes, and / or dangerous areas, and providing voice notifications to assist the user in paying attention and walking include:
[0176] Obtain information within a certain range that the camera can detect, including distance data, height data, shape data, and analyze it;
[0177] Computationally analyze the acquired information to determine the characteristics and location of surrounding objects and terrain;
[0178] After calculation, the information that is important or may affect the user's navigation is screened out, including obstacles, potholes and dangerous places;
[0179] Based on the screened information, determine whether there are obstacles, potholes or other dangerous factors in the surrounding environment and conduct risk assessment;
[0180] When obstacles, potholes or other dangerous situations are confirmed, users will be reminded through voice broadcasts and specific precautions and suggestions will be provided.
[0181] The steps of obtaining information within a predetermined range from the camera and performing analysis, calculation, and screening; determining whether there are obstacles, potholes, and / or dangerous areas, and providing voice notifications to assist the user in paying attention and walking also include:
[0182] For navigation destinations you have already visited, turn on the habit learning mode to memorize the route;
[0183] The control will automatically turn on the familiar route mode to provide navigation reminders for the next navigation to the same destination, as described above.
[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0185] In summary, the present invention provides a navigation processing method, device, smart terminal and storage medium for visually impaired people based on smart phones. Compared with ordinary advertising, the present invention adds new functions to the advertising terminal: it has the function of skipping advertisements by voice; by skipping advertisements as an attraction point, the user reads the name of the current advertisement key information by voice, thereby realizing the ability to skip advertisements and deepening the user's impression of the brand; for users, they skip watching lengthy advertisement content and improve user favorability; for advertisers, users read the key information of the product, deepen the brand impression and achieve the purpose of product promotion.
Claims
1. A navigation processing method for visually impaired people based on a smart phone, characterized in that: include: Get voice commands to start the navigation function and set the navigation destination; Plan the navigation route according to the set navigation destination and start the real-time navigation voice broadcast; During real-time navigation, the camera is controlled to intelligently measure the distance and height of objects around the navigation route, identify objects and classify them, and realize the recognition of roads, objects, obstacles, and potholes; Obtain information within a predetermined range from the camera and perform analysis, calculation, and screening; Determine whether there are obstacles, potholes and / or dangerous areas, and use voice broadcasts to prompt and assist users to pay attention and walk.
2. The navigation processing method for visually impaired people based on a smart phone according to claim 1, characterized in that: The predetermined range is 60 / 80 degrees from the camera's viewing angle and 10 meters in front.
3. The navigation processing method for visually impaired people based on a smart phone according to claim 1, characterized in that: In the real-time navigation process, the steps of controlling the camera to intelligently measure the distance and height of objects around the navigation route, identifying and classifying the objects, and realizing the identification of roads, objects, obstacles, and potholes also include: When a voice command is received to describe the surrounding environment, the camera is controlled to intelligently measure and identify the environment and objects within the predetermined range around the camera, and a real-time voice broadcast is performed on the object category and its orientation, color, shape, and size within the predetermined range around the camera.
4. The navigation processing method for visually impaired people based on a smart phone according to claim 1, characterized in that: It also includes the steps of: When the user's voice question about the type of item is obtained, the camera is controlled to identify the items within the viewing angle to search and feedback the results; Based on the feedback results, it is determined whether it is necessary to move the viewing angle of the smartphone camera to control the continued search until successful recognition and feedback and confirmation are given, and then the control provides direction guidance prompts.
5. The navigation processing method for visually impaired people based on a smart phone according to claim 1, characterized in that: In the real-time navigation process, the steps of controlling the camera to intelligently measure the distance and height of objects around the navigation route, identifying and classifying the objects, and realizing the identification of roads, objects, obstacles, and potholes include: When performing real-time navigation, the camera function is automatically activated to capture real-time images around the navigation route; The image data obtained by the camera is used to intelligently measure the distance and height of objects using computer vision technology; Perform object detection and recognition on the captured images, identify various objects in the images, and classify them; the identified objects will be classified into roads, ordinary objects, obstacles, and potholes; Comprehensively analyze the results of recognition and classification, make corresponding navigation decisions based on road conditions and object location information, provide navigation guidance, and remind people to avoid risk areas such as obstacles and potholes.
6. The navigation processing method for visually impaired people based on a smart phone according to claim 1, characterized in that: The information within a predetermined range from the camera is obtained and analyzed, calculated and screened; The steps of determining whether there are obstacles, potholes and / or dangerous areas, and providing voice notifications to assist the user to pay attention and walk include: Obtain information within a certain range that the camera can detect, including distance data, height data, shape data, and analyze it; Computationally analyze the acquired information to determine the characteristics and location of surrounding objects and terrain; After calculation, the information that is important or may affect the user's navigation is screened out, including obstacles, potholes and dangerous places; Based on the screened information, determine whether there are obstacles, potholes or other dangerous factors in the surrounding environment and conduct risk assessment; When obstacles, potholes or other dangerous situations are confirmed, users will be reminded through voice broadcasts and specific precautions and suggestions will be provided.
7. The method for navigation for visually impaired persons based on a smart phone according to claim 1, characterized in that: The information within a predetermined range from the camera is obtained and analyzed, calculated and screened; After determining whether there are obstacles, potholes and / or dangerous areas, and providing voice announcements to assist the user to pay attention and walk, the steps also include: For navigation destinations you have already visited, turn on the habit learning mode to memorize the route; The next time you navigate to the same destination, the familiar route mode will be automatically turned on for navigation reminders.
8. A navigation processing device for visually impaired people based on a smart phone, characterized in that: The device comprises: Voice navigation is turned on, get voice commands to turn on the navigation function and set the navigation destination; The navigation broadcast module is used to plan the navigation route according to the set navigation destination and start the real-time navigation voice broadcast; The surrounding object recognition module is used in real-time navigation to control the camera to intelligently measure the distance and height of objects around the navigation route, identify objects and classify them, and realize the recognition of roads, objects, obstacles, and potholes; The prompt module is used to obtain information within a predetermined range from the camera and perform analysis, calculation, and screening; determine whether there are obstacles, potholes, and / or dangerous areas, and provide voice notifications to assist users in paying attention and walking; The surrounding environment detection and broadcasting module is used to control the camera to intelligently measure and identify the environment and objects within the predetermined range around the camera when a voice command for describing the surrounding environment is obtained, and to perform real-time voice broadcasting of the object type and its orientation, color, shape, and size within the predetermined range around the camera; The search prompt module is used to control the camera to identify objects within the field of view and feedback the results when the user's voice question about the type of object is obtained; based on the feedback results, it is determined whether it is necessary to move the field of view of the smartphone camera to control the continued search until the recognition is successful and feedback and confirmation are given, and then the direction guidance prompt is given.
9. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in any one of claims 1 to 7.
