Blind person object grabbing navigation system based on intelligent glasses

Through the integrated object detection, single-double eye ranging and speech recognition technology of smart glasses, the problem of blind people grabbing objects is solved, real-time and accurate object recognition and grasping guidance is achieved, and the quality of life and convenience of blind people are improved.

CN119027792AInactive Publication Date: 2024-11-26THE ACAD OF TIANJIN UNIV HEFEI

Patent Information

Application Number
CN202411142263.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing technology lacks a system to guide blind people to actually grasp objects by hand, and it is difficult for blind people to accurately obtain and grasp daily necessities in daily life.

Method used

Smart glasses are used to integrate object detection, single-double eye ranging and voice recognition technology, and through voice wake-up, object detection, distance measurement and path planning, blind people are guided to grasp objects to ensure accuracy and safety.

Benefits of technology

The blind people's ability to identify and grasp objects in real time in daily life has been realized, the quality of life has been improved, and the real-time and psychological needs have been met through lightweight models and a wide range of applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027792A_ABST
    Figure CN119027792A_ABST
Patent Text Reader

Abstract

The invention discloses a blind person object grabbing navigation system based on intelligent glasses, which comprises computer vision, artificial intelligence and monocular and binocular distance measurement technologies, can be widely applied to visual impaired people to grab objects in daily life, is wide in application environment, can recognize various objects, and can be used for recognizing various objects in consideration of complex terrains. Obstacles in a path can be monitored in real time, and a user is prompted to avoid. The blind person grabbing device greatly meets the requirement of the blind person for obtaining daily necessities in the actual life, can meet the object grabbing requirement of the blind person in the daily life, and brings substantial improvement to the daily life of the blind person.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction and intelligent wearable technology, including computer vision, artificial intelligence, and monocular and binocular ranging technology, and in particular to an object grasping and navigation system for the blind based on smart glasses. Background Art

[0002] With the widespread application of information technology, the application of wearable devices and target detection technology is becoming more and more widespread, and applications based on these technologies are also penetrating into all aspects of industry and life. At present, many applications for indoor navigation, face recognition, and object recognition for the blind have been developed, but there is a lack of a system to guide the blind to actually grasp objects with their hands. The blind not only need to know the location of the object, but also need to meet the application needs of actually grasping the object. Therefore, based on practical requirements, smart glasses are selected as the basic equipment to realize the function of guiding the blind to grasp objects with their hands. The technology can be mainly divided into three directions: real-time target detection, single and binocular ranging, and speech recognition technology.

[0003] Real-time object detection technology is a key component of this research, including efficient single-stage and two-stage detection methods. Single-stage methods such as YOLO and SSD optimize the balance between processing speed and accuracy by predicting the location and category of the target directly from the image, and are suitable for application environments that require fast response. YOLO transforms object detection into a regression problem, predicting bounding boxes and category probabilities directly from image pixels. SSD performs detection on feature maps at multiple scales to improve detection accuracy. Two-stage methods such as Faster R-CNN first generate candidate regions, and then accurately classify and predict the location of each region. Their accuracy is usually higher than that of single-stage methods, and they are particularly suitable for scenarios with high accuracy requirements.

[0004] Monocular ranging technology relies on analyzing visual clues in a single image to estimate the distance of an object. This method is low-cost and simple to implement, but its accuracy is limited by the quality of the image and the complexity of the processing algorithm. Binocular ranging technology uses two cameras to capture images from slightly different perspectives and calculates the distance of the object by comparing the parallax of the two images. This method can provide more accurate distance data and is suitable for applications that require high-precision spatial information, but it requires higher computing resources and algorithm optimization.

[0005] Speech recognition technology also plays an important role in this topic. It enables the system to interact with users through natural language processing, which is the most important interaction method for blind people with visual impairments. Speech input allows users to communicate with the system through oral instructions, increasing the usability and interactivity of the system. Speech output enables the system to convert complex data and operation instructions into easy-to-understand language feedback, helping visually impaired people understand the surrounding environment and perform necessary operations. The application of this technology not only improves the user experience, but also makes the application of technology more extensive and in-depth, especially showing its important value in the design of assistive devices.

[0006] Therefore, under the existing technical background, it is expected that a system that can guide the blind to grasp objects with their hands can be realized. The system recognizes the needs of the blind through voice, detects objects that can be obtained in the environment, measures the direction and distance of the object from the blind, and guides the blind to approach and grasp the object, thus filling the gap in application in this area. Summary of the invention

[0007] The purpose of the present invention is to make up for the defects of the existing technology and provide a blind object grasping navigation system based on smart glasses. The present invention greatly meets their needs for obtaining daily necessities in real life, can realize the blind's need for object grasping in daily life, and bring substantial improvements to their daily life.

[0008] The present invention is achieved through the following technical solutions: A blind object grasping and navigation system based on smart glasses includes a target detection module, a monocular and binocular ranging module, a speech recognition module, a speech wake-up module and a speech prompt module. The specific contents are as follows: Step 1-1, the user wakes up the system through the voice wake-up module and starts environmental object detection; Step 1-2, identifying the obtainable objects in the environment through the target detection module, and prompting the user of the obtainable objects in the environment through the voice prompt module, and waiting for further instructions from the user; Step 1-3: The user sends a voice command to specify the object to be grabbed. The voice recognition module obtains the voice command information and converts it into valid characters to determine the target object. The target detection module determines whether the target object is in the detected object information library. If it exists, it proceeds to the next step; if not, the voice prompts the user to reselect; Steps 1-4, the monocular and binocular ranging module calculates the distance from the target object to the user's head by calibrating the pixel size of the object and the target object in the field of view, and divides the distance into coarse and fine granularity; at a longer distance, the coarse granularity is used to guide the user to approach the object, and at a closer distance, the fine granularity is used to guide the user to grasp the object, and the specific position of the target object in the user is calculated based on the pixel coordinates of the target object in the canvas, and the path of the user to move to the target object is calculated; when the target object does not pass through the central axis of the field of view and is located on the left or right side of the field of view, the smart glasses guide the user to turn left or right through voice so that the target object is located on the central axis of the field of view, that is, the user is facing the target object at this time; when the user is facing the target object, the smart glasses guide the user to move forward through voice until it is detected that the target object deviates from the central axis of the field of view again; repeat the above steps until the user reaches the vicinity of the target object; Step 1-5: When the single-eye or binocular ranging module detects that the blind person is near the target object, the user is guided by voice to stretch out his palm. The target detection module detects the target object and the user's palm at the same time, and plans the path for hand grasping, and guides the user to grasp the object by voice. Step 1-6: In order to prevent the user from colliding with obstacles during movement, the target detection module monitors in real time whether there are other objects in the path. If there are, the voice prompt module prompts the user to avoid and re-plan the route; Step 1-7: After the user successfully grabs the target object, the system resets and waits for the next voice wake-up.

[0009] The target detection module, monocular and binocular ranging module, speech recognition module, speech wake-up module and speech prompt module all run on the built-in chip of the smart glasses, wherein the target detection module and monocular and binocular ranging module call the built-in camera of the smart glasses, and the speech recognition module, speech wake-up module and speech prompt module call the microphone and earpiece of the smart glasses.

[0010] The target detection module uses a camera to capture the environment image, and the target detection module detects the obtainable objects in the environment and obtains the label name corresponding to each object.

[0011] In step 1-5, when the monocular and binocular ranging modules detect that the blind person is near the target object, the user is guided by voice to stretch out his palm. The target detection module simultaneously detects the target object and the user's palm, and plans the path for the hand to grasp, and guides the user to grasp the object by voice. The specific steps are: first, the monocular and binocular ranging modules detect the pixel position of the target object in the field of view image, and then detect the pixel position of the hand in the field of view image, and compare the positions, and guide the palm to move up, down, left and right according to the relative orientation between the target object and the hand. When the palm is detected to pass through the center of the target object, the voice prompts the palm to stop moving. At this time, the position of the palm and the target object coincide, and the palm can be stretched forward to grasp the object.

[0012] The advantages of the present invention are: (1) The model is lightweight and has strong real-time performance. The present invention integrates multiple lightweight models into smart glasses. The system can run on the glasses itself without the need for cloud data transmission, which greatly meets the real-time requirements; (2) Wide range of applications. The present invention can be widely used in daily life for visually impaired people to grasp objects, and has a wide range of application environments and can identify a variety of objects. At the same time, considering complex terrain, it can monitor obstacles in the path in real time and prompt users to avoid them; (3) Taking care of the psychological needs of special groups. The present invention is different from other wearable devices. Glasses are common wearable items in daily life. Visually impaired people generally do not want to be looked at strangely in life. Therefore, the system based on smart glasses can take into account the psychological identity of special groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the system module diagram; Figure 2 It is a picture of smart glasses; Figure 3 is the target detection map; Figure 4 It is an operation diagram. DETAILED DESCRIPTION

[0014] like Figure 1 As shown in the figure, a blind object grasping and navigation system based on smart glasses includes a target detection module, a monocular and binocular ranging module, a voice recognition module, a voice wake-up module and a voice prompt module. These modules are integrated on the smart glasses. The smart glasses are low-cost and easy to wear. The technical principles of the smart glasses used are clear and definite, and no complex materials are required. They can be mass-produced, and users can wear glasses at any time without carrying complex and bulky equipment. Figure 2 As shown in Figure 3 As shown, taking the target object as a bottle as an example.

[0015] like Figure 4 As shown, the specific operation steps are as follows: Step 1: The camera captures the environment image, and the object detection module detects the graspable objects in the environment and obtains the label name corresponding to each object.

[0016] Step 2: Output the names of existing graspable objects through voice, and wait for the user to input voice.

[0017] Step 3: The user performs voice input, and if the target object exists in the image, distance detection is performed. If the target object does not exist, the user is prompted.

[0018] Step 4: Detect the distance between the target object and the user. If it is greater than the specified distance, it is classified as coarse-grained. If it is less than the specified distance, it is classified as fine-grained.

[0019] Step 5, when the user is in the coarse-grained range, the system infers the actual position of the object from the user through the distance between the object and the user and the pixel coordinates of the object in the canvas, and calculates a path; when the target object does not pass through the central axis of the field of view, but is located on the left or right side of the field of view, the smart glasses guide the user to turn left or right through voice so that the target object is located on the central axis of the field of view, that is, the user is facing the target object at this time; when the user is facing the target object, the smart glasses guide the user to move forward through voice until it is detected that the target object deviates from the central axis of the field of view again; repeat the above steps until the user reaches the vicinity of the target object; Step 6: During the user's movement, monitor in real time whether there are other objects near the central axis of the canvas. If there are, they are considered obstacles and the user is prompted to avoid them.

[0020] Step 7: When the user is less than a certain distance from the object, the process switches to fine-grained mode and prompts the user to stretch out his hand. The camera detects both the hand and the object at the same time.

[0021] In step 8, the system calculates the path of the hand to the object based on the distance and orientation of the hand and the object from the camera, and provides movement guidance. The specific steps are: first, the pixel position of the target object in the field of view image is detected, then the pixel position of the hand in the field of view image is detected, and the position is compared. The palm is guided to move up, down, left and right by the relative orientation between the target object and the hand. When the palm is detected to pass the center of the target object, the voice prompts the palm to stop moving. At this time, the palm and the target object coincide, and the palm can be stretched forward to grab the object.

[0022] Step 9: When the hand successfully grasps the object, the navigation ends, the system resets, and waits for the user to wake up next time.

Claims

1. A blind object grasping and navigation system based on smart glasses, characterized by: It includes target detection module, monocular and binocular ranging module, speech recognition module, speech wake-up module and speech prompt module. The specific contents are as follows: Step 1-1, the user wakes up the system through the voice wake-up module and starts environmental object detection; Step 1-2, identifying the obtainable objects in the environment through the target detection module, and prompting the user of the obtainable objects in the environment through the voice prompt module, and waiting for further instructions from the user; Step 1-3: The user sends a voice command to specify the object to be grabbed. The voice recognition module obtains the voice command information and converts it into valid characters to determine the target object. The target detection module determines whether the target object is in the detected object information library. If it exists, it proceeds to the next step; if not, the voice prompts the user to reselect; Steps 1-4: The monocular and binocular ranging module calculates the distance from the target object to the user's head by calibrating the pixel sizes of the object and the target object in the field of view, and divides the distance into coarse and fine granularity. At a longer distance, the coarse granularity is used to guide the user to approach the object, and at a closer distance, the fine granularity is used to guide the user to grasp the object with his hand. The specific position of the target object in the user is calculated based on the pixel coordinates of the target object in the canvas, and the path of the user to move to the target object is calculated. When the target object does not pass through the central axis of the visual field, but is located on the left or right side of the visual field, the user is guided by voice to turn left or right so that the target object is located on the central axis of the visual field, that is, the user is facing the target object at this time; when the user is facing the target object, the user is guided by voice to move forward until it is detected that the target object deviates from the central axis of the visual field again; the above steps are repeated until the user reaches the vicinity of the target object; Step 1-5: When the single-eye or binocular ranging module detects that the blind person is near the target object, the user is guided by voice to stretch out his palm. The target detection module detects the target object and the user's palm at the same time, and plans the path for hand grasping, and guides the user to grasp the object by voice. Step 1-6: In order to prevent the user from colliding with obstacles during movement, the target detection module monitors in real time whether there are other objects in the path. If there are, the voice prompt module prompts the user to avoid and re-plan the route; Step 1-7: After the user successfully grabs the target object, the system resets and waits for the next voice wake-up.

2. The object grasping and navigation system for the blind based on smart glasses according to claim 1 is characterized in that: The target detection module, monocular and binocular ranging module, speech recognition module, speech wake-up module and speech prompt module all run on the built-in chip of the smart glasses, wherein the target detection module and monocular and binocular ranging module call the built-in camera of the smart glasses, and the speech recognition module, speech wake-up module and speech prompt module call the microphone and earpiece of the smart glasses.

3. The object grasping and navigation system for the blind based on smart glasses according to claim 2 is characterized in that: The target detection module uses a camera to capture the environment image, and the target detection module detects the obtainable objects in the environment and obtains the label name corresponding to each object.

4. The object grasping and navigation system for the blind based on smart glasses according to claim 2 is characterized in that: In step 1-5, when the monocular and binocular ranging modules detect that the blind person is near the target object, the user is guided by voice to stretch out his palm. The target detection module simultaneously detects the target object and the user's palm, and plans the path for the hand to grasp, and guides the user to grasp the object by voice. The specific steps are: first, the monocular and binocular ranging modules detect the pixel position of the target object in the field of view image, and then detect the pixel position of the hand in the field of view image, and compare the positions, and guide the palm to move up, down, left and right according to the relative orientation between the target object and the hand. When the palm is detected to pass through the center of the target object, the voice prompts the palm to stop moving. At this time, the position of the palm and the target object coincide, and the palm can be stretched forward to grasp the object.

Citation Information

Patent Citations

  • Intelligent blind assisting system and method based on auditory and tactile guidance

    CN110559127A

  • Blind guiding system and method based on smart phone image recognition

    CN116370277A

  • Blind aid navigation system for real time translation of visual depth information to auditory speech

    IN201741010332A

  • Training of vehicles to improve autonomous capabilities

    WO2019095013A1

Cited By

  • Blind person auxiliary system based on artificial intelligence and smart home

    CN122172611A