A blind real-time object searching guiding method and system based on binocular vision and target detection

Abstract

The application belongs to the cross field of assistive technology and computer vision, and discloses a blind real-time object searching guiding method and system based on binocular vision and target detection. The method receives and analyzes the object searching instruction of the user through voice interaction; the image is collected by using a binocular camera, and the target object and the palm of the user are identified by using an improved YOLO target detection model; the three-dimensional coordinates of the target object are calculated by using an optimized stereo matching algorithm; whether the palm is detected or not, the three-dimensional coordinates are converted into a clock direction and distance description based on the first visual angle of the user as a reference system, and a voice guiding instruction is generated; meanwhile, safety monitoring is carried out, and a warning is triggered when dangerous objects are approached; through real-time voice broadcast and cyclic update, the user is guided until the object is successfully grabbed, and multiple rounds of tasks are supported. The application combines visual perception, spatial positioning and intuitive direction description, and provides accurate, safe and intuitive indoor object searching guidance for visually impaired people.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of assistive technology and computer vision, specifically relating to a method and system for real-time object-finding guidance for blind people based on binocular vision and target detection. Background Technology

[0002] Visually impaired individuals face significant challenges in finding personal belongings in indoor settings such as homes and offices due to their lack of visual perception. Unlike the struggles of the general population, visually impaired individuals cannot quickly scan their environment and locate objects using their vision. They must rely solely on other senses such as touch and hearing, resulting in extremely low efficiency in finding items and increasing the risk of accidentally touching dangerous objects such as knives or hot water cups during the search process, thus posing safety hazards.

[0003] Existing object-finding assistance technologies for visually impaired individuals can be mainly categorized as follows: RFID-based auxiliary technologies: Location is achieved by attaching RFID tags to items and using handheld readers. However, this technology requires additional tags for items, which is costly and the tags are easy to fall off or lose. At the same time, the positioning accuracy is limited, and it can only achieve regional positioning, without providing accurate location and distance information. Assistive devices based on single vision or GPS, such as some smart glasses products, can only provide feedback on the presence or absence of a target object by capturing images and recognizing objects through a monocular camera, lacking spatial positioning capabilities; while GPS positioning technology is affected by indoor environmental obstructions, resulting in extremely large positioning errors, which cannot meet the accuracy requirements for indoor object finding at all. Traditional voice guidance technology: It broadcasts the location information of the item through voice, but this technology cannot dynamically update the location of the item. When the item is moved, the guidance fails, and it does not take into account the safety warning needs during the item search process, which poses a safety hazard.

[0004] In addition, existing technologies also have the following core defects: Location descriptions are not intuitive: Most technologies use coordinates or abstract angles to describe the location of objects, which are difficult for visually impaired people to quickly understand and translate into action instructions; Fixed and rigid reference system: The reference system is only based on the device or a fixed point and does not dynamically switch according to the user's body movements (such as reaching out), which makes the guidance accuracy greatly affected by the user's posture. Lack of safety protection mechanisms: Real-time safety warning functions were not set up to address the weak perception of dangerous items by visually impaired individuals; Incoherent interaction flow: Most technologies only support single-round item search tasks and cannot achieve multi-round continuous item search, resulting in a poor user experience.

[0005] Therefore, there is an urgent need for a blind person's object-finding guidance technology that integrates real-time visual perception, precise spatial positioning, intuitive orientation description and safety warning functions, in order to solve the problems of low positioning accuracy, unintuitive guidance, lack of safety protection and disjointed interaction process in the existing technology, and help visually impaired people to efficiently and safely find objects indoors, thereby improving their autonomy and convenience in life. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time object-finding guidance method and system for blind people based on binocular vision and target detection, which solves the technical problems of low positioning accuracy, unintuitive orientation description, lack of safety protection and disjointed interaction process in the prior art.

[0007] This invention is achieved through the following technical solution: This invention discloses a real-time object-finding guidance method for blind people based on binocular vision and object detection, comprising the following steps: Step 1: Monitor ambient audio in real time, and preset the wake-up command as a specific keyword; when the wake-up command is recognized, enter the command receiving mode, collect the user's voice command for finding the item, convert the voice command for finding the item into a text command, and parse the name of the target item to be found from the text command. Step 2: Real-time acquisition of binocular images of the indoor scene; use the improved YOLO object detection model to identify the category and two-dimensional pixel position of all objects in the scene; and determine whether the user's palm is detected. Step 3: Based on the optimized binocular stereo matching algorithm, perform stereo matching on the indoor scene binocular images acquired in Step 2, and calculate the three-dimensional spatial coordinates of the target object and the user's hand in the camera coordinate system; the camera coordinate system is relatively fixed relative to the user's first viewpoint. Step 4: If the user's palm is not detected in step 2, the camera's optical center is used as the reference point; if the user's palm is detected, the three-dimensional spatial coordinates of the user's palm obtained in step 3 are used as the reference point. Step 5: Calculate the three-dimensional spatial coordinates of the target item relative to the selected reference point based on the three-dimensional spatial coordinates of the selected reference point and the target item; calculate the horizontal azimuth, straight-line distance and vertical coordinates based on the three-dimensional spatial coordinates of the target item relative to the selected reference point, map the horizontal azimuth to the clock coordinate system, and generate voice guidance information including clock direction, vertical direction up / down and straight-line distance. A list of hazardous materials is pre-stored, and the three-dimensional Euclidean distance between the selected reference point and each item in the list is calculated. If any distance is less than a preset safety threshold, a voice warning message containing the name, location, and distance of the hazardous material is generated, and the voice warning message has a higher priority than the voice guidance message. Step 6: Real-time broadcast of the voice guidance and voice warning information generated in Step 5, and continuously loop through Steps 2 to 5 during this process, dynamically updating the item location information until the task ends and enters the next task or standby state.

[0008] Furthermore, in step 2, the improved YOLO object detection model is a model based on the improved YOLOv8 architecture, and the specific improved structure is as follows: Replace the original backbone network with an EfficientNetV1 network; Embed convolutional block attention modules in the neck network of the original model; The detection head is reconstructed using a variety of branch modules, with three parallel branches to process high semantic features, medium semantic features, and low semantic features respectively.

[0009] Furthermore, in step 3, the optimized binocular stereo matching algorithm is a post-processed optimized semi-global block matching algorithm. The specific optimization process is as follows: The original disparity map output by the semi-global block matching algorithm is standardized, and negative disparity points marked as invalid are filtered out. The disparity map after initial filtering is first subjected to median filtering to remove salt-and-pepper noise, and then bilateral filtering is performed to smooth the disparity map while preserving object edge information. For the disparity hole regions caused by occlusion after filtering, interpolation is performed to fill them based on the disparity values ​​of the effective pixels in the surrounding neighborhood.

[0010] Furthermore, in step 2, binocular images of the indoor scene are acquired in real time using a binocular camera fixed to the user's wearable device; In step 3, the three-dimensional spatial coordinates of the target object and the user's hand in the camera coordinate system are calculated, specifically as follows: The intrinsic parameter matrix K is obtained through camera calibration, and the expression for the intrinsic parameter matrix K is: K =

[0011] in, , These are the focal lengths of the camera in the x and y directions, respectively. , The coordinates of the camera's principal point; Let the pixel coordinates of the target object in the left camera image be ( , The corresponding matching pixel coordinates in the right camera image are ( , If the disparity is d, then the disparity value d = d =

[0012] Based on the principle of triangulation, the formula for calculating the three-dimensional coordinates (X, Y, Z) of the target object in the camera coordinate system is as follows:

[0013] Where b is the horizontal baseline length of the binocular camera, Z is the depth of the target object from the optical center of the camera, and X and Y are the coordinate offsets in the horizontal and vertical directions, respectively. The three-dimensional spatial coordinates of the user's palm in the camera coordinate system are obtained by substituting the corresponding pixel coordinates of the user's palm in the left and right camera images into the calculation formula.

[0014] Furthermore, in step 5, mapping the horizontal azimuth angle to the clock coordinate system specifically involves: Based on the three-dimensional spatial coordinates of the selected reference point and the three-dimensional spatial coordinates of the target object, calculate the three-dimensional spatial coordinates of the target object relative to the selected reference point. , , ),based on( , , Calculate the horizontal azimuth angle and the straight-line distance D; The value range is -90° to 90°, where 0° corresponds to the user's direct front, 90° corresponds to the user's direct right, and -90° corresponds to the user's direct left; the calculation formula is: ; Straight-line distance D,

[0015] Based on vertical coordinates Determining the positive or negative sign in the vertical direction: Slightly above average. Slightly lower; The clock coordinate system uses the direction directly in front of the user as 12 o'clock, the direction directly to the left as 9 o'clock, and the direction directly to the right as 3 o'clock. The horizontal azimuth angle is mapped onto this clock coordinate system to generate voice guidance information.

[0016] Furthermore, in step 5, the dangerous goods warning specifically includes: The system calculates the distance between the user and items in a pre-set list of hazardous materials in real time and generates warning messages, specifically: After each target identification and 3D coordinate calculation, the 3D Euclidean distance between the selected reference point and each hazardous item in the list is calculated using the following formula:

[0017] in,( , , ( ) represents the three-dimensional spatial coordinates of the hazardous material; , , () represents the three-dimensional spatial coordinates of the selected reference point; If the distance of any dangerous item If a preset safety threshold is reached, a warning message will be generated and broadcast first. When the warning message is broadcast, the guidance message will be paused until the user moves away from the dangerous item.

[0018] Furthermore, the list of dangerous items includes knives, hot water cups, sharp utensils, power strips, and lighters.

[0019] This invention also discloses a real-time object-finding guidance system for the blind based on binocular vision and object detection, comprising: The processing unit is used to perform all the calculations and logic scheduling for target detection, 3D coordinate calculation, reference frame selection, orientation transformation, security monitoring and voice interaction control. The image acquisition module consists of a pair of binocular cameras, which are electrically connected to the processing unit and are used to acquire synchronized left and right view scene images in real time; the binocular cameras are fixed relative to the user's first view when worn. The voice interaction module integrates a microphone and a speaker, and is electrically connected to the processing unit. It is used to collect user voice commands and broadcast guidance and warning information. A network communication module, connected to the processing unit, is used to enable the system to access the Internet in order to call the cloud-based speech recognition service interface; The power supply module is electrically connected to each functional module and is used to supply power to the system.

[0020] Furthermore, the system is integrated into a wearable glasses device, and the binocular camera is fixed to the front end of the frame of the wearable glasses.

[0021] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described thereon.

[0022] Compared with the prior art, the present invention has the following beneficial technical effects: This invention discloses a real-time object-finding guidance method for blind people based on binocular vision and target detection. Through the technical route of "voice command parsing - binocular image acquisition and YOLO detection - stereo matching and three-dimensional coordinate calculation - dynamic reference system selection - clock coordinate system orientation mapping and voice guidance - dangerous item warning - cyclic update and multi-round interaction", it realizes the whole process guidance for visually impaired people to find objects indoors. The combination of binocular vision and stereo matching algorithms enables the system to acquire the precise 3D spatial coordinates of the target object in the camera coordinate system, providing a data foundation for subsequent orientation calculations. The dynamic reference system selection strategy allows the guidance reference to adaptively switch based on whether the user reaches out, ensuring rapid positioning based on the viewpoint at long distances and precise guidance based on the hand at close distances. The "clock coordinate system" orientation mapping converts the abstract horizontal azimuth angle into "time of the hour" directions easily understood by visually impaired individuals, and, combined with distance and vertical offset descriptions, reduces cognitive burden. The hazardous material warning mechanism monitors the distance between the reference point and hazardous materials in real time during guidance, prioritizing the broadcast of warning information when the distance is less than a safety threshold, providing safety assurance for visually impaired individuals. The cyclic update mechanism ensures the real-time effectiveness of guidance information when the user moves or the object's position changes. Multi-round interaction and low-power standby mode enable the system to continuously complete multiple object-finding tasks and extend device battery life. These technical features work together to solve the technical problems of insufficient positioning accuracy, abstract orientation descriptions, fixed and rigid reference systems, lack of safety protection, and inconsistent interaction processes in existing technologies.

[0023] Furthermore, the improved YOLO object detection model's specific structure involves replacing the original backbone network with an EfficientNetV1 network, embedding convolutional block attention modules in the neck network, and reconstructing the detection head using diverse branch modules with three parallel branches to process high, medium, and low semantic features respectively. This technical solution enhances feature extraction capabilities while maintaining a moderate increase in model parameters through a composite scaling strategy of the backbone network; it enhances the feature saliency of small objects in complex backgrounds through a dual weighting mechanism of channel attention and spatial attention; and it enables the model to simultaneously adapt to the detection needs of large, medium-sized, small, and partially occluded objects by processing semantic features at different scales through multi-branch parallel processing. The information fusion of the three branches improves the localization accuracy of bounding boxes for objects with an occlusion area of ​​no more than 40%. These improvements work together to significantly improve the recognition accuracy of small indoor objects and partially occluded objects while maintaining real-time inference speed, providing reliable two-dimensional pixel position input for subsequent 3D coordinate calculation and orientation guidance.

[0024] Furthermore, the specific post-processing steps of the optimized binocular stereo matching algorithm are defined. These steps involve: standardizing the original disparity map output by the semi-global block matching algorithm, filtering out invalid negative disparity points, using median filtering to remove salt-and-pepper noise, using bilateral filtering to smooth the disparity map while preserving edge information, and using neighborhood interpolation to fill disparity holes caused by occlusion. This technical solution unifies the disparity map data range through standardization, facilitating subsequent filtering operations; filtering out negative disparity points avoids interference from invalid data in the calculation results; median filtering effectively removes isolated salt-and-pepper noise points; bilateral filtering smooths the disparity map while maintaining the sharpness of object edges, avoiding target boundary blurring due to over-smoothing; and the neighborhood interpolation filling strategy completes disparity hole areas, ensuring the integrity of the disparity map and the continuity of subsequent 3D coordinate calculations. These optimization steps are interconnected, collectively improving the quality of the disparity map and providing a reliable disparity data foundation for accurately calculating the 3D spatial coordinates of target objects and the user's hand in the camera coordinate system.

[0025] Furthermore, a formula for calculating 3D spatial coordinates based on the principle of binocular vision triangulation is defined. The intrinsic parameter matrix is ​​obtained through camera calibration, and the disparity value is calculated using the corresponding pixel coordinates of the target object in the left and right camera images. Then, based on triangulation relationships, the 3D coordinates (X, Y, Z) of the target object in the camera coordinate system are calculated. The same formula is used to obtain the 3D spatial coordinates of the user's hand. This technical solution, through the geometric constraints of binocular vision, transforms the differences in pixel coordinates in 2D images into depth information in 3D space, achieving precise quantification of the 3D spatial coordinates of the target object and the user's hand in the camera coordinate system. This provides high-precision 3D spatial data for subsequent reference frame selection, relative orientation calculation, and distance measurement, with the 3D spatial positioning error controllable within 3 cm.

[0026] Furthermore, the specific implementation method of dynamic reference frame selection is defined: when the user's palm is not detected, the camera's optical center is used as the reference point; when the user's palm is detected, the three-dimensional spatial coordinates of the user's palm obtained in step 3 are used as the reference point. This technical solution adaptively switches the guidance reference benchmark by detecting whether the user reaches out. When the user does not reach out, the camera's optical center (i.e., the user's head / eye position) is used as the reference point for long-distance orientation guidance, helping the user quickly determine the approximate direction of the target object; when the user reaches out, it automatically switches to using the palm as the reference point, enabling the user to accurately perceive the position of the target object relative to their hand, guiding them to grasp it precisely. This intelligent adaptation mechanism, which requires no manual intervention from the user, not only conforms to the natural object-finding behavior of visually impaired users—"first visually locate from a distance, then reach out and feel"—but also solves the problem of insufficient close-range guidance accuracy caused by the fixed and rigid reference frame in existing technologies.

[0027] Furthermore, the specific implementation method of the clock coordinate system mapping is defined. Specifically, the three-dimensional spatial coordinates of the target object relative to the reference point are calculated based on the three-dimensional spatial coordinates of the selected reference point and the target object. Based on these relative coordinates, the horizontal azimuth angle θ and the straight-line distance D are calculated. The vertical direction is determined by the sign of ΔY, and the horizontal azimuth angle θ is mapped to a clock coordinate system with the user's front being 12 o'clock, the left being 9 o'clock, and the right being 3 o'clock. This technical solution maps continuous horizontal azimuth angles θ to discrete clock directions according to specific intervals, combined with a straight-line distance D accurate to centimeters and a clear "upper / lower" vertical direction description, converting abstract spatial coordinate data into natural language descriptions of "o'clock position, centimeters, upper or lower." This intuitive way of expressing orientation allows visually impaired individuals to quickly understand the spatial position of the target object relative to themselves or their hand without coordinate conversion or angle understanding, and without additional learning costs. This allows them to translate the position into specific action commands (such as reaching to the left front or feeling to the right lower), effectively solving the problem of unintuitive orientation descriptions in existing technologies.

[0028] Furthermore, the specific implementation method of the dangerous item warning is defined. After each target identification and 3D coordinate calculation, the 3D Euclidean distance between the selected reference point and each item in the dangerous item list is calculated. When any distance is less than a preset safety threshold, a warning message is immediately generated and prioritized for broadcast, while the guidance information broadcast is paused until the user moves away from the dangerous item. This technical solution continuously monitors the spatial distance between the reference point (camera optical center or hand) and dangerous items such as knives and hot water cups during the guidance process, and makes real-time judgments based on preset safety thresholds. When the user approaches a dangerous item, a warning message containing the name, location, and distance of the dangerous item is issued promptly. The warning message has a higher priority than the guidance information, ensuring that safety warnings are not overridden by the guidance voice. Upon receiving the warning, the user can immediately stop their current action and take evasive action. This mechanism provides proactive safety protection for visually impaired individuals who have a weaker ability to perceive dangerous items, effectively preventing injuries caused by touching dangerous items during the search process. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the workflow of a real-time object-finding guidance method for blind people based on binocular vision and target detection according to the present invention. Figure 2 This is a diagram of the improved YOLOv8 target detection model network structure of the present invention; Figure 3 This is a schematic diagram illustrating the stereo matching vision imaging principle of the binocular camera of the present invention; Figure 4 This is a schematic diagram of the clock coordinate system orientation mapping relationship of the present invention; Figure 5 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the present invention, and not all of them.

[0031] The detailed description of the embodiments of the present invention provided in the following figures is not intended to limit the scope of the claimed invention, but merely to illustrate one selected embodiment of the invention. All other embodiments obtained by those skilled in the art based on the figures and embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0032] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0033] Example 1 like Figure 1 As shown, this invention discloses a real-time object-finding guidance method for blind people based on binocular vision and object detection, including the following steps: Step 1: Monitor ambient audio in real time and preset the wake-up command as a specific keyword; when the wake-up command is recognized, enter the command receiving mode, collect the user's voice command for finding the item through the microphone of the voice interaction module, call the cloud voice recognition service interface to convert it into a text command, and parse the name of the target item to be found from the text command. Step 2: In real time, the binocular camera fixed to the user's wearable device acquires binocular images of the indoor scene. The improved YOLO object detection model is used to identify the category of all objects in the scene and their two-dimensional pixel positions. At the same time, it is determined whether the user's palm is detected. Step 3: Based on the optimized binocular stereo matching algorithm, perform stereo matching on the binocular images acquired in Step 2, and calculate the three-dimensional spatial coordinates of the target object and the user's hand (if detected) in the camera coordinate system; the camera coordinate system is relatively fixed relative to the user's first viewpoint, and the spatial position of the camera's optical center can indirectly represent the user's relative position. Step 4: Select a dynamic reference frame. If the user's hand is not detected in Step 2, the camera's optical center is used as the reference point; if the user's hand is detected, the three-dimensional spatial coordinates of the hand calculated in Step 3 are used as the reference point. Step 5: Based on the three-dimensional spatial coordinates of the selected reference point and the target object, calculate the horizontal azimuth angle θ, the straight-line distance D, and the vertical coordinate Y. Map the horizontal azimuth angle θ to the "clock coordinate system" (12 o'clock is directly in front of the user, 9 o'clock is directly to the left, and 3 o'clock is directly to the right) to generate the first voice guidance information containing "N o'clock direction", "vertical direction slightly up / slightly down" and the straight-line distance D. A list of hazardous materials is pre-stored, and the three-dimensional Euclidean distance between the selected reference point and each item in the list is calculated. If any distance is less than a preset safety threshold, a second voice warning message containing the name, location, and distance of the hazardous material is generated, and the warning message has higher priority than the guidance message. Step 6: The voice commands generated in step 5 are broadcast in real time through the speaker of the voice interaction module. During this process, steps 2 to 5 are continuously executed in a loop, and the item location information is dynamically updated until the task ends and the next task or standby state is entered.

[0034] Furthermore, in step 1, the specific keywords of the preset wake-up command include "startup guide", "help me find", "item finder", etc. When the wake-up command is not recognized, only low-power audio monitoring state is maintained to reduce device power consumption; Furthermore, in step 1, the cloud-based speech recognition service interface is invoked to convert it into text commands, specifically as follows: When calling the cloud-based speech recognition service interface, the HTTP protocol is used for data transmission. The audio data collected by the microphone is encoded into MP3 format and uploaded to the cloud. After the cloud service completes speech recognition, it returns the corresponding text command. Then, the target item name is parsed through a string matching algorithm. For example, the keyword "glasses" is extracted from the text command "Help me find my glasses" as the target item name. If parsing fails (e.g., the specific item name is not recognized), the system will announce "Target item not specified, please re-command" via voice and return to command receiving mode, waiting for the user to input again.

[0035] Furthermore, in step 2, the binocular camera is fixed to the front end of the wearable glasses frame, the horizontal baseline length b of the two cameras is 6 cm, the camera focal length is 3.6 mm, and the camera frame rate is set to 30 fps to ensure the real-time and synchronous nature of image acquisition. Furthermore, in step 2, the improved YOLO object detection model is a model based on the improved YOLOv8 architecture, with the specific improved structure as follows: First, the original backbone network was replaced with the EfficientNetV1 network. Through a composite scaling strategy (adjusting the network width, depth and input resolution at the same time), the feature extraction capability of small target objects was improved without increasing the number of model parameters by more than 15%. Secondly, a convolutional block attention module (CBAM) is embedded in the neck network of the original model (between the C2f module and the SPPF module). This module can enhance the feature saliency of small targets through a dual attention mechanism. Finally, the detection head is reconstructed using a variety of branch modules, with three parallel branches to process high, medium, and low semantic features respectively: the high semantic branch corresponds to a 640×640 input scale and is used to detect large-sized objects; the medium semantic branch corresponds to a 320×320 input scale and is used to detect medium-sized objects; and the low semantic branch corresponds to a 160×160 input scale and is used to detect small-sized and partially occluded objects (occlusion area ≤40%). Through the branch fusion strategy, the bounding box localization accuracy of partially occluded objects is improved by more than 20%.

[0036] Furthermore, in step 3, the optimized stereo matching algorithm is a post-processed optimized semi-global block matching (SGBM) algorithm. The specific optimization process is as follows: First, the original disparity map output by the SGBM algorithm is standardized to map the disparity values ​​to the interval [0, 255], as shown in the following formula (1): (1) in, (x,y) represents the original disparity values. and These are the minimum and maximum disparity values ​​in the disparity map, respectively. (x,y) represents the standardized disparity value; simultaneously, negative disparity points marked as invalid are filtered out. (x,y)<0); Secondly, the disparity map after initial filtering is subjected to median filtering (window size set to 5×5) to remove salt-and-pepper noise, followed by bilateral filtering (spatial standard deviation). =5, Standard deviation of grayscale value =10), preserving object edge information while smoothing the disparity map; Finally, for disparity hole regions that still exist after filtering due to occlusion or other reasons, a neighborhood interpolation-based filling method is adopted: traverse the hole pixels (x,y) in the disparity map, search for effective disparity values ​​in their 8 neighborhoods, calculate the mean of the effective disparity values ​​as the filling value for the hole pixel, and if there are no effective disparity values ​​in the neighborhood, expand the search range to 16 neighborhoods to ensure the integrity of the disparity map.

[0037] Furthermore, in step 3, the three-dimensional spatial coordinates of the target object and the user's hand (if detected) in the camera coordinate system are calculated. The specific principle is as follows: The principle of 3D coordinate calculation is based on binocular vision triangulation. The core is to use the geometric constraints of binocular cameras and the disparity values ​​obtained by matching to calculate the depth of spatial points. The specific process is as follows: First, the intrinsic parameter matrix K and extrinsic parameters are obtained through camera calibration. The expression for the intrinsic parameter matrix K is: K =

[0038] in, , These are the camera's focal lengths in the x and y directions, respectively (in pixels). , The coordinates of the camera principal point (offset from the origin of the pixel plane); Secondly, let the pixel coordinates of the target object in the left camera image be ( , The corresponding matching pixel coordinates in the right camera image are ( , If ), then the disparity value d = ; Finally, based on the principle of triangulation, the formula for calculating the three-dimensional coordinates (X,Y,Z) of the target object in the camera coordinate system is as follows: (2) (2) Where Z is the depth (in centimeters) from the target object to the camera's optical center, and X and Y are the coordinate offsets in the horizontal and vertical directions, respectively. Combined with the effective disparity value d in the optimized disparity map, the precise 3D spatial coordinates can be calculated. Similarly, the 3D spatial coordinates of the user's hand in the camera coordinate system are obtained using the same formula by substituting the corresponding pixel coordinates of the user's hand in the left and right camera images into the formula.

[0039] Furthermore, in step 4, a dynamic reference frame is selected, specifically as follows: First, obtain the three-dimensional spatial coordinates of the target object in the camera coordinate system calculated in step 3, as well as the determination result of whether the user's palm has been detected. If the user's palm is not detected, the relative position and distance between the camera's optical center and the target object are calculated using the camera's optical center as a reference point (coordinates set to (0,0,0)). If the user's palm is detected, the three-dimensional spatial coordinates of the user's palm in the camera coordinate system are first calculated using the binocular stereo matching algorithm optimized in step 3. Then, the relative position and distance between the palm and the target object are calculated using the palm coordinates as a reference point. The advantage of dynamic reference system selection is that it perfectly matches the practical operation logic of visually impaired users in finding objects: when the user does not reach out, the relative position of the object is announced based on the user's field of vision, helping to quickly locate the approximate position; when the user reaches out to explore, it automatically switches to an announcement based on the palm of the hand, allowing the user to accurately perceive the real-time relative position of the object and the hand. This intelligent adaptation, which does not require manual settings, not only conforms to the user's natural movement habits, but also greatly improves the accuracy of close-range operations, making the object-finding guidance more targeted and practical, and helping users retrieve their target items efficiently and smoothly.

[0040] Furthermore, in step 5, the clock coordinate system mapping and guidance information generation are specifically implemented as follows: First, obtain the three-dimensional spatial coordinates of the selected reference point and the target item obtained in step 4, and then calculate the three-dimensional spatial coordinates of the target item relative to the selected reference point. , , ),based on( , , Calculate the horizontal azimuth angle And the straight-line distance D.

[0041] The value range is -90° to 90°, where 0° corresponds to the user's front, 90° corresponds to the right, and -90° corresponds to the left. The calculation expression is shown in the following formula (3): (3) The expression for calculating the straight-line distance D (unit: centimeters) is shown in the following formula (4): (4) Determine the vertical direction based on the sign of the vertical coordinate Y: "Slightly above" | centimeters” "lower" | centimeters”; Secondly, establish a clock coordinate system with the user's direct front as the 12 o'clock direction, the left as the 9 o'clock direction, and the right as the 3 o'clock direction, and set the horizontal azimuth angle... Mapped to the range from 9 o'clock to 3 o'clock in this coordinate system, such as Figure 4 As shown, the mapping rule is: At 165°~180° on the clock coordinate system (corresponding to θ∈[-90°,-75°]): 9 o'clock direction; At 135°~165° on the clock coordinate system (corresponding to θ∈[-75°,-45°]): 10 o'clock direction; At 105°~135° in the clock coordinate system (corresponding to θ∈[-45°,-15°]): 11 o'clock direction; At 75°~105° in the clock coordinate system (corresponding to θ∈[-15°,15°]): the 12 o'clock direction; In the clock coordinate system, at positions 45°~75° (corresponding to θ∈[15°,45°]): 1 o'clock direction; At 15°~45° on the clock coordinate system (corresponding to θ∈[45°,75°]): 2 o'clock direction; In the clock coordinate system, at 0°~15° (corresponding to θ∈[75°,90°]): 3 o'clock direction; Finally, based on the mapping results, the generated format is "Target item is located in your viewpoint / palm [N o'clock] direction [D] cm, slightly above / slightly below [| The voice guidance command is: "At [centimeter], please maintain the forward direction / correct your orientation!"

[0042] Furthermore, in step 5, the distance between the user and items in the preset list of dangerous items is calculated in real time and an early warning message is generated, specifically as follows: Set a preset safety threshold, such as 15 centimeters; store a list of dangerous items, including common dangerous items such as knives, hot water cups, sharp utensils, power strips, and lighters; After each target identification and 3D coordinate calculation, only the 3D Euclidean distance between the selected reference point and each hazardous item in the list is calculated, as shown in the following formula (5): (5) in,( , , ( ) represents the three-dimensional spatial coordinates of the hazardous material; , , () represents the three-dimensional spatial coordinates of the selected reference point; If the distance of any dangerous item If the distance is less than 15 cm, an early warning message will be generated immediately and broadcast with priority. The broadcast format is: "Warning: [Danger item name] is located at your viewpoint / palm [N o'clock] direction, [..." At ] centimeters, slightly above / slightly below [| "|] cm, please avoid," the warning message will be broadcast while the guidance message is paused until the user is away from the dangerous item. Recovery occurs after ≥15 cm.

[0043] Furthermore, in step 6, the voice interaction logic is specifically as follows: The voice broadcast module uses the Pyttsx3 text-to-speech library and supports speech rate adjustment (default 150 words / minute) and volume adjustment (default 80%) to ensure that visually impaired people can hear clearly. After a single task is completed (i.e., the user successfully grabs the target item, determined by the three-dimensional distance between the palm and the target item being less than 5 cm), the system will announce the question, "The target item has been found. Do you need to find any other items?" The response time is set to 5 seconds. The voice interaction module listens for user responses in real time. Positive voice responses include "Yes", "Need", and "Continue", while negative voice responses include "No", "Don't Need", and "End". If a positive response is received within 5 seconds, repeat steps 1 to 5 to start a new round of item finding; if a negative response is received or there is no response after a timeout, shut down the image acquisition module and retain only the low-power audio monitoring function until the next wake-up command is triggered.

[0044] Example 2 A detailed introduction to step S2, including the improved YOLO object detection model network structure and experimental results, such as... Figure 2 As shown, it specifically includes: S21. The optimized model structure uses the EfficientNetV1 network to replace the original backbone structure of the YOLOv8n model. The model size is balanced through a compound scaling mechanism to improve the detection accuracy and efficiency. The CBAM attention mechanism is integrated into the neck network to enhance the expression of key information and improve the feature saliency of small targets through channel and spatial attention mechanisms. Diverse branch modules are introduced to reconstruct the detection head to enhance the model's feature parsing ability and training efficiency.

[0045] S22. To verify the effectiveness of the YOLOv8 model improvements, ablation experiments were conducted to validate the model in detail. The experimental model was trained on a server with the following environment: Ubuntu 18.04 operating system, Intel CPU E5-2686 v4 processor, 60GB of memory, and an NVIDIA RTX A4000 graphics card with 16GB of video memory. The development tools were PyCharm, Python 3.8, CUDA version 11.3, cuDNN version 8.0, and PyTorch version 1.10. The parameter configurations during the experimental training process are shown in Table 1.

[0046] Table 1 Model training parameter configuration

[0047] To test the performance of the improved YOLOv8 model, precision (P), recall (R), and mean average precision (mAP) were selected as metrics for evaluating recognition accuracy. The mathematical expressions are shown below: (6) (7) (8) (9) In the above formula: FP—The number of samples where the actual item category does not match the target item category; TP – The number of samples where the detected real item category matches the target item category; FN—The number of samples of real target items that were not detected by the network; N – The number of categories of items.

[0048] The ablation experimental data of the improved YOLOv8 model are shown in Table 2.

[0049] Table 2 Ablation Experiment

[0050] Compared to the original YOLOv8 model, the average accuracy improved by 1.8 percentage points after introducing the EfficientNetV1 backbone network. The average accuracy improved by another 0.7 percentage points after adding the CBAM attention mechanism. Finally, the average accuracy increased by 0.6% by reconstructing the detection head using various branch modules, eventually reaching 86.4%. This demonstrates the rationality of the model design and its better detection performance.

[0051] S23. To effectively evaluate the advancement of YOLOv8n-ECD, Faster R-CNN, SSD, YOLOv3, YOLOv5s, YOLOv7, YOLOv7-tiny, YOLOv9, YOLOv8n, YOLOv10, YOLOv12n, and the current mainstream detection models in this field, PP-YOLOE and RT-DETR-R18, were compared with the improved YOLOv8 model. The results are shown in Table 3. "Ours" in Table 3 refers to the improved detection model presented in this paper.

[0052] Table 3 Comparison of recognition performance of different models

[0053] As can be seen from the model comparison table, when identifying target objects, the improved model YOLOv8n-ECD is the best in terms of precision, recall, and average accuracy compared to the currently popular target detection models Faster R-CNN, SSD, YOLOv3, YOLOv5s, YOLOv7, YOLOv7-tiny, YOLOv9, YOLOv10, as well as the mainstream detection models PP-YOLOE and RT-DETR-R18 in this field.

[0054] S24. Finally, the improved YOLOv8 object detection model was deployed on the JETSON NANO processing unit. In actual testing on the JETSON NANO, the improved model achieved an inference speed (FPS) of 22 frames per second, meeting real-time requirements. On the self-built test set, the mAP50 reached 86.4%, a 3.1% improvement over the original YOLOv8 model, with particularly significant improvements in recall for small targets and hands.

[0055] Example 3 In step S3, the SGBM binocular stereo matching algorithm optimizes and calculates the 3D spatial coordinates of the target object. The specific process is as follows: S31. Use Zhang Zhengyou's calibration method to perform joint calibration of the stereo camera and obtain high-precision intrinsic parameter matrix, distortion coefficients and stereo extrinsic parameters (rotation matrix R and translation vector T).

[0056] S32. Optimized SGBM stereo matching algorithm, the contents of which include: Invalid point removal: Directly remove pixels with negative disparity values ​​or those exceeding a reasonable range; Median filtering: A median filter using a 5x5 window effectively removes isolated noise points (salt and pepper noise). Bilateral filtering: Bilateral filtering is performed on the disparity map, which can smooth the disparity map while preserving the sharpness of object edges; Hole filling: For disparity holes that still exist after filtering (usually caused by occlusion), the neighborhood mean filling method is used. That is, with the hole pixel as the center, search for the effective disparity values ​​in its 8-neighborhood, and take the average value to fill the hole.

[0057] S33. Finally, perform three-dimensional coordinate calculation; for the pixel point (u, v) of the target object and its optimized disparity value d, calculate its three-dimensional coordinates (X, Y, Z) in the left camera coordinate system using formula (2). This coordinate is the basis for all subsequent spatial calculations. The principle of stereo matching vision imaging with binocular cameras is as follows: Figure 3 As shown.

[0058] Example 4 The specific implementation of the selection of the dynamic reference frame and the mapping of the clock coordinate system in steps S4 and S5 is described in detail below: The logic for selecting a dynamic reference frame is as follows: S41, Judgment Condition: The system detects the user's hand while detecting an object. It checks the "hand" category every frame. If the confidence level is greater than a threshold (e.g., 0.7) and the bounding box size is reasonable, it is considered that a "hand has been detected."

[0059] S42. Coordinate Transformation: Camera reference frame: When no hand is detected, the reference point is the camera optical center (0,0,0). Target object coordinates ( , , ) is the relative coordinate.

[0060] Hand reference frame: When a hand is detected, the three-dimensional coordinates of the center point of the palm are first calculated. , , The coordinates of the target item relative to the palm are: ( , , ) = ( , , ).

[0061] The mapping process of the clock coordinate system is as follows: S51. Calculate the horizontal azimuth angle θ using the two-parameter arctangent function to avoid division by zero errors and automatically determine the correct quadrant. The calculation formula is: ; Where (X, Z) are the coordinates of the target object relative to the selected reference point (camera optical center or palm) on the horizontal plane. The function arctan2 returns the radian value from the positive Z-axis (directly in front of the user) to the point (X, Z), with the result ranging from (-π, π]. The corresponding angle value range is approximately (-180°, 180°).

[0062] S52. Map the target object's position to a clock coordinate system. To generate a directional description of "what time of day" that is intuitively easy for visually impaired users to understand, map the object's position in the visible sector (-90° to 90°) to the 9 o'clock to 3 o'clock direction. Specific mapping rules are shown in Table 4 below: Table 4 Mapping Rules Table

[0063] S53. After the clock coordinate system mapping is completed, combine the calculated horizontal distance D and vertical distance. Generate a complete voice-guided description. A value greater than 0 indicates "slightly above," while a value greater than 0 indicates "slightly below." Ultimately, the speech synthesis module generates a guiding statement in the format of "The target item is located at your [viewpoint / palm] [N o'clock] direction, at a distance of approximately [D] centimeters, [slightly above / slightly below]."

[0064] Example 5 The detailed implementation logic of the cyclic execution of steps 2 to 5 in step S6 is described in detail, including: S61. To achieve smooth management of low-power standby, single boot, and multi-round tasks, the system software kernel maintains a clear state machine, including the following states: S611, Sleep State: Only the voice interaction module subunit operates in an extremely low-power state, continuously listening for the preset wake-up word. The image acquisition module and the target detection module are completely shut down.

[0065] S612, Command Receiving Status (LISTENING): Entered after wake-up. The system plays a prompt tone, initiates the full voice recognition process, and waits for and parses the user's specific item-finding command.

[0066] S613, Real-time Guidance State: Entered after successful command parsing. In this state, the image acquisition module, target detection module, 3D solution module, and voice broadcasting module run at full speed, looping through the detection-calculation-broadcasting process.

[0067] S614, QUESTIONING: Entered when a single item retrieval task is considered completed.

[0068] S62. Determining the End of a Single Task. This involves the system continuously monitoring while in GUIDING mode to determine whether the current GUID loop should be terminated. The determination condition is "the target item has been successfully captured," specifically determined through a comprehensive logic as follows: S621, Condition 1 (Close Proximity): Calculate the real-time three-dimensional Euclidean distance between the target object and the palm in the "palm reference frame". If this distance remains less than 5 cm for more than 1 second, the object is considered to have been touched.

[0069] S622, Condition 2 (Visual Feedback): At the same time, the detection model reports that the IOU (Intersection over Union) of the bounding box of the "target item" and the bounding box of the "hand" is greater than 0.5, or the target item continues to disappear from the screen for more than 5 frames due to being held by the hand.

[0070] When both conditions 1 and 2 are met, or when condition 1 is met and the system remains in palm reference mode for a certain period of time, the system determines that the item search is successful.

[0071] S63. Multi-round interaction logic. That is, after the judgment task is completed, the system immediately switches from the GUIDING state to the QUESTIONING state. The specific process is as follows: S631. The system asks via voice announcement: "[Item Name] has been found. Would you like to look for any other items?" S632, Response Monitoring and Timeout Setting: Start a 5-second timer while listening for user voice responses. The affirmative response vocabulary includes "Yes," "Need," "Continue," etc.; the negative response vocabulary includes "No," "No Need," "End," etc.

[0072] S633, Logical Branches: If a positive response is received within 5 seconds: the system immediately jumps back to the LISTENING state, announces "Please name the next item," and starts a new round of item finding. Image acquisition and other modules remain running to achieve seamless integration; If a negative response is detected or no response is received within the timeout period, the system will announce "The item finding assistant has been closed." Subsequently, the image acquisition module and the target detection process will be closed in sequence, and finally the main control program will be switched to SLEEP state, retaining only the basic voice monitoring thread.

[0073] S64. Exception Handling and Status Reset. In GUIDING state, if the task end judgment is not triggered after more than 3 minutes (e.g., the target item never enters the field of view), the system will actively announce: "[Item Name] not found temporarily, please try moving or redefining the item." It will then automatically return to LISTENING state, allowing the user to reissue the command or give up, avoiding getting stuck in an invalid loop.

[0074] like Figure 5 As shown, the present invention also discloses a real-time object-finding guidance system for blind people to implement the above method. The system is integrated into a wearable glasses device and specifically includes: The processing unit uses the JETSON NANO embedded development board as the core processor. This development board is equipped with a Quad-Core ARM A57 CPU and a 128-core NVIDIA Maxwell GPU, which supports CUDA acceleration and is used to perform all calculations and logic scheduling such as target detection, 3D coordinate calculation, reference frame selection, orientation transformation, security monitoring and voice interaction control. The image acquisition module consists of a pair of visible light cameras fixed to the front of the wearable glasses frame, forming a binocular camera with a resolution of 2560×720, a focal length of f=3.6 mm, and a horizontal baseline length of b=6 cm. It is electrically connected to the processing unit via a USB 3.0 interface and is used to acquire synchronized left and right view scene images in real time. The voice interaction module integrates an omnidirectional microphone and a small speaker. The microphone sampling rate is set to 16kHz, which supports long-distance voice acquisition (effective distance ≤3 meters). The speaker output power is ≥1W, and the frequency response range is 200Hz~8kHz. It is electrically connected to the processing unit through the I2S interface and is used to collect user voice commands and broadcast guidance and warning information. The network communication module, which adopts a WiFi 802.11 b / g / n module, is integrated on the processing unit and connected to it via an SPI interface. It is used to enable the system to access the Internet and call the cloud-based voice recognition service interface. The power module uses a rechargeable lithium battery with a capacity of ≥1000mAh, an output voltage of 5V / 2A, supports continuous operation for ≥4 hours, and has overcharge, over-discharge, and overcurrent protection functions. It can be charged via a USB-C interface.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for real-time object-finding guidance for blind people based on binocular vision and object detection, characterized in that, Includes the following steps: Step 1: Monitor ambient audio in real time, and preset the wake-up command as a specific keyword; when the wake-up command is recognized, enter the command receiving mode, collect the user's voice command for finding the item, convert the voice command for finding the item into a text command, and parse the name of the target item to be found from the text command. Step 2: Real-time acquisition of binocular images of the indoor scene; use the improved YOLO object detection model to identify the category and two-dimensional pixel position of all objects in the scene; and determine whether the user's palm is detected. Step 3: Based on the optimized binocular stereo matching algorithm, perform stereo matching on the indoor scene binocular images acquired in Step 2, and calculate the three-dimensional spatial coordinates of the target object and the user's hand in the camera coordinate system; the camera coordinate system is relatively fixed relative to the user's first viewpoint. Step 4: If the user's palm is not detected in step 2, the camera's optical center is used as the reference point; if the user's palm is detected, the three-dimensional spatial coordinates of the user's palm obtained in step 3 are used as the reference point. Step 5: Calculate the three-dimensional spatial coordinates of the target item relative to the selected reference point based on the three-dimensional spatial coordinates of the selected reference point and the target item; calculate the horizontal azimuth, straight-line distance and vertical coordinates based on the three-dimensional spatial coordinates of the target item relative to the selected reference point, map the horizontal azimuth to the clock coordinate system, and generate voice guidance information including clock direction, vertical direction up / down and straight-line distance. A list of hazardous materials is pre-stored, and the three-dimensional Euclidean distance between the selected reference point and each item in the list is calculated. If any distance is less than a preset safety threshold, a voice warning message containing the name, location, and distance of the hazardous material is generated, and the voice warning message has a higher priority than the voice guidance message. Step 6: Real-time broadcast of the voice guidance and voice warning information generated in Step 5, and continuously loop through Steps 2 to 5 during this process, dynamically updating the item location information until the task ends and enters the next task or standby state.

2. The method for real-time object-finding guidance for the blind based on binocular vision and target detection according to claim 1, characterized in that, In step 2, the improved YOLO object detection model is a model based on the improved YOLOv8 architecture, and the specific improved structure is as follows: Replace the original backbone network with an EfficientNetV1 network; Embed convolutional block attention modules in the neck network of the original model; The detection head is reconstructed using a variety of branch modules, with three parallel branches to process high semantic features, medium semantic features, and low semantic features respectively.

3. The method for real-time object-finding guidance for the blind based on binocular vision and target detection according to claim 1, characterized in that, In step 3, the optimized stereo matching algorithm is a post-processed optimized semi-global block matching algorithm. The specific optimization process is as follows: The original disparity map output by the semi-global block matching algorithm is standardized, and negative disparity points marked as invalid are filtered out. The disparity map after initial filtering is first subjected to median filtering to remove salt-and-pepper noise, and then bilateral filtering is performed to smooth the disparity map while preserving object edge information. For the disparity hole regions caused by occlusion after filtering, interpolation is performed to fill them based on the disparity values ​​of the effective pixels in the surrounding neighborhood.

4. The method for real-time object-finding guidance for the blind based on binocular vision and target detection according to claim 1, characterized in that, In step 2, binocular images of the indoor scene are acquired in real time using a binocular camera fixed to the user's wearable device; In step 3, the three-dimensional spatial coordinates of the target object and the user's hand in the camera coordinate system are calculated, specifically as follows: The intrinsic parameter matrix K is obtained through camera calibration, and the expression for the intrinsic parameter matrix K is: K = in, , These are the focal lengths of the camera in the x and y directions, respectively. , The coordinates of the camera's principal point; Let the pixel coordinates of the target object in the left camera image be ( , The corresponding matching pixel coordinates in the right camera image are ( , If the disparity is d, then the disparity value d = d = ; Based on the principle of triangulation, the formula for calculating the three-dimensional coordinates (X, Y, Z) of the target object in the camera coordinate system is as follows: Where b is the horizontal baseline length of the binocular camera, Z is the depth of the target object from the optical center of the camera, and X and Y are the coordinate offsets in the horizontal and vertical directions, respectively. The three-dimensional spatial coordinates of the user's palm in the camera coordinate system are obtained by substituting the corresponding pixel coordinates of the user's palm in the left and right camera images into the calculation formula.

5. A method for real-time object-finding guidance for the blind based on binocular vision and target detection according to claim 1, characterized in that, In step 5, mapping the horizontal azimuth angle to the clock coordinate system specifically involves: Based on the three-dimensional spatial coordinates of the selected reference point and the three-dimensional spatial coordinates of the target object, calculate the three-dimensional spatial coordinates of the target object relative to the selected reference point. , , ),based on( , , Calculate the horizontal azimuth angle and the straight-line distance D; The value range is -90° to 90°, where 0° corresponds to the user directly in front, 90° corresponds to the user directly to the right, and -90° corresponds to the user directly to the left; the calculation formula is: ; Straight-line distance D, Based on vertical coordinates Determining the positive or negative sign in the vertical direction: Slightly above average. Slightly lower; The clock coordinate system uses the direction directly in front of the user as 12 o'clock, the direction directly to the left as 9 o'clock, and the direction directly to the right as 3 o'clock. The horizontal azimuth angle is mapped onto this clock coordinate system to generate voice guidance information.

6. The method for real-time object-finding guidance for the blind based on binocular vision and target detection according to claim 1, characterized in that, In step 5, the dangerous goods warning specifically includes: The system calculates the distance between the user and items in a preset list of hazardous materials in real time and generates warning messages, specifically: After each target identification and 3D coordinate calculation, the 3D Euclidean distance between the selected reference point and each hazardous item in the list is calculated using the following formula: in,( , , ( ) represents the three-dimensional spatial coordinates of the hazardous material; , , () represents the three-dimensional spatial coordinates of the selected reference point; If the distance of any dangerous item If a preset safety threshold is reached, a warning message will be generated and broadcast first. When the warning message is broadcast, the guidance message will be paused until the user moves away from the dangerous item.

7. A method for real-time object-finding guidance for the blind based on binocular vision and target detection according to claim 6, characterized in that, The list of dangerous items includes knives, hot water cups, sharp utensils, power strips, and lighters.

8. A real-time object-finding guidance system for the blind based on binocular vision and object detection, used to implement the method as described in any one of claims 1 to 7, characterized in that, Specifically, it includes: The processing unit is used to perform all the calculations and logic scheduling for target detection, 3D coordinate calculation, reference frame selection, orientation transformation, security monitoring and voice interaction control. The image acquisition module consists of a pair of binocular cameras, which are electrically connected to the processing unit and are used to acquire synchronized left and right view scene images in real time; the binocular cameras are fixed relative to the user's first view when worn. The voice interaction module integrates a microphone and a speaker, and is electrically connected to the processing unit. It is used to collect user voice commands and broadcast guidance and warning information. A network communication module, connected to the processing unit, is used to enable the system to access the Internet in order to call the cloud-based speech recognition service interface; The power supply module is electrically connected to each functional module and is used to supply power to the system.

9. A real-time object-finding guidance system for the blind based on binocular vision and target detection according to claim 8, characterized in that, The system is integrated into a wearable glasses device, and the binocular camera is fixed to the front end of the frame of the wearable glasses.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images