Direction identification method and system of blind person intelligent glasses in outdoor environment

Abstract

The invention discloses a direction identification method and system of blind person intelligent glasses in an outdoor environment, and relates to the technical field of direction positioning of the blind person intelligent glasses. And according to the current position of the blind person intelligent glasses and the current position of each feature of the stereo model, determining the primary direction of the blind person intelligent glasses in the outdoor environment, and carrying out primary direction identification in the stereo model. Therefore, a direction deviation value is determined according to the ultrasonic positioning information, the front side image and the characteristics recorded by the three-dimensional model in the primary direction; according to the direction deviation value, the primary direction and the moving direction of the blind intelligent glasses, the final direction of the blind intelligent glasses in the outdoor environment is determined, dynamic movement of the blind intelligent glasses is fully considered, and the accuracy of the final direction of the blind intelligent glasses in the outdoor environment is further guaranteed; and multi-stage identification from the primary direction to the final direction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of smart glasses for the blind, and in particular to a method and system for identifying the direction of smart glasses for the blind in an outdoor environment. Background Art

[0002] With the development of science and technology, smart glasses for the blind are electronic devices that can take photos and perform voice recognition. Blind people wear smart glasses for the blind and move in outdoor environments. In the existing technology, smart glasses for the blind take photos to collect multiple images, and determine the corresponding directional features based on the recognition of multiple images, so as to determine the direction of the smart glasses in the outdoor environment by calculating the directional features. However, the direction of the smart glasses in the outdoor environment is recognized along a single dimension of the image, resulting in low accuracy of the direction. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a method and system for direction recognition of blind smart glasses in outdoor environments.

[0004] An embodiment of the present invention provides a method for direction recognition of blind smart glasses in an outdoor environment, comprising: determining the type of outdoor environment based on surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment types include outdoor roads, outdoor parks, or outdoor sports fields; constructing a three-dimensional model of the outdoor environment based on a distribution map of the outdoor environment types and the surrounding images; determining, in the three-dimensional coordinate system of the three-dimensional model, the primary direction of the blind smart glasses in the outdoor environment based on the current position of the blind smart glasses and the current position of each feature of the three-dimensional model; determining ultrasonic positioning information and a front image based on external detection of the blind smart glasses, and determining a direction deviation value based on the ultrasonic positioning information, the front image, and the features recorded in the primary direction of the three-dimensional model; and determining the final direction of the blind smart glasses in the outdoor environment based on the direction deviation value, the primary direction, and the moving direction of the blind smart glasses.

[0005] An embodiment of the present invention provides a system for recognizing the direction of blind smart glasses in an outdoor environment. The system is applied to the above-mentioned method for recognizing the direction of blind smart glasses in an outdoor environment. The system comprises: An outdoor environment type module is used to determine the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment types include outdoor roads, outdoor parks, or outdoor sports fields; A stereo module, configured to construct a stereo model of the outdoor environment based on a distribution map of outdoor environment types and the surrounding image; a primary direction module, configured to determine, in the stereo coordinate system of the stereo model, a primary direction of the blind smart glasses in the outdoor environment according to the current position of the blind smart glasses and the current positions of the various features of the stereo model; a direction deviation value module, configured to determine ultrasonic positioning information and a front image based on external detection by the blind smart glasses, and determine a direction deviation value according to the ultrasonic positioning information, the front image, and features recorded by the stereo model in the primary direction; The final direction module is used to determine the final direction of the blind smart glasses in the outdoor environment according to the direction deviation value, the primary direction and the moving direction of the blind smart glasses.

[0006] Compared with the prior art, the present invention has the following beneficial effects: In an embodiment of the present invention, the method in the embodiment of the present invention is used to determine the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment types include outdoor roads, outdoor parks, or outdoor sports fields; a three-dimensional model of the outdoor environment is constructed based on a distribution map of the outdoor environment types and the surrounding images; in the three-dimensional coordinate system of the three-dimensional model, the primary direction of the blind smart glasses in the outdoor environment is determined based on the current position of the blind smart glasses and the current positions of each feature of the three-dimensional model, and preliminary direction recognition is performed in the three-dimensional model.

[0007] Therefore, based on the external detection of the blind smart glasses, the ultrasonic positioning information and the front image are determined, and the direction deviation value is determined according to the ultrasonic positioning information, the front image and the features recorded by the stereo model in the primary direction; the final direction of the blind smart glasses in the outdoor environment is determined according to the direction deviation value, the primary direction and the moving direction of the blind smart glasses, which is compatible with the overall consideration of the direction deviation value, the primary direction and the moving direction of the blind smart glasses, and fully considers the dynamic movement of the blind smart glasses, further ensuring the accuracy of the final direction of the blind smart glasses in the outdoor environment, realizing multi-level recognition from the primary direction to the final direction, and thus realizing the direction positioning of the blind in the outdoor environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 2 is a flow chart of a method for direction recognition in an outdoor environment using smart glasses for the blind according to an embodiment of the present invention; Figure 2 1 is a flow chart of step S11 in the method for direction recognition of smart glasses for the blind in an outdoor environment in an embodiment of the present invention; Figure 3 2 is a flow chart of step S12 in the method for direction recognition of smart glasses for the blind in an outdoor environment in an embodiment of the present invention; Figure 41 is a flow chart of step S13 in the method for direction recognition of smart glasses for the blind in an outdoor environment in an embodiment of the present invention; Figure 5 1 is a flow chart of step S14 in the method for direction recognition of smart glasses for the blind in an outdoor environment in an embodiment of the present invention; Figure 6 1 is a flow chart of step S15 in the method for direction recognition of smart glasses for the blind in an outdoor environment in an embodiment of the present invention; Figure 7 FIG. 1 is a schematic diagram of the structural composition of a direction recognition system of the blind smart glasses in an outdoor environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0010] See also Figures 1 to 7 , a method for direction recognition of smart glasses for blind people in outdoor environments includes: Step S11: determining the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment types include outdoor roads, outdoor parks, or outdoor sports fields; Step S12: constructing a three-dimensional model of the outdoor environment based on the distribution map of outdoor environment types and the surrounding image; Step S13: determining the primary direction of the blind smart glasses in the outdoor environment in the 3D coordinate system of the 3D model according to the current position of the blind smart glasses and the current positions of the various features of the 3D model; Step S14: determining ultrasonic positioning information and a front image based on external detection by the blind smart glasses, and determining a direction deviation value according to the ultrasonic positioning information, the front image, and features recorded by the three-dimensional model in the primary direction; Step S15: determining the final direction of the blind smart glasses in the outdoor environment according to the direction deviation value, the primary direction, and the moving direction of the blind smart glasses; refer to Figure 2 In step S11, the type of outdoor environment is determined based on the surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment type includes an outdoor road, an outdoor park, or an outdoor sports field; In the specific implementation process of the present invention, the specific steps are: S111: Multiple peripheral cameras of the blind smart glasses shoot in different directions and collect multiple peripheral images, determine the features of each object based on recognition of the multiple peripheral images, and mark the relative positions of the blind smart glasses and each object feature; S112: Each sound collector of the blind smart glasses collects multiple environmental sounds, and triggers matching of the multiple environmental sounds with object features based on the collection directions of the multiple environmental sounds and the response signals of each sound collector, thereby introducing sound information of the object features; S113: Determine a first outdoor parameter based on the sound information of the object feature and the shape of the object feature, determine a second outdoor parameter based on the relative position of the blind smart glasses and the object feature and the shape of the object feature, and determine the type of outdoor environment based on the first outdoor parameter, the second outdoor parameter and the outdoor mapping relationship.

[0011] In an embodiment of the present application, multiple peripheral cameras of the blind smart glasses shoot along different directions and collect multiple peripheral images. The features of each object are determined based on the recognition of multiple peripheral images, and the relative positions of the blind smart glasses and each object feature are marked, introducing the relative positions of the blind smart glasses and each object feature.

[0012] At this time, smart glasses for the blind are usually equipped with multiple cameras, which are designed to cover different directions around the glasses; for example, including cameras facing the front, left and right sides, and rear, which capture images of the surrounding environment from different perspectives; optionally, the camera uses a wide-angle lens to capture a wider field of view, and uses image processing algorithms (such as image stitching) to integrate images from different cameras to form a more complete view of the environment.

[0013] When the cameras capture the surrounding environment, they generate a series of image data, which is stored in the local memory of the smart glasses or transmitted to the cloud in real time via wireless connection for processing; at the same time, the image acquisition process involves real-time compression and transmission of images to ensure fast data processing and low-latency feedback.

[0014] Using computer vision technology, the system will perform feature extraction and recognition on the collected images, which includes detecting features such as edges, textures, colors in the images, and using deep learning models to identify specific objects (such as trees, buildings, road signs, etc.); optionally, feature extraction uses deep learning models such as convolutional neural networks (CNNs), which are trained to identify key features in images; the recognition process involves matching the extracted features with a predefined object database.

[0015] Once the object features are identified, the system calculates the relative position of the smart glasses and these features based on information such as scale and perspective in the image and the known position of the smart glasses (such as obtained through GPS). This involves calculating parameters such as distance and direction; optionally, position marking uses 3D reconstruction technology such as structure from motion (SfM) or multi-view stereo vision (MVS) to create a 3D model of the surrounding environment and determine the position of the smart glasses in the model.

[0016] Specifically, suppose a blind user is walking on a city street wearing smart glasses; the four peripheral cameras (front, rear, left, and right) of the smart glasses simultaneously capture images of the street; the front camera captures the shops and pedestrians in front of the street, the rear camera captures the vehicles behind, and the left and right cameras capture the buildings and trees on both sides of the street. These images are collected in real time and stored in the memory of the glasses.

[0017] The system extracts features from the image and identifies features such as store signs, pedestrian outlines, vehicle shapes, and doors and windows of buildings; at the same time, the deep learning model identifies specific objects, such as "coffee shop" and "bus stop"; based on the scale and perspective relationship in the image, the system calculates the relative position between the smart glasses and the identified object features; for example, the smart glasses are about 10 meters away from the coffee shop in front and about 5 meters away from the building on the left; through these steps, the blind smart glasses can accurately identify the features of objects in the surrounding environment and determine their relative position to the smart glasses, providing key information for subsequent direction recognition. This information is ultimately conveyed to users in the form of voice prompts, tactile feedback, etc., to help them better understand and navigate their surroundings.

[0018] Furthermore, each sound collector of the blind smart glasses collects multiple ambient sounds, and triggers the matching of multiple ambient sounds with object features based on the collection directions of the multiple ambient sounds and the response signals of each sound collector, thereby introducing the sound information of the object features and realizing the control of the sound information of the object features.

[0019] At this time, smart glasses for the blind are usually equipped with multiple sound collectors, such as microphone arrays, which are designed to capture ambient sounds from different directions; they can not only capture the intensity of the sound, but also determine the direction of the sound through sound source localization technology; optionally, the sound collector uses beamforming technology to enhance the sound signal in a specific direction, and uses arrival time difference (TDOA) or other sound source localization algorithms to determine the direction of the sound.

[0020] Once the sound is collected, the system will use the sound source localization algorithm to calculate the approximate source direction of the sound based on the layout and response signal of the sound collector. This involves time difference analysis or phase difference analysis of the sound signals received by multiple microphones. Optionally, the technical implementation is: the sound source localization algorithm is based on the principle of geometric acoustics and estimates the position of the sound source by comparing the time difference or phase difference of the sound signals received by different microphones.

[0021] After determining the direction of the sound source, the system will match the collected sound features with a known object feature sound library. This sound library contains the characteristics of various environmental sounds, such as birdsong, vehicle sounds, human voices, etc., as well as the association between these sounds and specific object features; optionally, the matching process involves the use of audio feature extraction (such as Mel-frequency cepstral coefficients MFCC, spectrograms, etc.) and machine learning models (such as support vector machines SVM, neural networks, etc.) to identify the type of sound and the source object.

[0022] Once the sound is successfully matched with the object features, the system will introduce the sound information as part of the object features, which means that the sound not only provides information about the environmental atmosphere, but is also directly related to specific objects or scenes; at the same time, the sound information is integrated into the environmental model and combined with the object features obtained through visual recognition to form a more comprehensive and rich description of the environment.

[0023] Specifically, suppose a blind user is walking in a park wearing smart glasses; the microphone array of the smart glasses captures ambient sounds from different directions; the microphone array captures the sound of birds singing, the laughter of children in the distance, and the occasional bicycle bell, which are collected and processed in real time; the system uses the sound source localization algorithm to calculate that the bird singing comes from the direction of the woods on the left, the children's laughter comes from the open area in front, and the bicycle bell comes from the path on the right.

[0024] The system matches these sound features with known features in the sound library; birdsong is matched with woods features, children's laughter is matched with open areas (playgrounds or lawns), and bicycle bells are matched with bike lanes or trails features; once a match is successful, the sound information is integrated into the environmental model; for example, the system marks the left side as "woods area", the front as "open area / playground", and the right side as "bike lane / trail"; through these steps, blind smart glasses can use sound information to enhance their understanding of the surrounding environment and provide users with a more accurate and rich navigation experience. This information is ultimately conveyed to users in the form of voice prompts, tactile feedback, etc., to help them better perceive and navigate the surrounding environment.

[0025] Therefore, the first outdoor parameter is determined according to the sound information of the object feature and the shape of the object feature, the second outdoor parameter is determined according to the relative position of the blind smart glasses and the object feature and the shape of the object feature, and the outdoor environment type is determined based on the first outdoor parameter, the second outdoor parameter and the outdoor mapping relationship. The overall consideration of the first outdoor parameter, the second outdoor parameter and the outdoor mapping relationship is compatible to ensure the accuracy of the outdoor environment type.

[0026] At this point, the system uses the sound information collected and matched in the previous step to determine the first set of outdoor parameters. These parameters include the type, intensity, frequency distribution, etc. of the sound, which together reflect the sound characteristics of the environment; optionally, the system uses audio feature extraction technology (such as MFCC, spectrum analysis, etc.) to quantify the sound information and input this information into the machine learning model to output parameters related to the specific environment.

[0027] The system combines the relative position (such as distance and direction) between the smart glasses and the identified object features and the morphology (such as size, shape, texture, etc.) of the object features to determine a second set of outdoor parameters. These parameters provide information about the spatial layout and structure of the environment; optionally, the system uses computer vision technology (such as edge detection, shape analysis, texture recognition, etc.) to extract morphological information of the object features, and combines GPS, INS and other sensor data to calculate relative position information. This information is integrated to form a second set of outdoor parameters.

[0028] The system uses the previously determined first and second sets of outdoor parameters as input, matches them with a pre-established outdoor mapping relationship (a machine learning model or lookup table) to output the type of outdoor environment; at the same time, the outdoor mapping relationship is learned based on a large amount of training data, and it can map different outdoor parameter combinations to specific environment types; for example, a parameter combination containing high bird song intensity and tree morphology is mapped to a "forest or park" environment, while a parameter combination containing high vehicle noise and building outlines is mapped to a "city street" environment.

[0029] In one embodiment of the present application, a preset outdoor environment type matching table is collected to map the first outdoor parameter and the second outdoor parameter to the outdoor environment type. The outdoor environment type matching table is shown in Table 1: Table 1 Outdoor environment type matching table

[0030] In this example, if the system detects high bird song sounds, dense trees, and an open area, the outdoor environment category will be determined to be "park and green space."

[0031] refer to Figure 3In step S12, a three-dimensional model of the outdoor environment is constructed based on the distribution map of the outdoor environment types and the surrounding image; In the specific implementation process of the present invention, the specific steps are: S121: collecting location information of the blind smart glasses, filtering a town name from the location information of the blind smart glasses, and determining an environment area where the blind smart glasses are located based on the town name, a town database, and the location information of the blind smart glasses; S122: Compare the environmental area with the outdoor environment type and generate a distribution map of the outdoor environment type; collect the surrounding image and determine the corresponding object features based on the recognition of the surrounding image; S123: Marking the shape of the object features, and performing image comparison between the shape of the object features and the distribution map of outdoor environment types, so as to gradually construct a three-dimensional model of the outdoor environment; In an embodiment of the present application, the location information of the blind smart glasses is collected, and the town name is filtered from the location information of the blind smart glasses. The environmental area where the blind smart glasses are located is determined based on the town name, the town database and the location information of the blind smart glasses. This is compatible with the overall consideration of the town name, the town database and the location information of the blind smart glasses, ensuring the accuracy of the environmental area where the blind smart glasses are located.

[0032] At this time, smart glasses for the blind are usually equipped with a GPS module or other positioning technologies (such as Bluetooth beacons, Wi-Fi positioning, inertial navigation system INS, etc.) to collect user location information in real time, including longitude, latitude, altitude, speed, etc.; optionally, the GPS module receives signals from satellites and calculates the precise position of the glasses through triangulation; other positioning technologies use the surrounding wireless signal characteristics or sensor data such as accelerometers and gyroscopes to assist in positioning.

[0033] Once the location information is collected, the system needs to compare this information with a pre-established geographic database to determine the town or region where the user is located. This usually involves matching location information (such as latitude and longitude) with geographic boundaries in the database. At this time, the system uses a geographic information system (GIS) database, which contains boundary information for various geographic features, such as towns, counties, states, etc. Through spatial query techniques (such as spatial joins, buffer analysis, etc.), the system determines the name of the town where the user is located.

[0034] After determining the town where the user is located, the system needs to further use the information in the town database to determine the specific environmental area where the user is located. This involves analyzing the relative position relationship between the user's location and different areas within the town (such as commercial areas, residential areas, industrial areas, parks, etc.); optionally, the town database contains detailed land use maps, building density maps, green space coverage and other information; the system uses this information to evaluate the environmental characteristics around the user's location and classify it into specific environmental areas; for example, if the user's location is close to multiple commercial buildings and a high-density population distribution, it is classified as a "commercial area."

[0035] Specifically, suppose a blind user is walking in a city wearing smart glasses; the GPS module of the smart glasses collects the user's current location information, including longitude 116.397128 and latitude 39.916527 (these values are fictitious and used for example); the system compares the location information with the GIS database and finds that the longitude and latitude coordinates are located in "Beijing City"; further querying the geographic boundary information in the database determines that the specific town where the user is located is "Chaoyang District, Beijing City."

[0036] The system further uses information from the town database to analyze the environmental characteristics around the user's location; based on the land use map and building density map, the system finds that there are multiple high-rise buildings and commercial facilities around the user's location, and the population density is relatively high; therefore, the system determines the user's environmental area as a "commercial area"; through the above steps, the blind smart glasses can accurately determine the town and environmental area where the user is located, providing users with more accurate and useful navigation information. This information is very important for blind users, helping them better understand and adapt to the surrounding environment and improve the safety and convenience of travel.

[0037] Furthermore, the environmental area is compared with the outdoor environment type, and a distribution map of the outdoor environment type is generated; the surrounding image is collected, and the corresponding object features are determined based on the recognition of the surrounding image; At this point, the system needs to compare the previously determined environmental areas (such as commercial areas, residential areas, parks, etc.) with known outdoor environmental types (such as urban streets, park green spaces, water bodies, etc.). This comparison is based on a pre-established mapping relationship or rule set to classify the environmental areas into specific outdoor environmental types; once the classification is completed, the system generates a distribution map of outdoor environmental types, which shows the distribution of different environmental types in geographic space; optionally, the system uses geographic information system (GIS) technology to create and display distribution maps; GIS allows geographic spatial data (such as latitude and longitude coordinates, environmental area boundaries, etc.) to be combined with attribute data (such as environmental type names, feature descriptions, etc.) to generate visual maps; the system also uses spatial analysis functions to evaluate the spatial relationships and interactions between different environmental types.

[0038] The system uses the camera of the blind smart glasses to collect images around the user, which are then input into the image recognition algorithm to identify the object features in the image, including buildings, trees, roads, vehicles, pedestrians, etc.; the image recognition algorithm is trained based on deep learning models (such as convolutional neural networks CNN) to identify various objects and extract their key features; optionally, the image recognition algorithm usually involves preprocessing steps (such as image scaling, denoising, enhancement, etc.), as well as feature extraction and classification steps; in the feature extraction stage, the algorithm uses convolutional layers to detect low-level features such as edges, textures and shapes in the image, and uses fully connected layers to combine these features to form high-level representations; in the classification stage, the algorithm matches the high-level representations with predefined object categories to determine the most object features in the image.

[0039] Specifically, suppose a blind user is wearing smart glasses and walking in a commercial district in a city; the system has previously determined that the user's environmental area is a "commercial district"; by comparing with known outdoor environmental types, the system classifies the commercial district as one of the "city street" environmental types (assuming that the commercial district is mainly composed of streets and buildings); then, the system generates a distribution map of outdoor environmental types, which shows the distribution of city streets, park green spaces (if any) and other environmental types around the user's area.

[0040] The camera of the smart glasses collects images of the user's surroundings, which include buildings, vehicles, pedestrians and some trees on the street; the system inputs these images into the image recognition algorithm, and the algorithm identifies the features of objects such as buildings, vehicles, pedestrians and trees; the algorithm also extracts key information of these features, such as the height and shape of buildings, the type and color of vehicles, the number and location of pedestrians, and the type and size of trees; through the above steps, the blind smart glasses can generate a detailed map of the user's surroundings and identify key object features. This information is very important for blind users, helping them better understand and navigate the surrounding environment and improve the safety and convenience of travel; for example, users use this information to plan safe walking routes, avoid obstacles and dangerous areas, and find nearby public facilities (such as shops, restaurants, public toilets, etc.).

[0041] Therefore, the morphology of the object features is marked, and the morphology of the object features is compared with the distribution map of the outdoor environment types to gradually build a three-dimensional model of the outdoor environment.

[0042] At this point, the system needs to provide a detailed morphological description and labeling of the object features previously determined through image recognition. This includes recording each object's size (such as height, width, depth, etc.), shape (such as round, square, irregular, etc.), color, texture, and any other features that help distinguish and identify objects. This information will be used for subsequent image comparison and three-dimensional model construction; optionally, the system uses feature extraction algorithms in computer vision technology to automatically identify and label the morphology of object features. These algorithms extract the morphological information of objects based on methods such as edge detection, contour extraction, and texture analysis; in addition, the system also uses deep learning models (such as convolutional neural networks) to learn and identify the morphological features of different objects and automatically label them.

[0043] The system needs to perform image comparison between the object features with marked morphology and the distribution map of outdoor environment types. This comparison involves matching the morphological information of the object features with the environmental type information in the distribution map to determine the position and relative relationship of the object features in the environment. Through comparison, the system further refines the description of the outdoor environment and prepares to build a three-dimensional model. Optionally, the system uses techniques such as image registration, image fusion or image stitching to align and fuse the morphology of the object features with the distribution map. These techniques allow the system to find common reference points between different images and combine them into a coherent view. During the comparison process, the system also uses spatial relationship reasoning to evaluate the relative position and relationship between object features to ensure the accuracy of the three-dimensional model.

[0044] The system uses the previously extracted object feature morphology and distribution map information to gradually build a three-dimensional model of the outdoor environment. This model is a three-dimensional, interactive representation of real objects and the environment. The system uses three-dimensional modeling technology (such as three-dimensional scanning, three-dimensional reconstruction, three-dimensional rendering, etc.) to create this model and allows users to view and interact with it through blind smart glasses. At the same time, the system uses technologies such as structure from motion (SfM) or stereo vision to extract three-dimensional information from two-dimensional images and build a three-dimensional model of the outdoor environment. These technologies use feature point matching and depth estimation in the image to restore the three-dimensional structure of the scene. In addition, the system also uses a three-dimensional rendering engine to render the model into a realistic image and provide it to the user for viewing and navigation.

[0045] Specifically, suppose a blind user is wearing smart glasses and walking in a park in the city; the system uses image recognition to determine the features of objects around the user, including trees, benches, street lamps, and flower beds; the system performs detailed morphological markings on these objects, such as recording the height and crown shape of the trees, the size and material of the benches, the height and shape of the street lamps, and the boundaries of the flower beds and the types of plants inside.

[0046] The system compares these morphologically marked object features with a distribution map of outdoor environment types (in this case, a distribution map of park green spaces). Through comparison, the system determines the exact position and relative relationship of each object feature in the park green space, such as trees next to flower beds, benches close to street lights, etc. Based on this information, the system gradually constructs a three-dimensional model of the outdoor environment, which includes the three-dimensional structure of the park green space, the three-dimensional shapes of trees and flower beds, the precise positions of benches and street lights, etc. Users view this model through the display screen of the blind smart glasses and interact with it, such as zooming in or out, rotating the model to view different angles, etc. Through this three-dimensional model, blind users can understand the surrounding environment more intuitively and make better navigation and decisions. For example, they determine which direction has trees to provide shade, which position has benches for rest, and how to bypass obstacles.

[0047] In one embodiment of the present application, a detailed morphological description of the identified object features is performed and recorded in a weighted matching table; the weighted matching table includes the object feature's unique identifier, name, morphological description (such as size, shape, color, etc.), and weight value (used for subsequent calculation of the matching score); the weighted matching table is shown in Table 2: Table 2 Weight matching table

[0048] Suppose there is an object feature "tree" (feature ID: 001), whose morphological description matches the "tree" feature in the park green space feature set; the score is calculated based on their similarity (assumed to be 0.9, indicating very similar) and weight (3): score = similarity * weight = 0.9*3 = 2.7; suppose after calculation, the following matching scores are obtained: park green space: total score = 2.7 (trees) + 2 (bench) + 1 (flower bed) = 5.7; city street: total score = 2 (street light) = 2 (because other features in the city street feature set do not match the identified object features); since the total score of the park green space is much higher than that of the city street, the park green space is selected as the main environment type of the current location, and object features such as trees, benches and flower beds are added to the three-dimensional model; the user now views this three-dimensional model containing the park green space and surrounding objects through blind smart glasses.

[0049] refer to Figure 4 In step S13, in the stereoscopic coordinate system of the stereoscopic model, the primary direction of the blind smart glasses in the outdoor environment is determined according to the current position of the blind smart glasses and the current positions of the various features of the stereoscopic model; In the specific implementation process of the present invention, the specific steps are: S131: Constructing a corresponding 3D coordinate system for the 3D model, and sequentially marking the blind smart glasses and various object features in the 3D model to mark corresponding current positions of the blind smart glasses and various object features; S132: Determine the relative position of the blind smart glasses relative to each feature based on a positional comparison between the current position of the blind smart glasses and the current position of each feature, and determine the corresponding relative orientation based on an analysis of the relative position of the blind smart glasses relative to each feature. At this time, the center position of the blind smart glasses and the center position of each feature are compared; S133: Collect the orientation of the blind smart glasses, determine the target orientation according to the orientation of the blind smart glasses and the current position of the blind smart glasses, and determine the primary orientation of the blind smart glasses in the outdoor environment according to multiple relative orientations, target orientations and direction information of the three-dimensional coordinate system.

[0050] In an embodiment of the present application, a corresponding three-dimensional coordinate system is constructed for the three-dimensional model, and the blind smart glasses and various object features in the three-dimensional model are marked in sequence to mark the corresponding current positions of the blind smart glasses and various object features, thereby introducing the current positions of the blind smart glasses and the current positions of various object features.

[0051] At this point, a three-dimensional coordinate system is constructed for the stereo model. This coordinate system is typically composed of three mutually perpendicular axes: the X-axis (usually representing the horizontal direction, such as east-west), the Y-axis (usually representing the vertical direction, such as up-down), and the Z-axis (usually representing the depth direction, i.e., the vertical component of the front-back or north-south direction, but simplified here to be the direction perpendicular to the XY plane). The origin of the coordinate system is set at the center of the stereo model or the initial position of the blind smart glasses, depending on the specific application scenario and requirements. After constructing the coordinate system, the current position of the blind smart glasses needs to be marked in the coordinate system. This is usually achieved by calculating the displacement of the smart glasses relative to the origin of the coordinate system. The position of the smart glasses is determined by the data of its built-in GPS sensor, accelerometer, gyroscope and other sensors, and is also estimated by matching the features of the surrounding environment.

[0052] In addition to marking the position of the smart glasses, the position of each object feature in the three-dimensional model also needs to be marked in the coordinate system. These object features are trees, buildings, street lights, etc. The position of the object features is determined by image recognition algorithms. These algorithms identify the objects in the image and calculate their position in the image. These algorithms are then combined with the focal length, viewing angle and other parameters of the camera to convert these positions into coordinates in the three-dimensional coordinate system.

[0053] Specifically, suppose a blind user is walking in a park with several trees, a bench, and a street lamp. The blind smart glasses have already built a three-dimensional model of these features through the previous steps. The center of the park is selected as the origin of the coordinate system, with the X-axis pointing to the east boundary of the park, the Y-axis pointing to the sky (directly above), and the Z-axis perpendicular to the XY plane, pointing to the north boundary of the park (in this example, the Z-axis is simplified to be perpendicular to the ground).

[0054] The blind smart glasses determine their own position through the built-in GPS sensor and accelerometer. This position is expressed as (x_eye, y_eye, z_eye) in the coordinate system; for example, suppose the position of the smart glasses is (10, 2, -5), which means that it is 10 meters east of the center of the park, 2 meters above the ground (that is, 2 meters above the ground), and slightly towards the southern boundary of the park (because the Z axis is north, so negative values indicate a deviation to the south).

[0055] Using an image recognition algorithm, the positions of trees, benches, and streetlights in the image are determined, and these positions are converted into coordinates in a three-dimensional coordinate system based on the camera parameters. For example, a tree's position is (15, 3, -3), indicating that it is 15 meters east of the park center, 3 meters above the ground, and slightly toward the park's northern boundary. A bench's position is (8, 1.5, -7), indicating that it is 8 meters east of the park center, 1.5 meters above the ground, and slightly toward the park's southern boundary. A streetlight's position is (20, 5, 0), indicating that it is 20 meters east of the park center, 5 meters above the ground (the height of the streetlight), and directly on the park's north-south centerline. These steps mark the positions of the blind smart glasses and the features of each object in the three-dimensional coordinate system, providing a basis for subsequently determining relative position and orientation.

[0056] Furthermore, the relative position of the blind smart glasses relative to each feature is determined based on the position comparison between the current position of the blind smart glasses and the current position of each feature, and the corresponding relative orientation is determined based on the analysis of the relative position of the blind smart glasses relative to each feature. At this time, the center position of the blind smart glasses and the center position of each feature are compared, which is compatible with the overall consideration of the analysis of the relative position of the blind smart glasses relative to each feature, thereby ensuring the accuracy of the corresponding relative orientation.

[0057] At this point, the current position of the blind smart glasses is compared with the current position of each object feature, which usually involves calculating the straight-line distance (Euclidean distance) or vector between the smart glasses and each feature in a three-dimensional stereo coordinate system; the position of the smart glasses and the position of each feature have been marked in the stereo coordinate system in the previous step.

[0058] By comparing positions, the relative position of the smart glasses relative to each feature is determined. This is usually achieved by calculating the vector from the smart glasses to each feature. The direction and length of the vector represent the relative orientation and distance respectively. The relative position is expressed in polar coordinates (distance and angle) or Cartesian coordinates (X, Y, Z offset relative to the smart glasses).

[0059] Once the relative position vector of the smart glasses relative to each feature is obtained, the vector is parsed to determine the relative orientation; the relative orientation is usually expressed in eight cardinal directions (east, southeast, south, southwest, west, northwest, north, northeast) or with finer angular resolution; to determine the relative orientation, the angle between the vector and the coordinate system axis is calculated, and the orientation is determined based on the value of the angle. At the same time, when determining the relative position and orientation, the center position of the smart glasses is usually compared with the center position of each feature. This is because the center position provides a simple and intuitive reference point that helps users understand the layout of the surrounding environment.

[0060] Therefore, the orientation of the blind smart glasses is collected, the target orientation is determined according to the orientation of the blind smart glasses and the current position of the blind smart glasses, and the primary direction of the blind smart glasses in the outdoor environment is determined according to multiple relative orientations, target orientations and direction information of the stereo coordinate system. Preliminary direction recognition is performed in the stereo model, which is compatible with the overall consideration of the orientation of the blind smart glasses and the current position of the blind smart glasses, thereby ensuring the accuracy of the target orientation.

[0061] At this time, the orientation of the blind smart glasses is collected, which is usually achieved through the built-in sensors of the smart glasses (such as gyroscopes, accelerometers and magnetometers). These sensors can detect changes in the orientation of the blind smart glasses. Optionally, the magnetometer is used to detect the direction of the earth's magnetic field, thereby determining the absolute orientation of the blind smart glasses (such as north, south, etc.). The gyroscope and accelerometer are used to detect changes in angular velocity and acceleration of the blind smart glasses to help track the dynamic orientation of the blind smart glasses.

[0062] Once the orientation of the smart glasses is known, the target orientation is determined based on the user's intention or instruction, which is the direction specified by the user through voice commands or the direction implied by the user through head movements; the target orientation is usually expressed as an angle relative to the current orientation of the device, which is an angular offset in a clockwise or counterclockwise direction.

[0063] In this step, the relative orientation of the smart glasses relative to surrounding features, the target orientation, and the directional information of the stereo coordinate system need to be combined to determine the primary orientation of the smart glasses in the outdoor environment. This typically involves a geometric analysis of the current position of the smart glasses, the target orientation, and the relative positions of surrounding features in the stereo coordinate system. The primary orientation is the approximate direction the smart glasses need to move to reach the target location or feature of interest to the user. Ultimately, the primary orientation is determined based on the results of the geometric analysis and presented to the user through voice prompts, tactile feedback, or other interactive methods suitable for blind users.

[0064] Specifically, suppose a blind user is walking in a park and wants to know how to get to a nearby bench. The blind smart glasses have already built a stereo model of the surrounding features through the previous steps and determined the relative orientation of the smart glasses relative to these features. The smart glasses detect through built-in sensors that the user is currently facing north. The user specifies through voice commands that he wants to go to the bench. Based on the stereo model and the relative orientation information, it is known that the bench is located in the southwest direction relative to the smart glasses. Therefore, the target direction is southwest, which requires a counterclockwise rotation of approximately 135 degrees (or a clockwise rotation of 45 degrees, but a smaller angular offset is usually chosen as a guide) relative to the north currently facing the user.

[0065] In the stereo coordinate system, the current position of the smart glasses (e.g., (10, 2, -5)), the target orientation (southwest direction), and the relative position of the bench (e.g., in the direction of (-5, 0, -1) relative to the smart glasses) are analyzed; through geometric analysis, it is determined that the smart glasses need to move a certain distance in the southwest direction to reach the bench; finally, the primary direction is determined to be the southwest direction, and the user is prompted through voice: "Please turn left and go straight, you will reach the bench;" or tactile feedback is used to provide direction guidance on the smart glasses; through these steps, the primary direction of the blind smart glasses in outdoor environments is determined, and the blind user is provided with accurate navigation information on how to reach the target location.

[0066] In one embodiment of the present application, a preset primary direction matching table is collected. The primary direction matching table maps the orientation of the smart glasses, the target orientation, and the relative relationship between them to specific primary directions. The primary direction matching table is shown in Table 3: Table 3 Primary direction matching table

[0067] refer to Figure 5 In step S14, the ultrasonic positioning information and the front image are determined based on the external detection of the blind smart glasses, and the direction deviation value is determined according to the ultrasonic positioning information, the front image, and the features recorded by the three-dimensional model in the primary direction; In the specific implementation process of the present invention, the specific steps are: S141: The blind smart glasses perform external detection when moving, and trigger ultrasonic positioning detection and image detection of the blind smart glasses; S142: Determining a moving area of the blind smart glasses according to the current position of the blind smart glasses and the three-dimensional model, and determining a priority of ultrasonic positioning detection and a priority of image detection based on the shape of the moving area and features of objects distributed in the moving area; S143: If the priority of ultrasonic positioning detection is lower than that of image detection, the front image determined by image detection is collected first, and the first deviation range is determined based on the comparison between the front image and the features recorded in the primary direction of the stereo model. The second deviation range is determined according to the front image and the ultrasonic positioning information determined by ultrasonic positioning detection, and the direction deviation value is determined based on the first deviation range and the second deviation range.

[0068] In an embodiment of the present application, the blind smart glasses perform external detection when moving and trigger the ultrasonic positioning detection and image detection of the blind smart glasses. Ultrasonic positioning detection and image detection are introduced to achieve multi-dimensional detection.

[0069] At this time, when the blind smart glasses are in a moving state, they will continue to detect the external environment. This detection is real-time and aims to ensure the safety of the user's walking and the accuracy of navigation. External detection includes the fusion of multiple sensor data, such as accelerometers, gyroscopes, etc., to monitor the movement status and direction changes of the glasses.

[0070] During external detection, if the system detects obstacles, unknown environments, or scenarios that require more precise distance measurements, it triggers ultrasonic positioning detection. Ultrasonic positioning detection calculates the distance between the smart glasses and surrounding objects by emitting ultrasonic waves and receiving the reflected signals. This technology is particularly effective for detecting obstacles at close range.

[0071] At the same time, the system also triggers image detection, which is usually achieved through the camera on the smart glasses to capture images of the environment in front; image detection not only helps identify objects and features in front, but also compares them with pre-built stereo models to provide more accurate navigation information.

[0072] Specifically, suppose a blind user is walking in the corridor of a shopping mall with the goal of going to a specific store; the blind smart glasses are providing navigation services to the user; when the user starts walking, the sensors of the smart glasses begin to monitor the user's movement status and direction changes in real time; for example, the accelerometer detects that the user is walking forward, and the gyroscope detects the user's slight turning movements.

[0073] While walking, the ultrasonic sensor of the smart glasses suddenly detected an unknown obstacle (a temporarily placed shelf) not far ahead; for safety reasons, the system immediately triggered ultrasonic positioning detection, emitting ultrasonic waves and receiving reflected signals; by calculating the round-trip time of the ultrasonic waves, the system determined the precise distance between the obstacle and the smart glasses, for example, 2 meters.

[0074] At the same time, the camera of the smart glasses also captured the image in front; through image recognition technology, the system identified that the obstacle was a shelf with some goods placed on the shelf; in addition, the image was compared with the pre-built three-dimensional model of the shopping mall to confirm the user's current location and the relative position of the store in front; in summary, step S141 ensures that the blind smart glasses can detect the external environment in real time during movement, and trigger ultrasonic positioning detection and image detection when necessary to provide accurate distance measurement and object recognition information, thereby supporting safer and more accurate navigation services; in this example, through the combined use of ultrasonic positioning detection and image detection, the system successfully identified the obstacle in front and determined the exact distance between the user and the obstacle, while confirming the user's current location and the relative position of the store in front.

[0075] Furthermore, the moving area of the blind smart glasses is determined according to the current position of the blind smart glasses and the stereoscopic model, and the priority of ultrasonic positioning detection and the priority of image detection are determined based on the shape of the moving area and the characteristics of objects distributed in the moving area. This is compatible with the overall consideration of the shape of the moving area and the characteristics of objects distributed in the moving area, ensuring the accuracy of the priority of ultrasonic positioning detection and the priority of image detection.

[0076] At this point, the blind smart glasses first determine their current location through the built-in GPS or other positioning technology; then, they match this location information with the previously constructed stereo model to determine the user's specific location in the current environment.

[0077] Based on the current location and the user's walking direction (provided by the orientation sensor of the smart glasses), the system estimates a movement area, which is usually a fan-shaped or elliptical area centered on the current location and with the user's expected walking distance as the radius; the shape and size of the movement area will be adjusted according to factors such as the user's walking speed and the complexity of the environment.

[0078] The system analyzes the terrain, obstacle distribution, road conditions and other morphological information within the moving area. At the same time, it also identifies the features of objects distributed within the moving area, such as buildings, trees, traffic signs, etc., and evaluates the importance of these features to navigation. Based on the morphology and object characteristics of the moving area, the system dynamically adjusts the priority of ultrasonic positioning detection and image detection. For example, in narrow or obstacle-filled areas, the priority of ultrasonic positioning detection will be increased because it can more accurately measure the distance to obstacles. In open or feature-rich areas, image detection will have a higher priority because image recognition can provide richer environmental information.

[0079] Specifically, suppose a blind user is walking in a city park, with the goal of reaching the fountain in the center of the park; the blind smart glasses are providing navigation services to the user; the smart glasses use GPS positioning technology to determine that the user is currently at the entrance of the park, near a parking lot; it matches this location information with the previously constructed three-dimensional model of the park to confirm the user's specific location.

[0080] Based on the current location and the user's walking direction (towards the center of the park), the system estimates a fan-shaped movement area with the current location as the center and the user's expected walking distance as the radius. This area covers the user's expected path from the entrance to the fountain; the system analyzes the terrain within the movement area and finds that the path is mainly flat grass and trails; it also identifies several key object features on the path, including a flower bed, several benches and some trees. These features are relatively less important for navigation because they do not hinder the user's walking; however, in the area near the fountain, the system detects a narrow passage with tall bushes on both sides, which poses certain challenges to the user's walking.

[0081] In most of the moving area, due to the flat terrain and obvious features, the system decided to increase the priority of image detection to capture the environmental image in front and identify key features; but in the narrow channel area close to the fountain, the system increased the priority of ultrasonic positioning detection to ensure that the user can accurately measure the distance to the bushes on both sides to avoid collision; in summary, step S142 ensures that the blind smart glasses can determine the moving area based on the current position and stereo model, and dynamically adjust the priority of ultrasonic positioning detection and image detection according to the morphology and object characteristics of the moving area; in this example, the system successfully identified the key features in the park and adjusted the detection priority according to the characteristics of different areas to provide safer and more accurate navigation services.

[0082] Therefore, if the priority of ultrasonic positioning detection is lower than that of image detection, the front image determined by image detection is collected first, and the first deviation range is determined based on the comparison between the front image and the features recorded in the primary direction of the stereo model. The second deviation range is determined according to the front image and the ultrasonic positioning information determined by ultrasonic positioning detection. The directional deviation value is determined based on the first deviation range and the second deviation range, which is compatible with the overall consideration of the first deviation range and the second deviation range, and ensures the accuracy of the directional deviation value.

[0083] At this point, the system has determined the priority of ultrasonic positioning detection and image detection based on the current position, moving area and object characteristics of the blind smart glasses; if the priority of ultrasonic positioning detection is lower than the priority of image detection, the system will give priority to image detection.

[0084] Specifically, suppose a blind user is walking in a large shopping mall with the goal of heading to a specific store. The blind smart glasses have already provided the user with preliminary navigation instructions and determined the primary direction. In this scenario, the interior environment of the shopping mall is relatively open and the store layout is clear, so the system determines that image detection has a higher priority than ultrasonic positioning detection.

[0085] The camera of the smart glasses will capture an image of the environment in front, namely the front image, which will be used for subsequent feature comparison and deviation calculation; the system will compare the front image with the features recorded by the three-dimensional model in the primary direction (that is, the user's expected walking direction); through comparison, the system will identify the differences between the objects in the image and the model features, thereby determining a preliminary directional deviation range, namely the first deviation range.

[0086] Specifically, the camera of the smart glasses captured the layout image of the store in front, including the store's sign, shelves and other features; the system compared the captured image with the store layout in the three-dimensional model; through comparison, the system found that there was a slight mismatch between the position of the store sign in the image and the position in the model, and preliminarily judged that the direction deviation was about 5 degrees.

[0087] At the same time, although ultrasonic positioning detection has a lower priority, the information it provides still plays an important role in determining the direction deviation value; the system combines the object position in the front image with the object distance information determined by ultrasonic positioning detection, and calculates a more accurate direction deviation range, namely the second deviation range. Based on the first deviation range and the second deviation range, the system finally determines a direction deviation value through comprehensive analysis and calculation. This direction deviation value reflects the actual degree of deviation between the current direction of the blind smart glasses and the primary direction, and is used to adjust navigation instructions and provide more accurate navigation information.

[0088] Specifically, although ultrasonic positioning detection has a lower priority in this scenario, the system still uses it to measure the distance between the user and the shelf in front; by combining the shelf position information in the image and the distance information of ultrasonic positioning detection, the system performs a more accurate calculation and concludes that the direction deviation is approximately 3 degrees; combining the first deviation range (5 degrees) and the second deviation range (3 degrees), the system ultimately determines the direction deviation value to be 4 degrees; based on this information, the system adjusts the navigation instructions, reminding the user to turn slightly to the left or right by a certain angle (such as 4 degrees) to ensure more accurate travel to the target store; this example shows that step S143 ensures that the blind smart glasses can comprehensively determine the direction deviation value based on the information of image detection and ultrasonic positioning detection, thereby providing more accurate navigation services. This comprehensive detection and analysis method improves navigation accuracy and user safety.

[0089] In one embodiment of the present application, it is assumed that the weight of the first deviation range is 0.6 and the weight of the second deviation range is 0.4; it is assumed that the first deviation range is 3 degrees and the second deviation range is 2 degrees; the direction deviation value = first deviation range * weight 1 + second deviation range * weight 2 direction deviation value = 3 degrees * 0.6 + 2 degrees * 0.4 = 1.8 degrees + 0.8 degrees = 2.6 degrees.

[0090] Image detection takes priority over ultrasonic positioning detection, and the front image is collected and compared with the stereo model; the first deviation range is calculated to be 3 degrees; the ultrasonic positioning information is combined with the image information, and the second deviation range is calculated to be 2 degrees; the comprehensive weight distribution (first deviation range 0.6, second deviation range 0.4) determines the direction deviation value to be 2.6 degrees; the navigation instruction is adjusted to remind the user to fine-tune 2.6 degrees to the left; through the above example, we can see how step S143 integrates the information of image detection and ultrasonic positioning detection in the navigation system of the blind smart glasses, accurately calculates the direction deviation value, and provides more accurate navigation services. This method combines the intuitiveness of the matching table and the accuracy of the weight score to ensure the reliability and accuracy of the navigation system.

[0091] refer to Figure 6 In step S15, the final direction of the blind smart glasses in the outdoor environment is determined according to the direction deviation value, the primary direction and the moving direction of the blind smart glasses; In the specific implementation process of the present invention, the specific steps are: S151: monitoring the moving direction of the blind smart glasses in real time, where the moving direction changes dynamically within a preset time, and determining a corresponding direction change amount according to the dynamic change in the moving direction of the blind smart glasses; S152: Determine a first direction parameter according to the primary direction and the direction deviation value, and determine a second direction parameter according to the primary direction and the direction change; S153: Collect the moving area of the blind smart glasses relative to the three-dimensional model, and match the corresponding direction mapping relationship according to the shape of the moving area and the position information of the moving area. The direction mapping relationship serves as the preset relationship of the blind smart glasses; determine the final direction of the blind smart glasses in the outdoor environment according to the first direction parameter, the second direction parameter and the corresponding direction mapping relationship.

[0092] In an embodiment of the present application, the moving direction of the blind smart glasses is monitored in real time, the moving direction changes dynamically within a preset time, and the corresponding direction change amount is determined according to the dynamic change of the moving direction of the blind smart glasses, thereby introducing the direction change amount.

[0093] At this time, the smart glasses for the blind are equipped with a variety of built-in sensors, such as gyroscopes, accelerometers and magnetometers, which can detect the movement direction and posture changes of the glasses (and the wearer) in real time; the system continuously monitors the movement direction of the glasses by reading the data from these sensors; optionally, the system starts the sensor monitoring program to collect sensor data at preset time intervals (such as every second or every half second) and update the movement direction information of the glasses in real time.

[0094] Since the direction of the user will change due to terrain, obstacles or their own intentions while walking or moving, the direction of movement of the glasses will also change dynamically; the system needs to be able to capture these changes and analyze and process them within a preset time window; optionally, the system sets a time window (such as the past 1 second or 2 seconds), within which the system continuously records and analyzes changes in the direction of movement of the glasses.

[0095] Based on the movement direction data collected in step 2, the system needs to calculate the actual change in the movement direction of the glasses within the preset time. This change is an angle change (such as how many degrees of deflection) or a direction vector change (such as a direction vector change in three-dimensional space). At the same time, the system analyzes the sensor data through an algorithm to calculate the change in the movement direction of the glasses within the preset time window, and uses this change as the basis for subsequent navigation adjustments.

[0096] Specifically, suppose a blind user is walking on a straight street with the goal of reaching the bus stop ahead; the blind smart glasses have provided the user with preliminary navigation instructions and determined the general direction to the bus stop; after the user starts walking, the sensors of the smart glasses begin to collect the user's movement direction data in real time, including the user's walking speed, direction deflection angle and other information.

[0097] When a user is walking, their direction may change slightly due to slight bends in the street or the user's own walking habits. The system sets a time window (such as the past 1 second) and continuously records and analyzes changes in the user's movement direction within this time window. By analyzing sensor data, the system finds that the user's direction has deviated by about 3 degrees in the past 1 second. This change is recorded and used as the basis for subsequent navigation adjustments.

[0098] Furthermore, a first direction parameter is determined according to the primary direction and the direction deviation value, and a second direction parameter is determined according to the primary direction and the direction change, thereby introducing the first direction parameter and the second direction parameter.

[0099] At this time, for the first direction parameter, the first direction parameter is determined based on the primary direction and the direction deviation value. The primary direction is the target direction set by the user, which is usually determined by the navigation system of the blind smart glasses or user input; the direction deviation value is calculated through the previous step (such as S143), which reflects the direction difference between the current position and the target position; optionally, the system calculates the first direction parameter based on the primary direction and the direction deviation value. This first direction parameter is the target direction after taking the direction deviation into account, and is used to guide the user on how to adjust the current walking direction to get closer to the target.

[0100] Specifically, suppose a blind user is walking on a street with the goal of heading to the subway station ahead; the blind smart glasses have provided the user with preliminary navigation instructions and determined the primary direction to the subway station; the primary direction set by the user is to go straight to the subway station; however, due to reasons such as street layout or building obstruction, the system calculates the direction deviation value as 5 degrees to the right through the previous steps; therefore, the first direction parameter is the target direction after taking the direction deviation into account, that is, the user needs to adjust the direction slightly to the left to offset this deviation and continue to go straight to the subway station.

[0101] For the second direction parameter, the second direction parameter is determined based on the primary direction and the direction change. The direction change is monitored and calculated in real time through step S151. It reflects the actual direction adjustment of the user during walking. This change is caused by various factors, such as terrain changes, user intentions or obstacles. Optionally, the system combines the primary direction and the direction change to calculate the second direction parameter through an algorithm. This parameter takes into account the current direction after the user's actual walking direction changes, and is used to evaluate the degree of deviation between the user's current walking state and the target direction.

[0102] Optionally, during walking, the system discovers through real-time monitoring that the user has actually deviated to the right by 2 degrees, and this direction change is calculated through step S151; therefore, the second direction parameter is the current direction after taking into account the change in the user's actual walking direction; the second direction parameter indicates that the user's current direction has deviated slightly from the primary direction, but the degree of deviation is less than the direction deviation value, because the user has unconsciously adjusted the direction to cope with the deviation; based on the first direction parameter and the second direction parameter, the system further adjusts the navigation instructions; for example, if the second direction parameter shows that the user is close to the first direction parameter, the system does not need to issue additional adjustment instructions; however, if the second direction parameter shows that the user is still far from the target direction, the system will issue an instruction to remind the user to turn left or right a certain angle to get closer to the target direction; from this example, it can be seen that step S152 ensures that the blind smart glasses can comprehensively consider the primary direction, direction deviation value and the user's actual walking direction changes, thereby providing users with more accurate navigation guidance.

[0103] Therefore, the moving area of the blind smart glasses relative to the three-dimensional model is collected, and the corresponding direction mapping relationship is matched according to the shape of the moving area and the position information of the moving area. The direction mapping relationship is used as the preset relationship of the blind smart glasses; the final direction of the blind smart glasses in the outdoor environment is determined according to the first direction parameter, the second direction parameter and the corresponding direction mapping relationship, which is compatible with the overall consideration of the first direction parameter, the second direction parameter and the corresponding direction mapping relationship, ensuring the accuracy of the final direction of the blind smart glasses in the outdoor environment. At the same time, it is compatible with the overall consideration of the direction deviation value, the primary direction and the moving direction of the blind smart glasses, and fully considers the dynamic movement of the blind smart glasses, further ensuring the accuracy of the final direction of the blind smart glasses in the outdoor environment, realizing multi-level recognition from the primary direction to the final direction, and thus realizing the direction positioning of the blind in the outdoor environment.

[0104] At this time, the blind smart glasses use built-in GPS, sensors (such as gyroscopes, accelerometers) and environmental perception devices (such as cameras, lidar, etc.) to collect real-time information about their own moving area relative to a preset three-dimensional model. This three-dimensional model is usually a digital environmental map that contains key information such as roads, buildings, and obstacles. Optionally, the system continuously collects the glasses' location data (longitude, latitude, altitude), orientation data, and environmental feature data, and compares this data with the three-dimensional model to determine the specific area where the glasses are currently located.

[0105] The direction mapping relationship is pre-set and is based on the relationship between each location point in the three-dimensional model and the direction in the actual environment. These relationships are obtained through field surveys, drone aerial photography, satellite image analysis, etc., and are integrated into the navigation system; optionally, the system finds the corresponding area in the three-dimensional model based on the moving area information collected in step 1, and matches the direction mapping relationship of the area. This relationship tells the system which direction to walk in order to reach the target location starting from the current location.

[0106] After obtaining the first direction parameter (the target direction after considering the direction deviation), the second direction parameter (the actual walking direction of the user) and the direction mapping relationship, the system needs to integrate this information to determine the final direction of the glasses (and the user) in the outdoor environment; optionally, the first direction parameter, the second direction parameter and the direction mapping relationship are integrated to calculate the optimal walking direction, which takes into account the user's actual walking status, the direction information in the environmental map, and the previously calculated direction deviation; finally, the system outputs this direction to the user as a navigation instruction.

[0107] Specifically, suppose a blind user is walking in a complex urban block with the goal of going to a specific store. The blind smart glasses have provided the user with preliminary navigation instructions and determined the primary direction to the store. After the user starts walking, the smart glasses use GPS and sensors to collect their own position, orientation, and surrounding environment characteristics in real time. This data is sent to the navigation system and compared with a preset three-dimensional model. The system determines that the user is currently near an intersection and facing north. Based on the information in the three-dimensional model, the system knows that from the current position (intersection), in order to reach the target store, the user needs to turn right (i.e., walk east). This direction mapping relationship is based on field surveys and satellite image analysis and is integrated into the navigation system.

[0108] After obtaining the first direction parameter (assuming the target direction is slightly deviated to the east to take into account the direction deviation), the second direction parameter (the user is currently facing north) and the direction mapping relationship (turn right), the system calculates that the final direction is that the user should turn right (i.e., east) and walk; the system outputs this direction as a navigation instruction to the user through voice prompts or vibration feedback; this example shows that step S153 ensures that the blind smart glasses can comprehensively consider the user's actual walking status, the direction information in the environment map and the previously calculated direction deviation, thereby providing the user with accurate and practical navigation guidance.

[0109] In one embodiment of the present application, in actual applications, the direction mapping relationship is more complex and multiple factors need to be considered; in order to more accurately determine the final direction, a weight and score method is adopted; Assume the following weight distribution: first direction parameter (target direction after considering direction deviation): 50% weight; second direction parameter (user's actual walking direction): 30% weight; direction mapping relationship: 20% weight.

[0110] Now, suppose there are the following direction parameters and mapping relationships: First direction parameter: 10 degrees east of north; Second direction parameter: walking east; Direction mapping relationship: Based on the current location, it is recommended to walk 5 degrees east of south to avoid obstacles; A score is calculated for each direction and a weighted average is taken to determine the final direction; for example: 10 degrees east of north: 50% * 100 points (assuming the full score is 100 points, indicating that it is completely in line with the target direction); walking to the east: 30% * 90 points (slightly deviating from the target direction, but still within an acceptable range); 5 degrees east of south: 20% * 80 points (taking into account environmental obstacles, but deviating slightly from the target direction); weighted total score = (50% *100) + (30% *90) + (20% *80) = 50 + 27 + 16 = 93 points (assuming a comprehensive score to evaluate the quality of the direction); then, the final direction is determined based on the weighted total score and the score ratio of each direction; in this example, 10 degrees east of north has the highest score, so the system recommends that the user continue walking in the east of north direction.

[0111] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of a direction recognition system for blind smart glasses in an outdoor environment according to an embodiment of the present invention; the direction recognition system for blind smart glasses in an outdoor environment includes: The outdoor environment type module 21 is used to determine the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment types include outdoor roads, outdoor parks, or outdoor sports fields; A stereo module 22 is configured to construct a stereo model of the outdoor environment based on the distribution map of outdoor environment types and the surrounding image; a primary direction module 23 for determining, in the stereoscopic coordinate system of the stereoscopic model, a primary direction of the blind smart glasses in the outdoor environment according to the current position of the blind smart glasses and the current positions of the various features of the stereoscopic model; a direction deviation value module 24 for determining ultrasonic positioning information and a front image based on external detection of the blind smart glasses, and determining a direction deviation value according to the ultrasonic positioning information, the front image, and features recorded by the stereo model in the primary direction; The final direction module 25 is used to determine the final direction of the blind smart glasses in the outdoor environment according to the direction deviation value, the primary direction and the moving direction of the blind smart glasses.

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for direction recognition using smart glasses for the blind in an outdoor environment, characterized in that: include: Determine the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; Types of outdoor environments include outdoor roads, outdoor parks, or outdoor sports fields; constructing a three-dimensional model of the outdoor environment based on the distribution map of outdoor environment types and the surrounding image; Determining, in the stereoscopic coordinate system of the stereoscopic model, a primary orientation of the blind smart glasses in the outdoor environment based on a current position of the blind smart glasses and current positions of various features of the stereoscopic model; Determine ultrasonic positioning information and a front image based on external detection of the blind smart glasses, and determine a direction deviation value according to the ultrasonic positioning information, the front image, and features recorded by the three-dimensional model in the primary direction; The final direction of the blind smart glasses in the outdoor environment is determined according to the direction deviation value, the primary direction and the moving direction of the blind smart glasses.

2. The method for direction recognition of smart glasses for the blind in an outdoor environment according to claim 1, characterized in that: The method of determining the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; Outdoor environments include outdoor roads, outdoor parks, or outdoor sports fields, including: The multiple peripheral cameras of the blind smart glasses shoot in different directions and collect multiple peripheral images. The features of each object are determined based on the recognition of the multiple peripheral images, and the relative positions of the blind smart glasses and each object feature are marked. Each sound collector of the blind smart glasses collects multiple ambient sounds, and triggers the matching of multiple ambient sounds with object features based on the collection directions of the multiple ambient sounds and the response signals of each sound collector, thereby introducing the sound information of the object features; The first outdoor parameter is determined according to the sound information of the object feature and the shape of the object feature, the second outdoor parameter is determined according to the relative position of the blind smart glasses and the object feature and the shape of the object feature, and the outdoor environment type is determined based on the first outdoor parameter, the second outdoor parameter and the outdoor mapping relationship.

3. The method for direction recognition of blind smart glasses in outdoor environments according to claim 1, characterized in that: The step of constructing a three-dimensional model of the outdoor environment based on the distribution map of outdoor environment types and the surrounding image includes: Collecting location information of the blind smart glasses, filtering a town name from the location information of the blind smart glasses, and determining an environmental area where the blind smart glasses are located based on the town name, a town database, and the location information of the blind smart glasses; Comparing the environmental area with the outdoor environmental types and generating a distribution map of the outdoor environmental types; collecting the surrounding image and determining the corresponding object features based on the recognition of the surrounding image; The shapes of the object features are marked, and the shapes of the object features are compared with the distribution map of the outdoor environment types to gradually build a three-dimensional model of the outdoor environment.

4. The method for direction recognition of blind smart glasses in outdoor environments according to claim 1, characterized in that: The method of determining the primary direction of the blind smart glasses in the outdoor environment according to the current position of the blind smart glasses and the current positions of the various features of the three-dimensional model in the three-dimensional coordinate system of the three-dimensional model includes: Constructing a corresponding 3D coordinate system for the 3D model, and marking the blind smart glasses and various object features in the 3D model in sequence, so as to mark the corresponding current positions of the blind smart glasses and various object features; The relative position of the blind smart glasses relative to each feature is determined based on the position comparison between the current position of the blind smart glasses and the current position of each feature, and the corresponding relative orientation is determined based on the analysis of the relative position of the blind smart glasses relative to each feature. At this time, the center position of the blind smart glasses and the center position of each feature are compared.

5. The method for direction recognition of smart glasses for the blind in an outdoor environment according to claim 4, characterized in that: The method further comprises determining the primary direction of the blind smart glasses in the outdoor environment according to the current position of the blind smart glasses and the current positions of the various features of the three-dimensional model in the three-dimensional coordinate system of the three-dimensional model, and further comprising: The orientation of the blind smart glasses is collected, the target orientation is determined according to the orientation of the blind smart glasses and the current position of the blind smart glasses, and the primary orientation of the blind smart glasses in the outdoor environment is determined according to multiple relative orientations, target orientations and direction information of the stereo coordinate system.

6. The method for direction recognition of blind smart glasses in outdoor environments according to claim 1, characterized in that: The method of determining the ultrasonic positioning information and the front image based on external detection of the blind smart glasses, and determining the direction deviation value according to the ultrasonic positioning information, the front image, and the features recorded by the stereo model in the primary direction, includes: The blind smart glasses perform external detection when moving, and trigger the ultrasonic positioning detection and image detection of the blind smart glasses; The moving area of the blind smart glasses is determined according to the current position of the blind smart glasses and the stereo model, and the priority of ultrasonic positioning detection and the priority of image detection are determined based on the shape of the moving area and the object features distributed in the moving area.

7. The method for direction recognition of smart glasses for the blind in an outdoor environment according to claim 6, characterized in that: The method of determining the ultrasonic positioning information and the front image based on the external detection of the blind smart glasses, and determining the direction deviation value according to the ultrasonic positioning information, the front image, and the features recorded by the three-dimensional model in the primary direction, further includes: If the priority of ultrasonic positioning detection is lower than that of image detection, the front image determined by image detection is collected first, and the first deviation range is determined based on the comparison between the front image and the features recorded in the primary direction of the stereo model. The second deviation range is determined based on the front image and the ultrasonic positioning information determined by ultrasonic positioning detection, and the direction deviation value is determined based on the first deviation range and the second deviation range.

8. The method for direction recognition of blind smart glasses in outdoor environments according to claim 1, characterized in that: Determining the final direction of the blind smart glasses in the outdoor environment according to the direction deviation value, the primary direction, and the moving direction of the blind smart glasses includes: The moving direction of the blind smart glasses is monitored in real time. The moving direction changes dynamically within a preset time, and the corresponding direction change amount is determined according to the dynamic change of the moving direction of the blind smart glasses.

9. The method for direction recognition of smart glasses for the blind in an outdoor environment according to claim 8, characterized in that: The method of determining the final direction of the blind smart glasses in the outdoor environment according to the direction deviation value, the primary direction, and the moving direction of the blind smart glasses further includes: Determine a first direction parameter according to the primary direction and the direction deviation value, and determine a second direction parameter according to the primary direction and the direction change; The moving area of the blind smart glasses relative to the three-dimensional model is collected, and a corresponding direction mapping relationship is matched according to the shape and position information of the moving area. The direction mapping relationship serves as a preset relationship of the blind smart glasses; the final direction of the blind smart glasses in the outdoor environment is determined according to the first direction parameter, the second direction parameter and the corresponding direction mapping relationship.

10. A direction recognition system for blind smart glasses in outdoor environments, characterized in that: The direction recognition system of the blind smart glasses in an outdoor environment is applied to the direction recognition method of the blind smart glasses in an outdoor environment as claimed in any one of claims 1 to 9. The direction recognition system of the blind smart glasses in an outdoor environment includes: An outdoor environment type module is used to determine the type of outdoor environment based on the surrounding images and ambient sounds collected by the blind smart glasses; the outdoor environment types include outdoor roads, outdoor parks, or outdoor sports fields; A stereo module, configured to construct a stereo model of the outdoor environment based on a distribution map of outdoor environment types and the surrounding image; a primary direction module, configured to determine, in the stereo coordinate system of the stereo model, a primary direction of the blind smart glasses in the outdoor environment according to the current position of the blind smart glasses and the current positions of the various features of the stereo model; a direction deviation value module, configured to determine ultrasonic positioning information and a front image based on external detection by the blind smart glasses, and determine a direction deviation value according to the ultrasonic positioning information, the front image, and features recorded by the stereo model in the primary direction; The final direction module is used to determine the final direction of the blind smart glasses in the outdoor environment according to the direction deviation value, the primary direction and the moving direction of the blind smart glasses.

Images