A complex environment recognition and intelligent interaction system for blind navigation

Abstract

The application discloses a complex environment recognition and intelligent interaction system for blind navigation, aiming to provide a safer and more personalized travel assistance for the blind or visually impaired. Based on the travel route planned by the map software, the YOLOv8 target detection algorithm with added CBAM attention mechanism is combined with the visual large model to recognize the complex scene, and the recognition result is transmitted to the large language model. The large language model generates active warning text or answers user questions according to the recognition result, and feeds back to the user in the form of voice, realizing intelligent interaction. At the same time, relying on the average step length calculated by the monocular vision technology and the autocorrelation analysis algorithm, the broadcast unit is converted from "meters" to "steps", improving the personalization of voice broadcast. The application combines front-end and back-end information transmission, introduces complex environment recognition, voice interaction and personalized guidance technology, and significantly improves the safety, interactivity and user experience of blind navigation assistance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of blind navigation, in particular to a complex environment recognition and intelligent interaction system for blind navigation. BACKGROUND

[0002] The pedestrian navigation auxiliary device is an important facility in the navigation field, which can be widely used in traffic guidance and daily travel. Through the combination with the traffic facilities and walking conditions along the route, the pedestrian navigation auxiliary device also has a broad application prospect in planning pedestrian travel, saving pedestrian travel time and improving walking comfort. The current pedestrian navigation auxiliary device has poor compatibility for special groups. For the travel assistance needs of a large number of blind people, some companies have proposed some blind travel assistance schemes based on wearable devices and mobile terminals, but there are still some problems that have not been effectively solved. The method faces the travel needs of the blind group, explores the complex environment recognition and intelligent interaction method for blind navigation, and tries to solve the problems of the blind group such as difficult travel, poor direction perception, and inconvenience to seek help.

[0003] The existing blind travel assistance system is mainly divided into travel assistance devices and application programs based on mobile intelligent terminals: the travel assistance device is usually based on wearable hardware facilities, which has the problems of heavy weight, complex carrying, and not fully considering the user's own conditions during design, and generally has high price and limited audience group; the mobile intelligent terminal based on mobile intelligent terminal is some application programs loaded on the daily carried mobile intelligent terminal, which solves the problems of convenience and use cost, but has the defects and deficiencies of low intelligent degree, insufficient interaction, and difficulty in meeting the travel needs of the blind group.

[0004] At present, in the field of travel assistance methods for the disabled, especially the blind group, most mainstream scholars use image recognition, artificial intelligence and other fields to study feasible methods. However, there are few studies related to intelligent interaction and humanized guidance in blind travel assistance, and the domestic and foreign achievements obtained still have certain deficiencies, such as navigation broadcast limited to the "several meters" that ordinary people are used to, not "several steps", not fully tapping the status of existing blind path facilities, etc. Even the methods and models proposed by some scholars have certain competitiveness in the effect of blind travel assistance, but it is still difficult to meet the higher interaction requirements and higher humanized travel assistance needs. SUMMARY

[0005] The application aims to provide a complex environment recognition and intelligent interaction system for blind navigation, and an implementation strategy for blind travel auxiliary navigation capable of recognizing complex environments and providing intelligent interaction, which fully utilizes the advantages of front-end and back-end cooperation, integrates multiple algorithms to realize more accurate road condition and blind path information monitoring and more humanized voice broadcasting and interaction functions, and provides better travel auxiliary services for the visually impaired.

[0006] To achieve the above-mentioned purpose, the technical solution provided by the application is: a complex environment recognition and intelligent interaction system for blind navigation, comprising a client device and a background server, the client device is equipped with a loudspeaker, a microphone, a pedometer, an acceleration sensor and a fixed-position camera, and is internally provided with a target detection algorithm and a map navigation module with marker points; the background server is internally provided with a visual large model and a large language model, and has a computing function;

[0007] The specific implementation of the system comprises the following steps:

[0008] S1, the client device acquires image data of the surrounding environment in real time through the camera, including continuous multiple frames of images, processes the image data using the target detection algorithm to obtain object category and size information labels in the image data, and identifies the marker point position information labels in the positioning surrounding range of the client device through the map navigation module with marker points;

[0009] S2, the object category and size information labels and the marker point position information labels are transmitted to the large language model of the background server in a specified data format to generate corresponding instructions, the content of the instructions is transmitted back to the client device, and the object category, size and position information are output in the form of voice, and the user can select a dialogue with the client device to realize interaction;

[0010] S3, the pedometer and the acceleration sensor are called to recognize a stable step cycle, distance data of the user walking in the stable step cycle is collected, and the average step length of the user walking is calculated;

[0011] S4, the client device measures the distance between the object and the user through monocular vision technology, combines the average step length, and outputs the number of steps required to walk to the object in voice, wherein the number of steps is obtained by dividing the distance by the average step length.

[0012] Further, in step S1, the target detection algorithm comprises a YOLOv8 target detection algorithm with added CBAM attention mechanism and calling a visual large model API to assist in accurate image recognition.

[0013] The network structure of the YOLOv8 target detection algorithm includes three parts: Backbone, Neck and Head. The Backbone part is a feature extraction network that extracts useful information from the input image. It uses a fast implementation of the CSP bottleneck module (C2f module) to improve feature extraction capability through the bottleneck block and the fast spatial pyramid pooling layer (SPPF module). The Neck part is located between the Backbone and the Head. It uses a path aggregation network-feature aggregation network structure (PAN-FAN structure) to perform multi-scale feature fusion through the path aggregation network (PAN) and the feature aggregation network (FAN). The Head part is responsible for predicting the category of the target. It uses an anchor-free decoupled head structure to separate the classification and detection heads.

[0014] The YOLOv8 target detection algorithm adds a CBAM attention mechanism in the Neck part. For different scale input image features extracted by the CSP LayerDarknet backbone network, the CBAM attention mechanism is used to perform attention calculation on the channel and spatial dimensions of the features. It distinguishes between different channel features and captures the spatial structure of the image. The different scale image features processed by the CBAM attention mechanism are transmitted to the Neck part for feature fusion.

[0015] When the confidence of the result obtained by the target detection algorithm is lower than the set threshold, the visual large model API is called to transmit the image data obtained by the camera in real time to the visual large model for recognition, and the object category and size information labels are given.

[0016] Further, in step S1, when the client device locates the marked point in the surrounding range, the map navigation module of the client device generates the position information label of the marked point.

[0017] Further, in step S2, the large language model outputs the text content in the form of voice after summarizing the information labels, forming an interaction with the user. The background server is based on the NVIDIA NeMo framework. The specified format specifies the field information when the client device and the background server transmit data. The original binary data is Base64 encoded, then serialized to JSON format, and the hash value is calculated to ensure integrity. Then the whole data containing data and hash value is compressed, including the "image_data, image_info, audio_data, landmark_name" fields, where image_data, image_info, audio_data, and landmark_name represent image data, image information, audio information, and landmark information, respectively.

[0018] Further, in step S3, a self-correlation analysis algorithm is used to reduce the influence of conditions that can affect the acceleration change during walking on the step value calculation;

[0019] First, the three-axis acceleration α x 、a y 、α z of the client device is taken modulo to calculate the overall acceleration α, and the motion state of the user is determined according to the standard deviation σ of the overall acceleration:

[0020]

[0021]

[0022] In the formula, u is the mean of the overall acceleration sequence {α1, α2,..., α k ,...,α N} of the user's step cycle, α k represents the kth overall acceleration, and N represents the number of overall accelerations. The user state is determined to be idle or walking according to the value of the standard deviation of the overall acceleration. When the standard deviation of the overall acceleration is less than a set threshold, it can be determined that the user's activity amplitude at this moment is small, and thus the motion state at this moment is determined to be idle. Otherwise, it is determined to be a walking state.

[0023] Since the user walks continuously and rhythmically, the overall acceleration has a clear periodic change over time. This periodic change can be used to determine the correlation between the current step cycle overall acceleration and the previous step cycle. According to the correlation, the user's motion state is further determined. When the user is walking, the acceleration sensor will continuously record data. After calculating the overall acceleration, the self-correlation coefficient x(m, t) of the overall acceleration is calculated using the following formula, where m is the step cycle and t is the sampling period:

[0024]

[0025] In the formula, u(m, t) and σ(m, t) represent the mean and standard deviation of the acceleration sequence {a(p), a(p+1),..., a(p+1-t)} in the current step period m within the sampling period t; u(m+t, t) and σ(m+t, t) are the values of u(m, t) and σ(m, t) after one sampling period t, p is an auxiliary time parameter starting from 0 to help calculate the appropriate period length and avoid calculation omission, a(p) is the overall acceleration at time p, a(m+p) is the overall acceleration at time m+p, and a(m+p+t) is the overall acceleration at time m+p+t; when the sampling period t approaches the user's step period m, the walking states represented by a(m+p) and a(m+p+t) in the formula are considered to be consistent, so the value of x(m, t) approaches 1, but the step frequencies of different users or the same user at different times are inconsistent, so t is a variable, and the dynamic determination of t needs to be realized using a bracketing algorithm, i.e., the value range of t is selected as [t min , t max ], t min represents the minimum value of the sampling period t, and t max represents the maximum value of the sampling period t, and a new autocorrelation coefficient ρ(m, t) is calculated:

[0026]

[0027] When ρ(m, t) reaches the maximum value, the value of t is the stable step period of the user, and in this period, the ratio of the walking distance to the step counter step number is the average step length.

[0028] Further, in step S4, the monocular vision technology uses a triangular transformation method to measure the distance between the object and the user, and there are two cases as follows:

[0029] The first case is when the measured object is directly in front of the camera:

[0030] Through the inverse trigonometric function, the angle between the camera optical axis and the world coordinate system Y axis and the angle between the connecting line of the measured point and the image coordinate system origin and the camera optical axis can be obtained. The difference between the two angles is obtained, and the angle between the connecting line of the measured point and the image coordinate system origin and the world coordinate system Y axis is obtained. The tangent value of ∠Υ divided by the camera height H can obtain the distance to be solved in this case;

[0031] The second case is when the measured object is not directly in front of the camera:

[0032] The to-be-measured point is denoted as Q, and the projection point of Q in the image coordinate system is denoted as Q'. The distance between Q' and the camera is obtained according to the first case, and thus the geometric distance from the origin of the camera coordinate system to the projection of the to-be-measured point on the Y axis of the world coordinate system is obtained. Since the triangle formed by the origin of the camera coordinate system, the to-be-measured point and the projection of the to-be-measured point on the Y axis of the world coordinate system is similar to the projection of the triangle in the image coordinate system, the distance from the origin of the camera coordinate system to the projection of the to-be-measured point can be calculated, and the distance in the case is obtained according to the similarity relationship.

[0033] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0034] 1、The present application combines target detection algorithm with large language model, bringing multiple advantages to the system, realizing the fusion of text and visual information. The target objects recognized by the target detection algorithm are associated with the natural language texts generated by the large language model, providing more rich and accurate environmental perception and understanding. On this basis, the intelligent semantic understanding and generation capability of the large language model enables the system to generate detailed descriptions, instructions and suggestions according to the recognized target objects, improving the communication effect and the accuracy of information transmission. The self-evolution of deep learning enables the system to continuously optimize the algorithm through interaction and feedback with the user, providing more intelligent and personalized auxiliary functions, providing more rich, accurate and intelligent auxiliary functions for the user, improving the user experience and use effect, and bringing substantial improvement to the field of blind travel.

[0035] 2、The present application has a wide range of applications, not limited to indoor navigation, but also including outdoor navigation, travel exploration and daily life assistance, etc. In view of the needs of blind users, the present application system provides comprehensive navigation and environmental perception services, enabling them to better integrate into society and live independently: in outdoor navigation, the present application system uses target detection algorithm and large language model to recognize and describe various target objects in the outdoor environment. Through real-time target recognition and language generation, the system can provide detailed navigation route guidance and also provide descriptions of the surrounding environment.

[0036] 3、The present application addresses the problem of perception difficulty in existing navigation systems using meters as length units, and combines pedometer, acceleration sensor function and monocular vision technology to change the broadcast unit from "meters" to "steps", which is more in line with the usage habits of blind users, improving the humanization level of the system, thereby providing more professional and user-demand-oriented navigation experience for blind users, and enhancing the safety protection for users. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a flowchart of deploying the system in a blind navigation device and collecting data.

[0038] Figure 2 is the geometric diagram of the ranging mode when the measured object is in front of the camera.

[0039] Figure 3 is the geometric diagram of the ranging mode when the measured object is not in front of the camera.

[0040] Figure 4 is the schematic diagram of realizing step unit personalized broadcast. DETAILED DESCRIPTION

[0041] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0042] As Figure 1 shown, the embodiment discloses a complex environment recognition and intelligent interaction system for blind navigation, which comprises a client device and a background server, the client device is equipped with a loudspeaker, a microphone, a pedometer, an acceleration sensor and a fixed position camera, and is internally provided with a target detection algorithm and a map navigation module with marked points; the background server is internally provided with a visual large model and a large language model, and has a computing function;

[0043] The specific implementation of the system comprises the following steps:

[0044] S1, the client device acquires image data of the surrounding environment in real time through the camera, including continuous multiple frames of images, processes the image data using a target detection algorithm to obtain object category and size information labels in the image data; the client device identifies the location information labels of the marked points in the positioning surrounding range in real time through the map navigation module with marked points; the specific implementation is as follows:

[0045] The target detection algorithm comprises a YOLOv8 target detection algorithm with added CBAM attention mechanism and calling a visual large model API to assist in accurate image recognition;

[0046] The network structure of the YOLOv8 target detection algorithm includes three parts: Backbone, Neck and Head. The Backbone part is a feature extraction network that extracts useful information from the input image. It uses a fast implementation of the CSP bottleneck module (C2f module) to improve feature extraction capability through the bottleneck block and the fast spatial pyramid pooling layer (SPPF module). The Neck part is located between the Backbone and the Head. It uses the path aggregation network-feature aggregation network structure (PAN-FAN structure) to perform multi-scale feature fusion through the path aggregation network (PAN) and the feature aggregation network (FAN). The Head part is responsible for predicting the category of the target. It uses an anchor-free decoupled head structure to separate the classification and detection heads.

[0047] The YOLOv8 target detection algorithm adds a CBAM attention mechanism in the Neck part. For different scales of input image features extracted by the CSP LayerDarknet backbone network, the CBAM attention mechanism is used to perform attention calculation on the channel and spatial dimensions of the features. It distinguishes between different channel features and captures the spatial structure of the image. The different scale image features processed by the CBAM attention mechanism are transmitted to the Neck part for feature fusion.

[0048] When the confidence of the result obtained by the target detection algorithm is lower than the set threshold, the visual large model API is called to transmit the image data acquired by the camera in real time to the visual large model for recognition, and the object category and size information labels are given.

[0049] When the client device locates the marked point within the surrounding range, the map navigation module of the client device generates the position information label of the marked point.

[0050] S2, the object category and size information labels and the marked point position information label are transmitted to the large language model of the background server in a specified data format to generate corresponding instructions. The content of the instructions is transmitted back to the client device and output in the form of voice. The user can choose to interact with the client device. The specific process is as follows:

[0051] After the large language model aggregates information tags, it outputs text content in speech form, creating an interaction with the user. The backend server is built on the NVIDIA NeMo framework. The specified format defines the field information for data transmission between the client and the backend server. The raw binary data is Base64 encoded, then serialized into JSON format, and a hash value is calculated to ensure integrity. Finally, the entire dataset containing the data and hash value is compressed. The transmitted information includes "image_data, image_info, audio_data, landmark_name", representing "image data, image information, audio information, and landmark information," respectively. In this example, image_data represents the detected image data, a Base64 encoded string; image_info represents basic image information, including width, height, and image format; audio_data represents the collected audio information, a Base64 encoded string; and landmark_name represents the landmark information identified by the map navigation module.

[0052] One example of the transmission is shown in Table 1:

[0053] Table 1 shows some examples of transmitted information.

[0054]

[0055] In the second row of Table 1, the height of the transmitted image information is 1280 pixels and the width is 1616 pixels. The Base64 encoded string is / 9j / 4ShJRXhpZgAATU0AKgAAAAg…, and the Base64 encoded string of the transmitted voice information is "UklGRhQAAABXQV…". Since there are no markers in the vicinity of the client device's location, the landmark_name is displayed as "No". In the third row of Table 1, the height of the transmitted image information is 474 pixels and the width is 632 pixels. The Base64 encoded string is / 9j / 4AAQSkZJRgABAQEAAAAAAAD…, and the Base64 encoded string of the transmitted voice information is "UklGRiAAAABXQVZF…". There is a marker "Garbage Station" in the vicinity of the client device's location, and the landmark_name is displayed as "Garbage Station".

[0056] S3. The pedometer and accelerometer are used to identify stable stepping cycles, collect distance data of the user's steps during these stable cycles, and calculate the user's average stride length, as detailed below:

[0057] An autocorrelation analysis algorithm is used to reduce the impact of possible acceleration changes during the movement on the step size calculation.

[0058] First, the three-axis acceleration α of the client device x a y α z The modulus is taken, the overall acceleration α is calculated, and the user's motion state is determined based on the standard deviation σ of the overall acceleration.

[0059]

[0060]

[0061] In the formula, u is the overall acceleration sequence of the user's step cycle {α1, α2, ..., α...} k ,...,α N The mean of}, α k Let N represent the k-th overall acceleration and N represent the number of overall accelerations. The user's state is determined as idle or walking based on the standard deviation of the overall acceleration. When the standard deviation of the overall acceleration is less than a set threshold, it can be determined that the user's activity amplitude is small at this moment, and thus the motion state at this moment is determined to be idle; otherwise, it is walking. In this embodiment, it is believed that the probability of the pedestrian being idle is relatively high during the period when the standard deviation of the overall acceleration is less than 0.5, so the motion state at this moment is determined to be idle to reduce the possibility of false counting; otherwise, it is walking.

[0062] Because users walk continuously and rhythmically, their overall acceleration exhibits a significant periodic change over time. This periodic change can be used to calculate the autocorrelation between the overall acceleration of the current step cycle and that of the previous step cycle. The magnitude of the autocorrelation can then be used to further determine the user's motion state. While the user is walking, the accelerometer continuously records data. After calculating the overall acceleration, the autocorrelation coefficient x(m,t) of the overall acceleration is calculated using the following formula, where m is the step cycle and t is the sampling period:

[0063]

[0064] In the formula, u(m,t) and σ(m,t) represent the mean and standard deviation of the acceleration sequence {a(p), a(p+1), ..., a(p+1-t}) within a sampling period t under the current step period m, respectively; u(m+t,t) and σ(m+t,t) are the values ​​of u(m,t) and σ(m,t) after one sampling period t; p is used as an auxiliary time parameter, starting from 0 to help calculate a suitable period length and avoid calculation omissions; a(p) is the overall acceleration at time p. a(m+p) is the overall acceleration at time m+p, and a(m+p+t) is the overall acceleration at time m+p+t. When the sampling period t is close to the user's step period m, the walking states represented by a(m+p) and a(m+p+t) in the formula can be considered consistent, so the value of x(m,t) is close to 1. However, the step frequency of different users or the same user at different times is inconsistent, so t is a variable. A bounding algorithm is needed to dynamically determine t, that is, to select the range of values ​​of t as [t...]. min , t max ], t min The minimum value of t represents the sampling period t. max Let t represent the maximum value of the sampling period t, and calculate the new autocorrelation coefficient ρ(m,t):

[0065]

[0066] The value of t when ρ(m,t) reaches its maximum value is the stable stepping cycle of the user. Under this cycle, the ratio of walking distance to the number of steps counted by the pedometer is the average step length.

[0067] S4. The client device uses monocular vision technology to calculate the distance between the object and the user. Combined with the average step length, it outputs the number of steps required to reach the object by voice. The number of steps is obtained by dividing the distance by the average step length.

[0068] The monocular vision technology uses triangulation to calculate the distance between an object and the user, and there are two scenarios:

[0069] The first scenario is when the object being measured is directly in front of the camera:

[0070] like Figure 2 As shown, let the camera height be H, O c The distance to Q' is f, and the component of f on the Y-axis of the world coordinate system is f. y Based on the tangent relationship, the formulas for calculating the angle ∠α between the camera's optical axis and the Y-axis of the world coordinate system, and the angle ∠β between the line connecting the point to be measured and the origin of the image coordinate system and the camera's optical axis can be obtained:

[0071]

[0072] Where O represents the origin of the image coordinate system, Oc O represents the origin of the camera coordinate system. w M is the origin of the world coordinate system, M is the intersection of the camera optical axis and the Y-axis of the world coordinate system, P is the measurement point on the ground plane, and O is the origin of the world coordinate system. w P represents the required vertical distance.

[0073] If point P has coordinates (u0, v0), and its projection point P' in the image coordinate system has coordinates (u, v), then OP' = v - v0. From this, we can derive the formula for calculating the vertical distance from point P:

[0074]

[0075] ∠Υ is obtained by subtracting ∠α and ∠β.

[0076] The second scenario is when the object being measured is not directly in front of the camera:

[0077] like Figure 3 As shown, the vertical distance O of the known target point is... w P and ∠γ, from which we can obtain O c P:

[0078]

[0079]

[0080] Where Q represents the distance measurement point in the world coordinate system, and Q' is its projection point in the image coordinate system.

[0081]

[0082] P′Q′=(u-u0)×dy

[0083] Therefore, the horizontal distance PQ to the target point can be calculated using the following formula:

[0084]

[0085] like Figure 4 As shown, this allows for distance measurement between the location of a blind person and the target point.

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A complex environment recognition and intelligent interaction system for navigation for the blind, characterized in that, It includes a client device and a backend server. The client device is equipped with a speaker, microphone, pedometer, accelerometer and fixed-position camera, and has a built-in target detection algorithm and map navigation module with marker points. The backend server has a built-in large visual model and a large language model, and is equipped with computing capabilities; The specific implementation of this system includes the following steps: S1. The client device acquires image data of the surrounding environment in real time through the camera, including continuous multi-frame images. It processes the image data using an object detection algorithm to obtain object category and size information labels in the image data. The client device identifies the location information labels of the marker points in the surrounding area of ​​its location in real time through a map navigation module with marker points. S2. The object category and size information labels and the marker point location information labels are transmitted to the backend server in a specified data format to generate corresponding instructions from the large language model. After the content of the instructions is transmitted back to the client device, the object category, size and location information are output in voice form. The user can choose to talk to the client device to achieve interaction. S3. Use the pedometer and accelerometer to identify stable stepping cycles, collect distance data of the user walking under stable stepping cycles, and calculate the average step length of the user. S4. The client device uses monocular vision technology to calculate the distance between the object and the user. Combined with the average step length, it outputs the number of steps required to reach the object via voice. The number of steps is obtained by dividing the distance by the average step length.

2. The complex environment recognition and intelligent interaction system for navigation for the blind according to claim 1, characterized in that, In step S1, the target detection algorithm includes using the YOLOv8 target detection algorithm with added CBAM attention mechanism and calling the Visual Large Model API to assist in accurate image recognition; The YOLOv8 object detection algorithm's network structure comprises three parts: Backbone, Neck, and Head. The Backbone is a feature extraction network that extracts useful information from the input image. It employs a fast-implemented CSP bottleneck module, enhancing feature extraction capabilities through a bottleneck block and fast spatial pyramid pooling layers. The Neck, located between the Backbone and Head, uses a path aggregation network-feature aggregation network structure, performing multi-scale feature fusion through these two networks. The Head is responsible for predicting the target's category and employs a decoupled head structure without anchor boxes, separating the classification and detection heads. The YOLOv8 object detection algorithm adds a CBAM attention mechanism to the Neck part. For input image features of different scales extracted by its CSP LayerDarknet backbone network, the CBAM attention mechanism is used to perform channel and spatial dimension attention calculation on the features, distinguish the features between different channels and capture the spatial structure in the image. The image features of different scales processed by the CBAM attention mechanism are then passed to the Neck part for feature fusion. When the confidence level of the result obtained by the object detection algorithm is lower than the set threshold, the visual big model API is called to transmit the image data acquired by the camera in real time to the visual big model for recognition, and to provide the object category and size information label.

3. The complex environment recognition and intelligent interaction system for navigation for the blind according to claim 1, characterized in that, In step S1, when a marker appears in the surrounding area of ​​the client device, the client device's map navigation module will generate a location information label for the marker.

4. The complex environment recognition and intelligent interaction system for navigation for the blind according to claim 1, characterized in that, In step S2, the large language model summarizes the information tags and outputs the text content in speech form to form an interaction with the user. The backend server is mounted on the NVIDIA NeMo framework. The specified format defines the field information when the client device and the backend server transmit data. The original binary data is Base64 encoded, and then the data is serialized into JSON format. The hash value is calculated to ensure integrity. Then, the whole containing the data and hash value is compressed, including the fields "image_data, image_info, audio_data, landmark_name", where image_data, image_info, audio_data, and landmark_name represent image data, image information, audio information, and landmark information, respectively.

5. A complex environment recognition and intelligent interaction system for navigation for the blind according to claim 1, characterized in that, In step S3, an autocorrelation analysis algorithm is used to reduce the impact of possible acceleration changes during the movement on the step size calculation. First, the three-axis acceleration α of the client device x a y α z The modulus is taken, the overall acceleration α is calculated, and the user's motion state is determined based on the standard deviation σ of the overall acceleration. In the formula, u is the overall acceleration sequence of the user's step cycle {α1, α2, ..., α...} k ,...,α N The mean of}, α k The value represents the kth overall acceleration, and N represents the number of overall accelerations. The user's state is determined as idle or walking based on the standard deviation of the overall acceleration. When the standard deviation of the overall acceleration is less than the set threshold, it can be determined that the user's activity amplitude is small at this moment, and thus the movement state at this moment is determined as idle; otherwise, it is walking. Because users walk continuously and rhythmically, their overall acceleration exhibits a significant periodic change over time. This periodic change can be used to calculate the autocorrelation between the overall acceleration of the current step cycle and that of the previous step cycle. The magnitude of the autocorrelation can then be used to further determine the user's motion state. While the user is walking, the accelerometer continuously records data. After calculating the overall acceleration, the autocorrelation coefficient x(m,t) of the overall acceleration is calculated using the following formula, where m is the step cycle and t is the sampling period: In the formula, u(m,t) and σ(m,t) represent the mean and standard deviation of the acceleration sequence {a(p), a(p+1), ..., a(p+1-t}) within a sampling period t under the current step period m, respectively; u(m+t,t) and σ(m+t,t) are the values ​​of u(m,t) and σ(m,t) after one sampling period t; p is used as an auxiliary time parameter, starting from 0 to help calculate a suitable period length and avoid calculation omissions; a(p) is the overall acceleration at time p; a(m+p) is the overall acceleration at time m+p; and a(m+p+t) is the overall acceleration at time m+p+t. When the sampling period t is close to the user's step period m, the walking states represented by a(m+p) and a(m+p+t) in the formula can be considered consistent, so the value of x(m,t) is close to 1. However, the step frequency of different users or the same user at different times is inconsistent, so t is a variable. A bounding algorithm is needed to dynamically determine t, that is, to select the range of values ​​of t as [t min , t max ], t min The minimum value of t represents the sampling period t. max Let t represent the maximum value of the sampling period t, and calculate the new autocorrelation coefficient ρ(m,t): The value of t when ρ(m,t) reaches its maximum value is the stable stepping cycle of the user. Under this cycle, the ratio of walking distance to the number of steps counted by the pedometer is the average step length.

6. A complex environment recognition and intelligent interaction system for navigation for the blind according to claim 1, characterized in that, In step S4, the monocular vision technology uses triangulation to calculate the distance between the object and the user, and there are two cases: The first scenario is when the object being measured is directly in front of the camera: By using inverse trigonometric functions, we can obtain the angle between the camera optical axis and the Y-axis of the world coordinate system, as well as the angle between the line connecting the point to be measured and the origin of the image coordinate system and the camera optical axis. By subtracting these two angles, we can obtain the angle ∠Υ between the line connecting the point to be measured and the origin of the image coordinate system and the Y-axis of the world coordinate system. By dividing the camera height H by the tangent of ∠Υ, we can obtain the distance required in this case. The second scenario is when the object being measured is not directly in front of the camera: Let the point to be measured be denoted as Q, and its projection point in the image coordinate system be denoted as Q'. From the first case, we obtain the distance between Q' and the camera, and thus the geometric distance from the origin of the camera coordinate system to the projection of the point to be measured onto the Y-axis of the world coordinate system. Since the triangle formed by the origin of the camera coordinate system, the point to be measured, and the projection of the point to be measured onto the Y-axis of the world coordinate system is similar to the projection of this triangle in the image coordinate system, we can calculate the required distance in this case by measuring the distance from the origin of the camera coordinate system to the projection of the point to be measured, and combining the above similarity relationship.

Images