Method for improving blind person obstacle avoidance efficiency and safety based on multi-modal large language model

Abstract

The invention discloses a method for improving blind person obstacle avoidance efficiency and safety based on a multi-mode large language model, and the method comprises the following steps: 1, obtaining environment data, including object position, distance and shape information, through a plurality of sensors, and carrying out environment perception and data collection; step 2, preprocessing the data acquired by the sensors, including data cleaning and feature extraction, and then fusing the data of different sensors by using a multi-modal big language model; 3, based on the fused environment understanding model, using a large language model to generate language description adaptive to perception of the blind; and 4, transmitting the language description to the blind user in a voice prompt or tactile feedback mode. According to the method for improving the obstacle avoidance efficiency and safety of the blind person based on the multi-mode large language model, the large language model is adopted to generate language description adaptive to perception of the blind person, and the blind person can better receive navigation information.

Description

Technical Field

[0001] The present invention relates to a method for improving the efficiency and safety of blind people in avoiding obstacles, and more specifically to a method for improving the efficiency and safety of blind people in avoiding obstacles based on a multimodal large language model. Background Art

[0002] Visually impaired people face many challenges in their daily lives, one of which is navigating safely and avoiding obstacles. Traditional blind-assistance devices such as guide dogs or sensor systems provide help, but they have certain limitations, especially in complex environments or unknown terrains.

[0003] For this purpose, there is a patent in the prior art with the patent number CN202410885100.0, entitled "An intelligent navigation system and device for blind people's travel". It discloses a method of storing the blind people's travel data and being able to connect to a satellite navigation system. It provides the blind people with the conditions on the road through a sensing detection module, a voice and a text recognition module to achieve navigation for the blind people's travel. However, the above method can only achieve simple voice command recognition when communicating with the blind, and has certain limitations when complex intersections occur, especially in complex environments or unknown terrains. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model through data preprocessing and multimodal fusion, as well as real-time understanding of the large language model and giving feedback suggestions.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model, comprising the following steps:

[0006] Step 1: Use a variety of sensors to obtain environmental data, including information such as object location, distance, and shape, to perform environmental perception and data collection;

[0007] Step 2: Preprocess the data acquired by the sensor, including data cleaning and feature extraction, and then use the multimodal large language model to fuse different sensor data to generate a comprehensive environmental understanding model;

[0008] Step 3: Based on the fused environment understanding model, a large language model is used to generate a language description adapted to the perception of the blind.

[0009] Step 4: Convey the language description to the blind user through voice prompts or tactile feedback to help him understand the surrounding environment and make corresponding action adjustments, thereby improving obstacle avoidance efficiency and safety.

[0010] As a further improvement of the present invention, the method of extracting features from the data in step 2 is: using computer vision algorithms, such as edge detection, feature point extraction, and target detection algorithms, to extract useful visual features from camera images, such as the position, size, and shape of obstacles, and to extract distance and depth information from lidar or infrared sensors.

[0011] As a further improvement of the present invention, after the feature extraction of the data in step 2, there is also a step of obtaining a feature vector, specifically: obtaining the feature vector through the extraction method of principal component analysis, and the specific method of the principal component analysis extraction method is: by calculating the covariance matrix of the data, and then performing eigenvalue decomposition on it to obtain eigenvalues ​​and eigenvectors, and selecting the eigenvector corresponding to the largest eigenvalue as the principal component.

[0012] As a further improvement of the present invention, the covariance matrix is ​​a square matrix used to measure the covariance relationship between multiple random variables. Assume that there are n random variables X1, X2, ..., X n , the elements σ of their covariance matrix Σ ij It means X i and X j The covariance of the random vector covariance matrix Σ can be expressed as:

[0013]

[0014] Among them, Var(X i ) is X i The variance of Cov(X i , X j ) is X i and X j The covariance of .

[0015] As a further improvement of the present invention, the specific method of performing data fusion in step 2 is: using a Transformer model to process and integrate multimodal data.

[0016] As a further improvement of the present invention, the environment understanding model in step 2 includes:

[0017] Obstacle recognition module, which uses a machine learning model to identify obstacles in the environment, such as pedestrians, vehicles, or other objects, based on the fused data;

[0018] Path planning algorithm module, which combines Dijkstra algorithm and dynamic path planning algorithm to generate safe navigation instructions for the blind based on the identified obstacles and environmental features;

[0019] The safety consideration module communicates with the path planning algorithm module to determine whether the path planned by the path planning algorithm module is the shortest and whether obstacles are avoided to ensure the stability and safety of navigation.

[0020] As a further improvement of the present invention, the method of processing and integrating multimodal data in step 2 is: each data modality is subjected to a feature extraction process so as to convert it into a mathematical representation, and then the features from different data modalities are integrated into a unified feature space. Common methods include concatenation, splicing, and superposition.

[0021] As a further improvement of the present invention, the method of processing and integrating multimodal data in step 2 is: mapping data of different views into a unified high-dimensional space through a kernel function to facilitate subsequent integration and analysis. For multidimensional data, tensor decomposition is used to extract key features of each modality and integrate them. The tensor decomposition is specifically as follows:

[0022] For a three-dimensional tensor X with dimensions I×J×K, the goal of CP decomposition is to express it as the sum of the following forms: CP decomposition expression:

[0023]

[0024] Where: ar: a vector of length I, representing the components in the first mode, br: a vector of length J, representing the components in the second mode, cr: a vector of length K, representing the components in the third mode, The vector outer product operator is used to combine vector components on different modes to construct a rank-1 tensor. R: The rank of the decomposition determines the complexity and accuracy of the approximated tensor. The larger the rank, the more accurate the approximation, but the higher the computational complexity and storage requirements.

[0025] The goal of optimizing the CP decomposition is then to minimize the difference between the original tensor X and the reconstructed tensor by minimizing the following objective function:

[0026]

[0027] As a further improvement of the present invention, the method of processing and integrating multimodal data in step 2 is: using a deep learning model to simultaneously process multiple data modalities and learn the complex relationships and representations between different modalities.

[0028] As a further improvement of the present invention, the method of processing and integrating multimodal data in step 2 is: learning the potential relationship and common representation between multiple data modalities by jointly modeling them.

[0029] The beneficial effect of the present invention is that, through the settings of steps 1 to 4, the blind can understand the surrounding environment more accurately and in real time, avoid the risk of collision with obstacles, and improve their independence and safety in daily life. At the same time, the information generation capability based on the large language model makes the feedback information more specific and vivid, which helps the blind users to better perceive and respond to complex environmental changes. Compared with traditional sensor obstacle avoidance, the combination of the large model has the following advantages:

[0030] 1. Accuracy: Multimodal data fusion and large language model generation provide more accurate environmental perception and language description.

[0031] 2. Real-time: Feedback information is generated and communicated in real time, helping blind users to respond quickly.

[0032] 3. User-friendliness: The language description is intuitive, vivid, easy to understand and apply. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an overall flow chart of the method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0035] Reference Figure 1 As shown, a method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model in this embodiment specifically includes the following steps:

[0036] 1. Environmental perception and data collection: Use a variety of sensors (such as cameras, lidar, etc.) to obtain environmental data, including object location, distance, shape and other information.

[0037] 2. Preprocess and multimodal fusion of data based on the rapid transmission of 5G high-speed network: Preprocess the data obtained by the sensor, including data cleaning, feature extraction, etc., and then use the multimodal large language model to fuse different sensor data to generate a comprehensive environmental understanding model.

[0038] 3. Language model generation and feedback: Based on the fused environment understanding model, a large language model is used to generate language descriptions that adapt to the perception of the blind, such as "There is a trash can about 1 meter high in front of the right", "There is a traffic light in front, it is red now, and the countdown is 9 seconds", etc.

[0039] 4. Information feedback and navigation support: The language description is conveyed to blind users through voice prompts or tactile feedback to help them understand the surrounding environment and make corresponding action adjustments, thereby improving obstacle avoidance efficiency and safety.

[0040] The core technologies in the above steps are data preprocessing and multimodal fusion, as well as real-time understanding of large language models and providing feedback suggestions.

[0041] Among them, data preprocessing and multimodal fusion can be achieved through the following steps:

[0042] 1. Sensor data collection:

[0043] Cameras and LiDAR: Cameras provide visual information, while LiDAR provides distance and environment shape information. These sensors obtain data through different physical principles, such as cameras capturing optical images and LiDAR measuring distance through laser pulses.

[0044] 2. Feature extraction:

[0045] Visual feature extraction: Use computer vision algorithms, such as edge detection, feature point extraction, and object detection, to extract useful visual features from camera images, such as the location, size, and shape of obstacles.

[0046] Distance and depth features: Extracting distance and depth information from lidar or infrared sensors is crucial for determining the spatial relationship between obstacles and the blind person.

[0047] We can obtain eigenvectors through the extraction method of principal component analysis (PCA). PCA projects the original data into a new coordinate system through linear transformation. The direction selected by the new coordinate system is the direction with the largest data variance. These directions are called principal components. By retaining the principal component with the largest variance, the dimension of the data can be reduced.

[0048] By calculating the covariance matrix of the data, and then performing eigenvalue decomposition on it, we get the eigenvalues ​​and eigenvectors, and select the eigenvector corresponding to the largest eigenvalue as the principal component. The covariance matrix is ​​a square matrix used to measure the covariance relationship between multiple random variables. Suppose there are n random variables X1, X2,...,X n , the elements σ of their covariance matrix Σ ij It means X i and X j The covariance of the random vector covariance matrix Σ can be expressed as:

[0049]

[0050] Among them, Var(X i ) is X i The variance of Cov(X i , X j ) is X i and X jThe covariance of .

[0051] Therefore, we can obtain eigenvalues ​​and eigenvectors by performing eigenvalue decomposition on the covariance matrix.

[0052] 3. Data Fusion:

[0053] Multimodal data representation: Fusing feature information from different sensors to build a comprehensive representation of the environment. This can be achieved through classic fusion techniques such as feature-level fusion or model-level fusion.

[0054] Transformer model application: Use the Transformer model to process and integrate multimodal data. This model can effectively capture long-distance dependencies and complex interactions between features.

[0055] 4. Obstacle identification and path planning:

[0056] Obstacle recognition module: Based on the fused data, a machine learning model (such as a large language model) is used to identify obstacles in the environment, such as pedestrians, vehicles, or other objects.

[0057] Path planning algorithm module: Combines Dijkstra algorithm, dynamic path planning algorithm, etc. to generate safe navigation instructions for the blind based on the identified obstacles and environmental features.

[0058] Safety consideration module: The algorithm not only considers the shortest path, but also avoids obstacles and ensures the stability and safety of navigation.

[0059] Among them, multimodal data means that we can achieve it through the following aspects

[0060] Feature extraction and fusion:

[0061] Feature extraction: Each data modality needs to go through a specific feature extraction process in order to convert it into a mathematical representation (such as vector or tensor form).

[0062] Feature fusion: Integrate features from different data modalities into a unified feature space. Common methods include concatenation, splicing, and superposition.

[0063] Multi-view learning:

[0064] Kernel method: The kernel function is used to map data from different views into a unified high-dimensional space for subsequent integration and analysis.

[0065] Tensor decomposition: For multidimensional data (such as image sequences or time series data), tensor decomposition techniques can be used to extract the key features of each modality and integrate them. Tensor decomposition is the process of decomposing a high-order tensor into multiple low-order tensor products. We can process data through CP decomposition (Canonical Polyadicdecomposition):

[0066] For a three-dimensional tensor X (dimension is I×J×K), the goal of CP decomposition is to express it as the sum of the following forms: CP decomposition expression:

[0067]

[0068] in:

[0069] ar: A vector of length I, representing the components in the first mode.

[0070] br: A vector of length J, representing the components on the second mode.

[0071] cr: A vector of length K, representing the components on the third mode.

[0072] : A vector outer product operator that combines vector components on different modes to construct a rank-1 tensor.

[0073] R: The rank of the decomposition determines the complexity and accuracy of the approximated tensor. The larger the rank, the more accurate the approximation, but the higher the computational complexity and storage requirements.

[0074] At the same time, we can optimize the CP decomposition by minimizing the following objective function to minimize the difference between the original tensor X and the reconstructed tensor:

[0075]

[0076] Deep Learning Methods:

[0077] Multimodal neural networks: Use deep learning models (such as multi-input neural networks) to process multiple data modalities simultaneously and learn complex relationships and representations between different modalities.

[0078] Collaborative Learning:

[0079] Joint Modeling: Jointly modeling multiple data modalities to learn the latent relationships and common representations between them

[0080] In summary, the method of this embodiment based on a multimodal large language model to improve the efficiency and safety of blind people's obstacle avoidance utilizes the powerful semantic understanding and decision-making capabilities of the multimodal large language model to improve the efficiency and safety of blind people's obstacle avoidance.

[0081] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for improving the efficiency and safety of obstacle avoidance for the blind based on a multimodal large language model, characterized by: The steps include: Step 1: Use a variety of sensors to obtain environmental data, including information such as object location, distance, and shape, to perform environmental perception and data collection; Step 2: Preprocess the data acquired by the sensor, including data cleaning and feature extraction, and then use the multimodal large language model to fuse different sensor data to generate a comprehensive environmental understanding model; Step 3: Based on the fused environment understanding model, a large language model is used to generate a language description adapted to the perception of the blind. Step 4: Convey the language description to the blind user through voice prompts or tactile feedback to help him understand the surrounding environment and make corresponding action adjustments, thereby improving obstacle avoidance efficiency and safety.

2. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 1, characterized in that: The method of extracting features from the data in step 2 is: using computer vision algorithms, such as edge detection, feature point extraction, and target detection, to extract useful visual features from camera images, such as the location, size, and shape of obstacles, and to extract distance and depth information from lidar or infrared sensors.

3. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 2, characterized in that: After feature extraction of the data in step 2, there is also a step of obtaining feature vectors, specifically: obtaining feature vectors through the extraction method of principal component analysis. The specific method of the extraction method of principal component analysis is: calculating the covariance matrix of the data, and then performing eigenvalue decomposition on it to obtain eigenvalues ​​and eigenvectors, and selecting the eigenvector corresponding to the largest eigenvalue as the principal component.

4. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 3, characterized in that: The covariance matrix is ​​a square matrix used to measure the covariance relationship between multiple random variables. Suppose there are n random variables X1, X2,..., X n , the elements σ of their covariance matrix Σ ij It means X i and X j The covariance of the random vector covariance matrix Σ can be expressed as: Among them, Var(X i ) is X i The variance of Cov(X i , X j ) is X i and X j The covariance of .

5. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 4, characterized in that: The specific method of performing data fusion in step 2 is: using the Transformer model to process and integrate multimodal data.

6. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to any one of claims 1 to 5, characterized in that: The environmental understanding model in step 2 includes: Obstacle recognition module, which uses a machine learning model to identify obstacles in the environment, such as pedestrians, vehicles, or other objects, based on the fused data; Path planning algorithm module, which combines Dijkstra algorithm and dynamic path planning algorithm to generate safe navigation instructions for the blind based on the identified obstacles and environmental features; The safety consideration module communicates with the path planning algorithm module to determine whether the path planned by the path planning algorithm module is the shortest and whether obstacles are avoided to ensure the stability and safety of navigation.

7. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 5, characterized in that: The method of processing and integrating multimodal data in step 2 is: subjecting each data modality to a feature extraction process in order to convert it into a mathematical representation, and then integrating the features from different data modalities into a unified feature space. Common methods include concatenation, splicing, and superposition.

8. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 5, characterized in that: The method of processing and integrating multimodal data in step 2 is: mapping data from different views into a unified high-dimensional space through a kernel function to facilitate subsequent integration and analysis. For multidimensional data, tensor decomposition is used to extract the key features of each modality and integrate them. The tensor decomposition is specifically as follows: For a three-dimensional tensor X with dimensions I×J×K, the goal of CP decomposition is to express it as the sum of the following forms: CP decomposition expression: Where: ar: a vector of length I, representing the components in the first mode, br: a vector of length J, representing the components in the second mode, cr: a vector of length K, representing the components in the third mode, The vector outer product operator is used to combine vector components on different modes to construct a rank-1 tensor. R: The rank of the decomposition determines the complexity and accuracy of the approximated tensor. The larger the rank, the more accurate the approximation, but the higher the computational complexity and storage requirements. The goal of optimizing the CP decomposition is then to minimize the difference between the original tensor X and the reconstructed tensor by minimizing the following objective function:

9. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 5, characterized in that: The method of processing and integrating multimodal data in step 2 is to use a deep learning model to simultaneously process multiple data modalities and learn the complex relationships and representations between different modalities.

10. The method for improving the efficiency and safety of blind people's obstacle avoidance based on a multimodal large language model according to claim 5, characterized in that: The method of processing and integrating multimodal data in step 2 is to learn the potential relationships and common representations between multiple data modalities by jointly modeling them.

Images