Blind person auxiliary interaction method and system based on multi-modal large model and storage medium

Abstract

The invention discloses a blind person auxiliary interaction method and system based on a multi-modal large model and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a user voice instruction and an external environment image, and carrying out the preprocessing of the obtained data; inputting the preprocessed data into a pre-trained multi-modal large model to obtain an environment understanding result; integrating the environment understanding result into feedback information for voice output, and adjusting output intonations according to environment characteristics; the acquisition method of the pre-trained multi-modal large model comprises the following steps: acquiring historical multi-modal data; and inputting the historical multi-modal data into a pre-constructed multi-modal large model, carrying out fine tuning on the trained multi-modal large model through a base layer in a frozen model to obtain a trained multi-modal large model, fusing multi-modal information through the multi-modal large model, accurately identifying a user intention and a corresponding object in a complex environment, and carrying out real-time identification on the user intention and the corresponding object in the complex environment. Therefore, accurate scene description is fed back.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a blind person assisted interaction method, system and storage medium based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence technology, significant progress has been made in areas such as image recognition, speech recognition, and natural language processing. The development of these technologies has promoted the innovation of multimodal interaction systems, enabling machines to better understand and respond to user needs. In this technological context, it is particularly important to develop software that can capture user voice and images in real time and analyze and respond through artificial intelligence. This software can be applied to a variety of scenarios, such as disability assistance.

[0003] Among the existing technical solutions for disability assistance through artificial intelligence, there is a head-mounted laser scanning obstacle avoidance device for blind people with application number 202410405329X, which scans the environment in real time through laser radar and visual scanning modules to provide obstacle avoidance reminders. Users obtain information through headphones and vibration feedback during walking, but they mainly rely on vibration feedback, lack voice prompts or other rich interaction methods, and cannot provide functions such as navigation and scene understanding; another example is the intelligent communication perception method for assisting the blind with application number 202311507254.8, which uses intelligent communication technology, combines environmental perception and user needs, and provides customized assistance services. The system can recognize the surrounding environment and interact with users through smart devices, mainly relying on vibration and simple touch interaction, which may cause users to respond inflexibly in complex situations, and mainly rely on sensors for perception, lacking in-depth understanding of the scene, and unable to identify dynamic obstacles or understand user intentions; there are also guide glasses and interaction methods with application number 202410698521.2, which rely on multiple sensors. In a busy environment, the delay in signal processing may affect the user's timely response and increase the risk of collision.

[0004] Based on the defects of the existing technology, there is an urgent need to provide a solution that can accurately understand the intention of user instructions and provide timely voice feedback and prompts according to dynamic scenarios. Summary of the invention

[0005] The present invention provides a blind-assisted interaction method, system and storage medium based on a multimodal large model. By freezing the base layer during the training process of the multimodal large model, the multimodal large model can better adapt to complex scenes, thereby being able to obtain fast and accurate scene understanding results based on user instructions and the external environment, and then dynamically adjusting the tone of the output voice feedback based on the feedback content, making the interaction process more natural and improving the user experience.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides a blind person assisted interaction method based on a multimodal large model, comprising:

[0008] Obtain user voice commands and external environment images and pre-process the acquired data;

[0009] Input the pre-processed user voice commands and external environment images into the pre-trained multimodal large model to obtain the environmental understanding results;

[0010] Integrating the environmental understanding results into voice feedback information, and adjusting the tone of the voice feedback information according to environmental characteristics;

[0011] Outputting voice feedback information after intonation adjustment to the user;

[0012] The method for obtaining the pre-trained multimodal large model includes:

[0013] Acquire historical multimodal data;

[0014] The historical multimodal data is input into a pre-built multimodal large model, and the multimodal large model in training is fine-tuned by freezing the base layer in the model to obtain a trained multimodal large model.

[0015] Optionally, obtaining user requirements and external environment images and preprocessing the obtained data includes:

[0016] The microphone collects the user's voice commands and the RGB camera collects image information of the user's surrounding environment;

[0017] Standardize and denoise the image information of the surrounding environment to obtain effective image data;

[0018] The voice instructions are converted into text instructions using a speech recognition model.

[0019] Optionally, the pre-built multimodal large model includes an encoding layer, a cross-modal attention layer, and a decoding layer connected in sequence;

[0020] The coding layer includes a parallel image encoder and a text encoder;

[0021] The cross-modal attention layer includes multiple groups of self-attention layers and feed-forward layers connected in sequence;

[0022] The pre-processed user voice commands and external environment images are input into the pre-trained multimodal large model to obtain the environmental understanding results, including:

[0023] Performing image feature extraction on input valid image data through an image encoder, and performing text feature extraction on input text instructions through a text encoder;

[0024] Obtaining scene understanding results according to the image feature vector and the text feature vector through multiple sets of self-attention layers and feed-forward layers;

[0025] The scene understanding result is output through the decoding layer.

[0026] Optionally, the extracting image features from the input valid image data by using an image encoder, and extracting text features from the input text instruction by using a text encoder, comprises:

[0027] The image encoder converts the valid image data into an image feature vector through a deep convolutional neural network, and the calculation formula is:

[0028] ,

[0029] in, represents the convolution operation, is the convolution kernel, is the valid image data, is the bias term, is the image feature vector;

[0030] The text encoder obtains the attention distribution of the word vector in the text instruction through the self-attention layer to obtain the text feature vector, and the calculation formula is:

[0031] ,

[0032] ,

[0033] in, is the text feature vector, is the word vector in the text instruction, , , are the weight matrices, , , The word vectors are After linear transformation, the matrix For the matrix The transpose of For the matrix Dimension.

[0034] Optionally, obtaining a scene understanding result according to the image feature vector and the text feature vector through multiple groups of self-attention layers and feed-forward layers includes:

[0035] The self-attention layer calculates the correlation between the text feature data and the image feature data to obtain the scene understanding result after the fusion of text and image. The calculation formula is as follows:

[0036] ,

[0037] in, is the fused scene understanding result, is the image feature vector, is the text feature vector, is the cross-modal attention mechanism, is the image feature The transpose of is the image feature Dimensions;

[0038] The feed-forward layer performs linear transformation on the scene understanding result and then outputs it.

[0039] Optionally, integrating the environmental understanding result into voice feedback information and adjusting the intonation of the voice feedback information according to environmental characteristics includes:

[0040] Generate text feedback information according to the scene understanding result;

[0041] Determining scene characteristics according to the scene understanding result, wherein the scene characteristics include scene complexity and urgency;

[0042] The text feedback information is converted into voice feedback information of corresponding intonation according to the scene feature.

[0043] Optionally, judging scene characteristics according to the scene understanding result includes:

[0044] Extracting key elements from the scene understanding result according to the target detection algorithm, wherein the key elements include dynamic objects and static objects;

[0045] The scenario complexity is evaluated based on the key factors, and the evaluation formula is as follows:

[0046] ,

[0047] in, is the complexity score of the scene, It is The size of an object, It is The distance between the object and the user, It is a nonlinear function that takes into account the number, distance and scale of objects;

[0048] According to the scene complexity and the extracted dynamic objects, the urgency of the scene is calculated, and the calculation formula is as follows:

[0049] ,

[0050] in is the urgency score, It is an object speed, It is an object Distance from the user;

[0051] The scenario characteristics are obtained based on the complexity and urgency of the scenario, and the calculation formula is as follows:

[0052] ,

[0053] in, is the scene feature, α and β are the weight parameters of complexity and urgency, respectively.

[0054] Optionally, converting the text feedback information into voice feedback information of corresponding intonation according to the scene feature includes:

[0055] The text feedback information is converted into voice feedback information through a voice generation model, and the voice generation model is fine-tuned according to the characteristics of the scene, and the voice feedback information is output according to the corresponding intonation. The voice feedback output formula is:

[0056] ,

[0057] in, It is voice feedback information. is a speech generation model, It is text feedback information.

[0058] In a second aspect, the present invention provides a blind person assisted interaction system based on a multimodal large model, comprising:

[0059] User demand and external environment acquisition module: used to obtain user voice commands and external environment images and pre-process the acquired data;

[0060] Environmental understanding result acquisition module: used to input the pre-processed user voice commands and external environment images into the pre-trained multimodal large model to obtain environmental understanding results;

[0061] Feedback generation module: used to integrate the environmental understanding results into voice feedback information, and adjust the tone of the voice feedback information according to environmental characteristics;

[0062] Feedback output module: used to output the voice feedback information after intonation adjustment to the user;

[0063] The method for obtaining the pre-trained multimodal large model includes:

[0064] Acquire historical multimodal data;

[0065] The historical multimodal data is input into a pre-built multimodal large model, and the multimodal large model in training is fine-tuned by freezing the base layer in the model to obtain a trained multimodal large model.

[0066] In a third aspect, the present invention provides a computer storage medium having a computer program stored thereon. When the computer program is executed by a processor, the blind person assisted interaction method based on a multimodal large model as described in any step of the first aspect is implemented.

[0067] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention combines visual and auditory perception modes to enable the multimodal large model to deeply fuse image and voice data, thereby being able to comprehensively understand and analyze the environment and achieve accurate scene perception. In the training process of the multimodal large model, the step of freezing the base layer enables the large model to better adapt to complex scenes, thereby accurately identifying obstacles, pedestrians and user intentions in complex environments, and providing accurate scene descriptions or navigation accordingly; combined with speech generation technology, richer and more natural voice interaction is achieved, and the emotion, tone, volume, etc. of the voice can be dynamically adjusted according to different scenes, making the interaction process more humane; and the present invention relies on a single RGB camera and microphone to achieve complex environmental perception functions, greatly simplifying hardware requirements and improving the portability of users during travel; the present invention can run independently through an embedded multimodal large model without relying on external devices or cloud services, ensuring the real-time and reliability of the system, and can maintain normal operation even without a network, greatly improving the user experience in various environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The figure is a flow chart of a blind person assisted interaction method based on a multimodal large model in one embodiment of the present invention;

[0069] Figure 2 Shown is a flow chart of a blind person assisted interaction method based on a multimodal large model in another embodiment of the present invention;

[0070] Figure 3 Shown is a schematic diagram of the structure of a multi-modal large model in an embodiment of the present invention;

[0071] Figure 4 Shown is a schematic structural diagram of a blind person auxiliary interactive entity system in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0073] Example 1

[0074] like Figure 1 As shown, this embodiment provides a blind person assisted interaction method based on a multimodal large model, including:

[0075] Obtain user voice commands and external environment images and pre-process the acquired data;

[0076] Input the pre-processed user voice commands and external environment images into the pre-trained multimodal large model to obtain the environmental understanding results;

[0077] Integrating the environmental understanding results into voice feedback information, and adjusting the tone of the voice feedback information according to environmental characteristics;

[0078] Outputting voice feedback information after intonation adjustment to the user;

[0079] The method for obtaining the pre-trained multimodal large model includes:

[0080] Acquire historical multimodal data;

[0081] The historical multimodal data is input into a pre-built multimodal large model, and the multimodal large model in training is fine-tuned by freezing the base layer in the model to obtain a trained multimodal large model.

[0082] Freezing the base layer during the training process of the multimodal large model can make the large model better adapt to complex scenarios, so that it can obtain accurate scene understanding results corresponding to the instructions based on the collected user instructions and external environment images, and judge whether the current scene is urgent based on the scene understanding results, and reflect the urgency in the tone of voice feedback, making the interaction process more natural and improving the user experience.

[0083] Example 2

[0084] Based on Example 1, this example also makes the following design.

[0085] like Figure 2 As shown, the blind person assisted interaction method based on the multimodal large model is divided into the following steps:

[0086] Step 1: User command collection and environmental perception

[0087] Start the RGB camera: When the system starts, the camera captures the environment around the user in real time. The image is sampled at a fixed frequency (such as 30 frames per second) to ensure that the image information obtained by the system is clear and real-time.

[0088] Microphone captures voice commands: The microphone is used to capture the user's voice input. The voice input is processed by the voice recognition model and converted into text data. This process is achieved through a deep learning voice recognition model to ensure accurate capture of the user's commands.

[0089] Step 2: Data processing

[0090] 1. Data preprocessing:

[0091] Image preprocessing: Standardize the images captured by the RGB camera and perform image enhancement or noise reduction on the image data to ensure that the quality of the input image is suitable for further analysis by the large model.

[0092] Speech preprocessing: The collected speech signal is converted into corresponding text through the speech recognition model. The formula is as follows:

[0093]

[0094] in, is the speech recognition function, which converts the speech command Convert to text instructions .

[0095] Speech Codec The speech signal is transformed through a series of convolutional or recurrent neural network (RNN / LSTM) layers Convert to text features.

[0096] Short-time Fourier transform (STFT): The speech signal is first extracted through STFT to extract spectral features:

[0097]

[0098] in, is the spectral characteristic of the speech signal, is a voice signal, is a window function, It's the frequency.

[0099] Speech to text conversion: These spectral features are then processed using a deep neural network to obtain the feature vector of the speech:

[0100]

[0101] in, is a text representation, It is a spectral feature, which is processed by RNN to generate text features.

[0102] 2. Multimodal large model processes preprocessed data

[0103] like Figure 3 As shown, the multimodal large model includes an encoding layer, a cross-modal attention layer, and a decoding layer connected in sequence. The encoding layer includes a parallel image encoder and a text encoder; the cross-modal attention layer includes a multi-layer overlapping self-attention layer and a feed-forward layer. There is a residual connection between the attention layer and the feed-forward layer to prevent the gradient from disappearing and accelerate training. There is a normalization layer after the output of each layer to make the model more stable and avoid gradient explosion. The self-attention layer is used to establish semantic associations between image and text vectors, understand user intent and elements in the current scene, and the feed-forward layer is used to further process the results of the cross-modal attention layer to produce the final environmental understanding results, including object recognition, user intent analysis, etc.

[0104] (1) Image encoding: The input of the multimodal large model first passes through the image encoder to convert the original image into a low-dimensional vector representation. These vectors can capture the core information in the image, such as the category, location, color, etc. of the object.

[0105] The image is converted into a feature vector through the encoding module , which can be achieved through the visual encoder, the formula of the process is as follows:

[0106]

[0107] in, is the image data input to the multimodal large model, is the function of the image encoder, outputting the image feature vector .

[0108] Image Encoder Through a multi-layer deep convolutional neural network, the object and environment information in the image is extracted and the input image is transformed into Converted into a feature vector that can be used by the fusion model .

[0109] For a given convolutional layer, the features of the input image can be obtained by convolution kernel Perform a convolution operation:

[0110]

[0111] in, represents the convolution operation, is the convolution kernel, is the image data input to the multimodal large model, is the bias term, the output is the image feature.

[0112] (2) Text encoding: The user’s text instructions are processed by a text encoder, which converts the text into a semantic vector that can be matched with the image vector.

[0113] The text is converted into a feature vector through the encoding module , which can be achieved through the visual encoder, the formula of the process is as follows:

[0114]

[0115] in, is the text data input into the multimodal large model, is the function of the text encoder, outputting the text feature vector .

[0116] Text Encoder The attention distribution of the word vector in the text instruction is obtained through the self-attention layer to obtain the text feature vector. The calculation formula is:

[0117]

[0118]

[0119] in, is the text feature vector, is the word vector in the text instruction, , , are the weight matrices, , , The word vectors are After linear transformation, the matrix For the matrix The transpose of For the matrix Dimension.

[0120] (3) Multimodal fusion: Within the large model, the outputs of the image and text encoders are fused together to achieve deep understanding of multimodal information through the cross-modal attention layer. This layer can capture the relationship between images and text. For example, when a user asks "What's in front?", the model will combine the object information in the image and the user's instructions to identify the type of object and generate relevant feedback.

[0121] The cross-modal attention layer takes image features and text features Interaction is achieved through a multimodal fusion mechanism. This mechanism provides a unified semantic understanding of visual and speech information, and the formula is:

[0122]

[0123] in, is the image feature vector, is the text feature vector, is the fused scene understanding result. The core task of the cross-modal attention mechanism is to integrate image features With text features Interact and calculate the correlation between them. The following is the specific calculation.

[0124] Calculate the attention weights:

[0125] Query Vector and key vector The dot product between them calculates the attention score and performs Processing to get weights:

[0126]

[0127] in is the query vector Query, which is the text feature in this case , is the key vector Key, which is the image feature here , Transpose it, is the dimension of the key vector, used for scaling to ensure numerical stability.

[0128] Weighted sum:

[0129] The attention weights and the value vector Multiply them together to get the final output:

[0130]

[0131] in is the value vector (Value), which is usually the same as the key vector. Here it is also the image feature ,Right now:

[0132]

[0133] Attention weights are used to weight the value vector , and then add the weighted value vectors to get the output. This allows the system to find relevant image features based on the input query, thereby generating targeted results. For example, in this method, the user may ask "What's ahead?" The model will find the relevant image features based on the text information. In visual input Find relevant image information in .

[0134] (4) Obstacle and scene recognition: The decoding layer outputs interactive information on the scene understanding results obtained by the cross-modal attention layer.

[0135] In this embodiment, some fine-tuning steps are added during the training process of the multimodal large model, as follows:

[0136] The large model is first pre-trained on a large-scale multimodal dataset to learn common image features and text understanding capabilities. The underlying network of this part of the model is already very powerful and suitable for a wide range of visual-language tasks.

[0137] Fine-tuning phase:

[0138] Freeze some layers: During fine-tuning, the low-level feature extraction network of the pre-trained model is frozen, such as the convolutional layer of the image, the embedding layer of the text encoding, etc. The weights of these layers are no longer updated, and the high-level parts of the model (such as the cross-modal attention mechanism, task-related decoding layers, etc.) can be trained to adapt to specific task scenarios.

[0139] Adjust the learning rate: Since the model has been pre-trained, use a smaller learning rate when fine-tuning, such as , to prevent over-updating of model parameters and ensure more stable and accurate performance for specific tasks.

[0140] Fine-tuning formula: ,in, are the current model parameters, is the loss function, For real scene data, is the gradient of the model.

[0141] If the basic perception ability of the large model changes, it will destroy the general features pre-trained in the previous stage, requiring more computing power and memory, and the training process will slow down. However, the fine-tuning step of the multimodal large model will keep the basic perception ability of the large model unchanged, so that the model's basic perception ability in processing images and texts (such as object recognition and text understanding) will not be affected by fine-tuning, avoiding the model from relearning general features and maintaining the original performance; secondly, it can enhance the customization ability of the model. Since only high-level features are fine-tuned, the model can better adapt to specific scenarios. For example, in complex scenarios, the generated feedback is more adapted to user needs, and the interaction is more natural and effective.

[0142] The fine-tuned model can retain the general feature processing capabilities while having higher scene adaptability. For example, in our blind guidance task, after freezing the low-level visual and language features, the model can generate more accurate interactive voice guidance through high-level scene perception and dynamic feedback.

[0143] Step 3: Feedback and Interaction

[0144] 1. Generate intelligent feedback: Based on the analysis results of the data processing module, the system will generate corresponding text feedback. For example, when an obstacle is identified ahead, the system generates feedback information: "There is an obstacle ahead, please detour."

[0145] 2. Speech synthesis: The generated text feedback is converted into speech information through the speech synthesis model. The speech synthesis model supports the generation of different emotions and intonations, thereby improving the naturalness of the user experience.

[0146] 3. Voice playback: Voice feedback is played to the user through the audio output module.

[0147] Step 4: Dynamic Adjustment

[0148] Emotional feedback adjustment: Based on the urgency or importance of the scenario, the system will affect the voice feedback by dynamically adjusting the model's output. For example, in complex or dangerous scenarios, the system may increase the tone, speed, and volume of the feedback so that the user can quickly understand the current danger or action recommendations.

[0149] Scene recognition formula:

[0150]

[0151] in It is the scene context feature, through the image feature and text input Figure out the urgency or complexity of the scenario.

[0152] The identification process is divided into the following steps:

[0153] 1. Extraction of key scene elements

[0154] Image features It is further processed to identify key elements in the scene, such as obstacles, pedestrians, vehicles and other dynamic and static objects, which is achieved through object detection algorithms.

[0155] The basic formula for target detection is as follows:

[0156]

[0157] This step produces a set of candidate boxes containing all detected objects in the scene and their categories (such as pedestrians, vehicles, etc.).

[0158] 2. Scenario Complexity Assessment

[0159] Based on the detected object information, the complexity and urgency of the scene can be evaluated by analyzing the following factors:

[0160] Speed ​​and direction of dynamic objects: For example, the direction and speed of a vehicle's movement can be calculated using a tracking algorithm.

[0161] Obstacle density: The number of obstacles in the field of view can be measured by counting the number of candidate boxes.

[0162] The evaluation formula of scene complexity can be expressed as:

[0163]

[0164] in is the complexity score of the scene, It is the scale of an object (the size or area of ​​the object), It is The distance between the object and the user, It is a nonlinear function that takes into account the number, distance, and scale of objects.

[0165] 3. Urgency judgment

[0166] The urgency of the scene is calculated based on the complexity of the scene and the movement of the detected dynamic objects (such as vehicles, pedestrians, etc.). The urgency can be evaluated based on the speed, distance and direction of movement of the object.

[0167] Urgency formula:

[0168]

[0169] in is the urgency score, It is an object speed, It is an object Distance from the user.

[0170] The higher the urgency, the more urgent the system feedback.

[0171] 4. Scene feature generation

[0172] Comprehensive scene complexity and urgency , and finally generate scene features :

[0173]

[0174] in: is the scene feature, α and β are weight parameters used to balance the influence of complexity and urgency.

[0175] Dynamic Adjustment of Feedback

[0176] In scenario-based fine-tuning, feedback not only takes into account the emotional level, but also includes adjustments to the feedback content. For example:

[0177] In normal environments, the system's feedback can be soft, calm, and gentle in tone.

[0178] In emergency scenarios (such as obstacles or sudden dangers appearing ahead), the system will generate feedback with an emergency tone by fine-tuning the model to ensure that users can respond quickly.

[0179] Feedback generation formula:

[0180]

[0181] in, is a speech generation model, is the input text, It is the result of scene recognition. The system fine-tunes according to the scene. Generate dynamically adjusted speech feedback .

[0182] Dynamic situational awareness: The system will not only dynamically generate corresponding voice feedback based on real-time changes in the user's environment, but will also promptly update the feedback content when newly identified dangers appear.

[0183] Application in specific scenarios:

[0184] Low-risk scenarios: In simple, low-risk scenarios (such as when a user is walking in an open, barrier-free environment), the system will generate concise, calm feedback, such as "The road ahead is good, please continue forward." The simplified feedback reduces the user's cognitive burden and improves the overall user experience.

[0185] High-risk scenarios: In complex, high-risk scenarios (such as a fast-approaching vehicle ahead), the system will increase the urgency of the feedback, such as "The vehicle ahead is approaching, please avoid it immediately", and increase the tension and volume in the voice to ensure that the user can perceive the danger and respond quickly.

[0186] Advantages of scenario-based fine-tuning:

[0187] Concise and clear answers: In order to improve the speed at which blind users understand feedback information, the fine-tuned model generates voice feedback that is more concise and easy to understand, removing unnecessary complex expressions, ensuring that users can quickly obtain key information in complex scenarios.

[0188] Enhanced dialogue guidance capabilities: The model will actively guide the user to the next step. For example, when a user asks for a route, the model will not only tell the user how to avoid obstacles, but also provide safe route suggestions to prevent the user from getting lost in complex scenarios.

[0189] Scenario-based recognition and feedback: The model identifies the complexity or urgency of the current scene through the visual and environmental perception system, and dynamically adjusts the tone and emotion of the feedback voice according to these scenes, making the interaction process more natural and humane. Compared with fine-tuning based on the user's tone, scenario-based emotion recognition can more accurately understand the dynamic changes of the external environment and make intelligent feedback based on these changes. Users do not need to actively adjust their tone in complex or urgent scenes, and the system can automatically generate feedback suitable for the scene.

[0190] Strong adaptability: Through scenario-based fine-tuning, the system can adapt to different environments and scenarios, thereby providing more personalized feedback. This feedback is not only based on the user's current state, but also closely related to the external environment.

[0191] The hardware devices and processors required in the above method steps are integrated into a Figure 4 On the physical system shown, the steps for users to use the physical system are as follows:

[0192] Step 1: Turn on the visual scanning module (RGB camera) and the voice acquisition module (microphone) to start capturing images of the surrounding environment and user voice commands.

[0193] Step 2: The RGB camera continuously captures the surrounding scene information, and the microphone collects user voice commands.

[0194] Step 3: The speech recognition model converts speech data into text data.

[0195] Step 3: The text data and image data are transferred to the multimodal large model for analysis.

[0196] Step 4: The multimodal large model performs image recognition and natural language processing on text data and image data to determine user needs and current environmental status.

[0197] Step 5: The multimodal large model returns the parsing results.

[0198] Step 6: Convert the parsing result into voice data through the speech generation model, and play it through the audio output module (such as speakers or headphones). In an emergency (such as an obstacle ahead), the voice prompt volume increases, such as "There is an obstacle ahead, please take a detour."

[0199] Step 7: The user takes the next step based on the system feedback, such as continuing to walk or stopping.

[0200] Step 8: Repeat the above steps to achieve real-time interaction.

[0201] The advantages of using this entity system are as follows:

[0202] 1. Simplify hardware dependencies, reduce costs and power consumption

[0203] Existing auxiliary devices usually rely on a variety of hardware sensors, such as cameras, radars, ultrasonic and other components. These devices are large in size, high in cost and high in power consumption, which affects their portability and applicability for long-term use. The present invention uses a multimodal large model for scene understanding and auxiliary decision-making, and relies on a single RGB camera and microphone to achieve complex environmental perception functions, which greatly simplifies hardware requirements, reduces the production cost and power consumption of the equipment, and improves the portability and endurance of the system.

[0204] 2. Enhance the flexibility and naturalness of interaction

[0205] The interaction methods in the prior art mainly rely on voice prompts and vibration feedback, and cannot dynamically adjust the feedback content or voice emotion according to the changes in the scene, and the interaction method is relatively rigid. In contrast, the present invention realizes a richer and more natural voice interaction through the voice generation model technology, and can dynamically adjust the voice emotion, tone, volume, etc. according to different scenarios, making the interaction process more humane. For example, the system can automatically increase the voice volume and emphasize the tone in an emergency to remind the user to pay attention, or provide more relaxed and gentle feedback in daily use.

[0206] 3. Achieve independent operation and reduce dependence on external devices

[0207] Many existing blind assistive devices need to rely on smartphones or cloud devices for data processing, which causes the system to not work properly when the network status is poor, and there are reliability issues. The present invention uses an embedded multimodal large model to run independently without relying on external devices or cloud services, ensuring the real-time and reliability of the system, and maintaining normal operation even without a network. This greatly improves the user experience in various environments.

[0208] 4. Multimodal information fusion to improve the ability to understand complex scenes

[0209] When dealing with complex and ever-changing scenarios, existing devices can often only perform simple obstacle recognition and are unable to accurately understand the user's intentions or respond to dynamic scenarios. The present invention uses a multimodal large model to integrate multi-source information such as images and voices, and can accurately identify obstacles, pedestrians, and user intentions in complex environments. The collision rate of users using this system in complex environments is significantly reduced. For example, while the user is walking, the system can not only identify physical obstacles, but also analyze the user's destination or needs through voice, and provide accurate navigation or scene descriptions accordingly. In addition, the system also has a dynamic prediction function, which can predict potential risks in advance and improve safety.

[0210] 5. Scenario Adaptability and Situational Awareness

[0211] The present invention has scene adaptability and can automatically adjust the feedback mode and content according to environmental changes (such as indoor and outdoor, day and night). For example, in the case of complex outdoor lighting, the system can improve the accuracy of obstacle recognition by adjusting the perception parameters; while in a quiet indoor environment, the system can reduce volume interference to meet the needs of users. This situational awareness enables the system to respond to different scenes more flexibly and improves the user's experience satisfaction.

[0212] Example 3

[0213] This embodiment provides a blind person assisted interaction system based on a multimodal large model, including:

[0214] User demand and external environment acquisition module: used to obtain user voice commands and external environment images and pre-process the acquired data;

[0215] Environmental understanding result acquisition module: used to input the pre-processed user voice commands and external environment images into the pre-trained multimodal large model to obtain environmental understanding results;

[0216] Feedback generation module: used to integrate the environmental understanding results into voice feedback information, and adjust the tone of the voice feedback information according to environmental characteristics;

[0217] Feedback output module: used to output the voice feedback information after intonation adjustment to the user;

[0218] The method for obtaining the pre-trained multimodal large model includes:

[0219] Acquire historical multimodal data;

[0220] The historical multimodal data is input into a pre-built multimodal large model, and the multimodal large model in training is fine-tuned by freezing the base layer in the model to obtain a trained multimodal large model.

[0221] Example 4

[0222] This embodiment provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the blind person assisted interaction method based on a multimodal large model as described in any step of Embodiment 2 is implemented.

[0223] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0224] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0227] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A blind person assisted interaction method based on a multimodal large model, characterized in that: include: Obtain user voice commands and external environment images and pre-process the acquired data; Input the pre-processed user voice commands and external environment images into the pre-trained multimodal large model to obtain the environmental understanding results; Integrating the environmental understanding results into voice feedback information, and adjusting the tone of the voice feedback information according to environmental characteristics; Outputting voice feedback information after intonation adjustment to the user; The method for obtaining the pre-trained multimodal large model includes: Acquire historical multimodal data; The historical multimodal data is input into a pre-built multimodal large model, and the multimodal large model in training is fine-tuned by freezing the base layer in the model to obtain a trained multimodal large model.

2. The blind person assisted interaction method based on a multimodal large model according to claim 1, characterized in that: The obtaining of user requirements and external environment images and preprocessing of the obtained data includes: The microphone collects the user's voice commands and the RGB camera collects image information of the user's surrounding environment; Standardize and denoise the image information of the surrounding environment to obtain effective image data; The voice instructions are converted into text instructions using a speech recognition model.

3. The blind person assisted interaction method based on a multimodal large model according to claim 1 or 2, characterized in that: The pre-built multimodal large model includes an encoding layer, a cross-modal attention layer, and a decoding layer connected in sequence; The coding layer includes a parallel image encoder and a text encoder; The cross-modal attention layer includes multiple groups of self-attention layers and feed-forward layers connected in sequence; The pre-processed user voice commands and external environment images are input into the pre-trained multimodal large model to obtain the environmental understanding results, including: Performing image feature extraction on input valid image data through an image encoder, and performing text feature extraction on input text instructions through a text encoder; Obtaining scene understanding results according to the image feature vector and the text feature vector through multiple sets of self-attention layers and feed-forward layers; The scene understanding result is output through the decoding layer.

4. The blind person assisted interaction method based on a multimodal large model according to claim 3 is characterized in that: The image feature extraction is performed on the input valid image data by using an image encoder, and the text feature extraction is performed on the input text instruction by using a text encoder, including: The image encoder converts the valid image data into an image feature vector through a deep convolutional neural network, and the calculation formula is: ， in, represents the convolution operation, is the convolution kernel, is the valid image data, is the bias term, is the image feature vector; The text encoder obtains the attention distribution of the word vector in the text instruction through the self-attention layer to obtain the text feature vector, and the calculation formula is: ， ， in, is the text feature vector, is the word vector in the text instruction, , , are the weight matrices, , , The word vectors are After linear transformation, the matrix For the matrix The transpose of For the matrix Dimension.

5. The blind person assisted interaction method based on multimodal large model according to claim 4 is characterized in that: The obtaining of scene understanding results according to the image feature vector and the text feature vector through multiple sets of self-attention layers and feed-forward layers includes: The self-attention layer calculates the correlation between the text feature data and the image feature data to obtain the scene understanding result after the fusion of text and image. The calculation formula is as follows: ， in, is the fused scene understanding result, is the image feature vector, is the text feature vector, is the cross-modal attention mechanism, is the image feature The transpose of is the image feature Dimensions; The feed-forward layer performs linear transformation on the scene understanding result and then outputs it.

6. The blind person assisted interaction method based on multimodal large model according to claim 5, characterized in that: The step of integrating the environmental understanding result into voice feedback information and adjusting the intonation of the voice feedback information according to environmental characteristics includes: Generate text feedback information according to the scene understanding result; Determining scene characteristics according to the scene understanding result, wherein the scene characteristics include scene complexity and urgency; The text feedback information is converted into voice feedback information of corresponding intonation according to the scene feature.

7. The blind person assisted interaction method based on a multimodal large model according to claim 6, characterized in that: The determining the scene characteristics according to the scene understanding result includes: Extracting key elements from the scene understanding result according to the target detection algorithm, wherein the key elements include dynamic objects and static objects; The scenario complexity is evaluated based on the key factors, and the evaluation formula is as follows: ， in, is the complexity score of the scene, It is The size of an object, It is The distance between the object and the user, It is a nonlinear function that takes into account the number, distance and scale of objects; According to the scene complexity and the extracted dynamic objects, the urgency of the scene is calculated, and the calculation formula is as follows: ， in is the urgency score, It is an object speed, It is an object Distance from the user; The scenario characteristics are obtained based on the complexity and urgency of the scenario, and the calculation formula is as follows: ， in, is the scene feature, α and β are the weight parameters of complexity and urgency, respectively.

8. The blind person assisted interaction method based on multimodal large model according to claim 7, characterized in that: The converting the text feedback information into voice feedback information of corresponding intonation according to the scene feature comprises: The text feedback information is converted into voice feedback information through a voice generation model, and the voice generation model is fine-tuned according to the characteristics of the scene, and the voice feedback information is output according to the corresponding intonation. The voice feedback output formula is: ， in, It is voice feedback information. is a speech generation model, It is text feedback information.

9. A blind person assisted interaction system based on a multimodal large model, characterized in that: include: User demand and external environment acquisition module: used to obtain user voice commands and external environment images and pre-process the acquired data; Environmental understanding result acquisition module: used to input the pre-processed user voice commands and external environment images into the pre-trained multimodal large model to obtain environmental understanding results; Feedback generation module: used to integrate the environmental understanding results into voice feedback information, and adjust the tone of the voice feedback information according to environmental characteristics; Feedback output module: used to output the voice feedback information after intonation adjustment to the user; The method for obtaining the pre-trained multimodal large model includes: Acquire historical multimodal data; The historical multimodal data is input into a pre-built multimodal large model, and the multimodal large model in training is fine-tuned by freezing the base layer in the model to obtain a trained multimodal large model.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the blind person assisted interaction method based on a multimodal large model as described in any one of claims 1 to 8 is implemented.

Images