Blind guiding scene identification method based on multi-modal visual large model

Abstract

The invention relates to the field of computer vision and the field of image processing, in particular to a blind guiding scene recognition method based on a multi-modal visual large model. Comprising the following steps: data acquisition: acquiring image or video data of a typical blind guiding scene, marking, constructing a data set, introducing an image prompting mechanism, and dividing a training set and a test set; constructing a multi-modal visual large model which comprises an image coding module, a semantic prompt coding module, a visual language fusion module and a scene analysis semantic decoding module; the model is tested and optimized, and a multi-task loss function is adopted for optimization; a blind guiding auxiliary system is constructed, image acquisition, visual understanding and voice broadcasting functions are integrated, and a closed-loop process is realized. According to the method, the problems of strong target dependence, poor universality and weak semantic comprehension ability of the existing blind guiding identification technology are solved, accurate identification and semantic feedback of complex scenes are supported, and the intelligent level and the environment adaptability of a blind guiding system are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a method for recognizing blind-guiding scenes based on a multimodal visual large model. Background Art

[0002] Against the backdrop of the rapid development of artificial intelligence and perceptual intelligence, large-scale model technology based on visual understanding and multimodal perception is becoming a core means of enhancing environmental cognition and human-computer interaction capabilities. With the continuous breakthroughs of large-scale visual models in areas such as image recognition, semantic segmentation, and multimodal alignment, their capabilities in tasks such as assisted perception, semantic understanding, and scene interpretation have been widely verified. In particular, in intelligent assistance applications for vulnerable groups, building a visual recognition system with high generalization and semantic understanding capabilities has become a key research direction in guide-assisted technology. Guide-assisted scene recognition, as a key link between environmental perception and human action, essentially involves multi-level semantic deconstruction of dynamic, complex, and ever-changing real-world environments and communicating the results in a perceptible manner to blind users.

[0003] Currently, mainstream scene recognition methods for blind guidance rely primarily on deep learning-based object detection algorithms. These methods pre-train models to identify specific categories of obstacles or traffic elements, thereby assisting users in avoiding risks. However, these methods face significant limitations in practical applications. First, recognition capabilities are limited by the object scope of the training dataset, making them incapable of handling newly discovered unknown objects in complex urban environments. Second, the models lack the ability to understand the global scene semantics, making it difficult to provide high-level, coherent scene explanations, such as sentence-level descriptions like "The traffic light is red, and a vehicle is crossing the crosswalk ahead." Third, they lack generalization capabilities in changing environments, particularly under varying geographic, lighting, and weather conditions, and are prone to recognition failures. Furthermore, existing technologies generally fail to generate personalized scene perception results based on the user's current needs or guidance intent, nor do they dynamically adjust and respond to natural language interactions with the user, significantly compromising the user experience and assistance effectiveness. Summary of the Invention

[0004] In response to the technical problems existing in the above-mentioned background technology, the present invention proposes a method for blind-guiding scene recognition based on a large multimodal visual model.

[0005] In order to achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0006] S1. Data acquisition: collecting image or video data in a variety of typical blind guide scenarios, including urban roads, crosswalks, traffic signals, and temporary obstacle environment information;

[0007] S2. Build a multimodal visual model, including an image encoding module, a semantic prompt encoding module, a visual language fusion module (VLM), and a scene analysis semantic decoding module;

[0008] S3. Test and optimize the multimodal vision model to verify its recognition accuracy, semantic generation effect, and generalization ability in different scenarios.

[0009] S4. Build a guide assistance system that integrates image acquisition, visual understanding, and voice broadcasting functions to achieve a complete closed-loop process from image to semantics to voice feedback.

[0010] Preferably, the specific implementation of step S1 is as follows:

[0011] S1.1. Use a wearable camera or mobile terminal to capture images and video data of typical blind guidance scenes;

[0012] S1.2. After acquiring the data, annotate the data to generate image semantic label data object = top_left, down_right, class, where top_left is the coordinate of the top left corner of the bounding box of the target in the image, down_right is the coordinate of the bottom right corner of the bounding box of the target in the image, and class is the category of the target in the image;

[0013] S1.3. Construct a multimodal visual large-scale scene semantic description dataset, pair each image segment with the corresponding scene semantic prompt description, and the language training data txt corresponding to the image;

[0014] S1.4. Introduce an image prompt mechanism to guide the multimodal visual model for tasks, and design corresponding semantic prompt word templates based on different blind guidance tasks;

[0015] S1.5. Get the final dataset D = img, prompt, object, txt, and divide the dataset into 80% training set and 20% test set.

[0016] Preferably, the steps of building the image encoding module in step S2 specifically include:

[0017] S2.1.1. Use the YOLOv8 backbone network as the basic framework for image feature extraction, input the image to obtain the image feature map;

[0018] S2.1.2. Divide the feature map into image blocks of fixed size, each image block representing a local region in the image;

[0019] S2.1.3. Perform patch encoding on each image block, mapping the original image block into a feature vector of uniform dimension through linear transformation;

[0020] S2.1.4. Introduce a relative position encoding mechanism to maintain the spatial order of image blocks in the image, ensure that the image feature vector has spatial distinctiveness and directionality when performing semantic alignment, and obtain an image feature vector representation suitable for semantic alignment.

[0021] Preferably, the steps of building the semantic hint encoding module in step S2 specifically include:

[0022] S2.2.1. Introduce a task-directive prompt mechanism to express user requirements or system default tasks in natural language as semantic prompts, which serve as control information to guide visual perception.

[0023] S2.2.2. Perform token-level word segmentation on the prompt text to split a complete semantic instruction into several semantic minimum units;

[0024] S2.2.3. Encode each token and map it into a fixed-dimensional word vector representation using a lookup table to preserve its semantic information.

[0025] S2.2.4. Introducing a rotational position encoding mechanism to combine position information with word vectors, enabling the model to understand the word order relationship between words;

[0026] S2.2.5. A 4-layer Transformer Encoder structure is used as the main module of the Prompt text encoder. Each layer consists of a multi-head self-attention mechanism and a feedforward neural network to model the global dependency relationship between words in the text and obtain the Prompt latent space feature vector.

[0027] Preferably, the steps of building the visual language fusion module VLM in step S2 specifically include:

[0028] S2.3.1. Based on the Transformer architecture, we construct a four-layer fused Transformer Encoder stack, where each layer consists of a multi-head cross-attention mechanism and residual connections.

[0029] S2.3.2. Use a multi-head cross-attention mechanism to achieve dynamic perception of language tokens by image features, and directional attention of language features to visual patches;

[0030] S2.3.3. The residual connection mechanism is used to ensure the stable transmission of features during the fusion process, and to obtain a context-sensitive, task-driven, and structurally robust multimodal semantic expression.

[0031] Preferably, the steps of building the scene analysis semantic decoding module in step S2 specifically include:

[0032] S2.4.1. A four-layer Transformer Decoder structure is used, each layer consists of a multi-head self-attention mechanism to adaptively model the fused representation and gradually generate content;

[0033] S2.4.2. Decoding outputs three types of key information, including natural language description of the scene, instruction-type semantic prompts, and structured target information.

[0034] Preferably, the construction process of the multimodal visual large model in step S2 adopts a dual-tower structure, in which the image encoding module and the semantic prompt encoding module encode the image and prompt respectively, and perform semantic fusion alignment in the visual language fusion module, and finally obtain the scene description and prompt, the target and coordinates in the scene through the scene analysis semantic decoding module, and finally obtain the multimodal visual large model.

[0035] As a preference, the specific operation of testing and tuning the multimodal visual large model in step S3 is:

[0036] S3.1. The training set in the dataset is fed into the multimodal vision model. The learning rate change strategy adopts warmup and the training is repeated for 50 rounds.

[0037] S3.2. The target category loss function in the scene adopts the cross entropy loss function, which is calculated as follows: Among them, p(class i ) is the true target category, q(class i ) is the predicted target category probability, n is the number of categories;

[0038] S3.3. The target coordinate loss function in the scene adopts the IOU loss function, which is calculated as follows: The annotation box is the actual location of the target in the dataset, and the prediction box is the location of the target inferred by the model;

[0039] S3.4, scene description and prompt loss function uses cosine similarity as the loss function, and the calculation formula is: Among them, T s is the real scene description in the dataset, s is the length of the real scene description, T t is the scene description predicted by the model, and t is the length of the scene description predicted by the model;

[0040] S3.5, combined with the loss functions of S3.2, S3.3, and S3.4, we get the total loss function of the multimodal vision model: Loss = Loss class +Loss palce +Loss scenes ;

[0041] S3.6. Feed the test set in the dataset into the model to test and tune the multimodal vision model to verify its recognition accuracy, semantic generation effect, and generalization ability in different scenarios.

[0042] Preferably, the blind guide assistance system in step S4 is constructed as follows:

[0043] S4.1. First, collect images of the blind-guiding scene;

[0044] S4.2. The user inputs the task prompt text using speech-to-text technology;

[0045] S4.3. Using a large multimodal visual model, the scene is analyzed after inputting images and task prompts, and the position data of each target in the scene in the world coordinate system is obtained to achieve obstacle detection, scene interpretation, and passage judgment tasks;

[0046] S4.4. Generate natural language prompts from the output semantic data and convert them into speech signals through text-to-speech technology, supporting multi-language and personalized voice selection;

[0047] S4.5. Conduct multi-scenario and multi-user testing to verify the system’s performance in different environments and adjust the system based on user feedback.

[0048] Compared with the existing technology, the advantages and positive effects of the present invention are that the present invention provides a method for blind guide scene recognition based on a large visual model, which solves the problems of strong target dependence, poor versatility, and weak semantic understanding ability in the existing blind guide recognition technology. This method introduces a large visual language model with strong generalization ability, and realizes the unified recognition and semantic analysis of various blind guide elements in complex environments without the need for training for specific targets. First, the system extracts environmental visual features based on image input, and guides the model to perform specific blind guide tasks through natural language prompts (Prompt), so that the model can adapt to different scene requirements and generate scene descriptions that conform to semantic logic. Secondly, combined with the continuous frame input processing mechanism and the multimodal alignment strategy, the coherence and stability of the model in dynamic scenes are improved, ensuring that the output information is highly consistent and practical.

[0049] Through the above-mentioned approach, the present invention can break through the limitations of traditional target detection methods, support accurate recognition and semantic feedback in the absence of targets, complex scenes and dynamic environments, and effectively improve the intelligence level and environmental adaptability of the guide system. This method also has good scalability and can be integrated into various intelligent guide terminals. It is widely applicable to various application scenarios such as travel assistance, safety reminders and scene understanding for the visually impaired, significantly improving the user's independence and travel safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0051] Figure 1 This is a schematic diagram of the image encoding module structure of a multimodal visual large model for a blind-guiding scene recognition method based on a multimodal visual large model.

[0052] Figure 2 This is a schematic diagram of the structure of the semantic cue encoding module of a multimodal visual large model for a blind-guiding scene recognition method based on a multimodal visual large model.

[0053] Figure 3 This is a schematic diagram of the visual language fusion module structure of a multimodal visual large model for a guide blind scene recognition method based on a multimodal visual large model.

[0054] Figure 4 This is a structural diagram of the scene analysis and semantic decoding module of a multimodal visual large model for a guide blind scene recognition method based on a multimodal visual large model.

[0055] Figure 5 The figure is a schematic diagram of the structure of a multimodal visual large model for a blind guide scene recognition method based on a multimodal visual large model.

[0056] Figure 6 The diagram shows the structure of a blind guide assistance system based on a blind guide scene recognition method of a multimodal visual large model. DETAILED DESCRIPTION

[0057] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] In the embodiment, the visually impaired group faces the core demand of complex environmental perception during travel. Existing guide devices mostly rely on traditional target detection technology, which is difficult to cope with dynamically changing urban scenes and unstructured semantic understanding. With the popularization of smart wearable devices, users are increasingly demanding real-time, accurate and natural language interactive guide systems. The present invention addresses the pain points of poor recognition versatility and weak semantic generation capabilities in the existing technology, relying on the strong generalization characteristics of multimodal visual large models to build a complete technology chain covering data collection, model training, and system integration. A guide scene recognition method based on a large visual model is proposed.

[0060] Considering that existing guide data is labeled specifically and lacks scene semantic description and task guidance mechanism, the model's ability to understand complex scenes is insufficient. This step constructs a multimodal dataset containing visual features, semantic labels, and task prompts through multi-source data collection and refined annotation, providing rich input information for model training. Use wearable cameras or mobile terminals to capture typical guide scene images and video data, including day and night, sunny and rainy days, urban roads, crosswalks, stairs, building entrances and exits, construction areas, and other environments to obtain representative real-world travel scene images and form a basic guide image dataset img. Use annotation tools (such as LabelMe, CVAT, etc.) to semantically annotate the image data and draw bounding boxes, including key visual elements such as sidewalks, traffic lights, traffic signs, obstacles, pedestrians, and vehicles, to generate image semantic label data object = top_left, down_right, class. Where top_left is the coordinate of the upper left corner of the target's bounding box in the image, down_right is the coordinate of the lower right corner of the target's bounding box in the image, and class is the category of the target in the image. A multimodal visual large-scale model scene semantic description dataset was then constructed. Each image segment was paired with a corresponding scene semantic prompt description (e.g., "There is a moving car ahead, please stop"), along with the corresponding image language training data (txt). An image prompt mechanism was introduced to guide the multimodal visual large-scale model through tasks, and corresponding semantic prompt word templates were designed based on different blind guidance tasks (such as obstacle detection, passage judgment, and scene interpretation). The resulting dataset D = img, prompt, object, txt was partitioned into 80% training and 20% test sets to ensure diverse and representative training data. The test data also covered challenging scenes such as complex, edge, and low-light scenes, allowing for a comprehensive evaluation of model performance and generalization.

[0061] Considering that traditional models rely on fixed target detection frameworks, lack the ability to dynamically respond to user needs, and the independent processing of visual and language semantics leads to low fusion efficiency. This step achieves a deep fusion of visual features and language prompts through a dual-tower structure and a cross-modal attention mechanism, enabling the model to have task-driven scene understanding capabilities. The present invention constructs a multimodal visual large model, including an image encoding module, a semantic prompt Prompt encoding module, a visual language fusion module VLM, and a scene analysis semantic decoding module.

[0062] First, build the image encoding module, such as Figure 1As shown in the figure, to effectively understand and express the surrounding environment, an image encoding module for semantic alignment tasks was built. Its core goal is to transform complex visual information into semantic feature representations suitable for language interaction. The YOLOv8 backbone network is used for image feature extraction in this module. After obtaining the feature map, it is divided into fixed-size image patches to enhance local semantic expression. Each patch represents a local region in the image, such as a portion of an object, a patch of ground, or the edge of an obstacle. This partitioning allows the system to more finely capture regional features in the image, facilitating fine-grained semantic distinctions in the subsequent description generation process, such as "There is a red car in front of the left" or "There is an obstacle on the sidewalk on the right." Patch encoding is performed on each image patch, mapping the original image patch into a feature vector of uniform dimension through a linear transformation. These feature vectors preserve the semantic and visual information of the local image region and serve as the basis for subsequent language generation or command generation. A relative position encoding mechanism is introduced to maintain the spatial order of image patches within the image. Spatial relationships are particularly critical in assistive scenarios for the blind. For example, descriptions such as "in front," "to the right," and "two meters away" rely on accurate spatial position recognition. By adding position encoding, we can ensure that the image feature vector has spatial distinction and direction when performing semantic alignment. Finally, we get the image encoding module and its output image feature vector representation V suitable for semantic alignment. img .

[0063] Then build the semantic prompt encoding module, such as Figure 2As shown, task-related text instructions are converted into vector representations with deep semantic expression capabilities. These serve as guiding signals for image semantic alignment and subsequent information generation, thereby improving the model's responsiveness to the blind user's intentions. A task-instruction prompt mechanism is introduced, explicitly expressing user requirements or system default tasks in natural language as "semantic prompts" to serve as control information guiding visual perception. Typical prompts include: "Is it safe to pass ahead?", "Is there an obstacle?", "Current traffic signal status?", "Is there a pedestrian crossing?", "Is there an approaching pedestrian?", etc. During the encoding process, the prompt text is first tokenized. This process breaks a complete semantic instruction into several minimal semantic units (such as words, subwords, or characters) to prepare for subsequent modeling. Each token is then encoded, typically using a table lookup to map it to a fixed-dimensional word vector representation, preserving its semantic information. A rotational position encoding mechanism is introduced to incorporate the sequential structure of tokens in the sequence and improve model generalization. By combining position information with word vectors, the model can understand the word order relationship between words. For example, "Is there an obstacle" and "Does the obstacle exist" are semantically close, but the order consistency needs to be maintained in attention modeling. A 4-layer TransformerEncoder structure is used as the main module of the Prompt text encoder. Each layer of TransformerEncoder consists of a multi-head self-attention mechanism and a feedforward neural network, which can model the global dependency relationship between each word in the text, so that the output representation is not just a stacking of local semantics, but a fusion of global contextual semantics. In the multi-head attention mechanism, the model calculates the attention weight of each Token with other Tokens in the sequence, thereby mining the semantic associations such as "pass" and "safety", "obstacle" and "existence", etc., which is especially suitable for task-based instruction recognition in blind assistance, such as judging whether the conditions for passage are met. Finally, the semantic prompt (Prompt) encoding module and its output Prompt latent space feature vector V are obtained. prompt .

[0064] For the construction of the visual language fusion module (VLM), such as Figure 3As shown in Figure 1, the design goal of this module is to deeply fuse the visual features extracted by the image encoding module with the semantic cue features generated by the prompt encoding module, thereby achieving collaborative modeling and semantic alignment of multimodal information. Based on the Transformer architecture, this fusion module constructs a four-layer fused TransformerEncoder stack. Each layer consists of a multi-head cross-attention mechanism and residual connections. This aims to efficiently capture the correlation between visual and language modalities and enhance the flexibility and stability of information exchange. The multi-head cross-attention mechanism is a key component of this module. Unlike the self-attention mechanism, which only models dependencies within a single modality, the cross-attention mechanism enables dynamic perception of language tokens based on image features and targeted attention of language features to visual patches. For example, when the prompt is "Is there an obstacle ahead?", cross-attention gives higher attention weight to the "front area" and "unusually shaped objects" in the image. Conversely, when a traffic light is detected in the image, the system can also use cross-attention to map semantic cues such as the "current traffic signal status." The residual connection mechanism ensures stable feature transfer during the fusion process, alleviating the problems of vanishing gradients or feature degradation that may arise from multi-layer information interaction. The input of each layer is weightedly fused with its output, effectively preserving low-level semantics while enhancing high-level semantic abstraction. This module not only achieves the fusion of graphic and text modalities, but also emphasizes the deep integration of semantically guided directionality and scene perception in its structural design. Through continuous cross-attention modeling, the system gradually forms a context-sensitive, task-driven, and structurally robust multimodal semantic expression.

[0065] Finally, we build the scene analysis semantic decoding module, such as Figure 4As shown in the figure, the high-dimensional features output by the aforementioned visual language fusion module (VLM) are further decoded into structured natural language and location information, thereby realizing a broadcastable and interactive scene semantic expression. This module is constructed using a 4-layer TransformerDecoder structure, each layer of which is composed of a multi-head self-attention mechanism to adaptively model the fusion representation and gradually generate content. Specifically, the content output by the semantic decoding module covers three types of key information: (1) Scene natural language description: including an overall description of the current environmental state, such as "There is a parked car ten meters ahead, a green belt on the left, and a clear sidewalk on the right front." This type of output helps blind users form a spatial impression of the scene and improve their travel confidence. (2) Instructive semantic prompts: Combining prompt content with visual perception results to generate task-oriented sentences, such as "The current traffic light is red, please wait", "You can detour to the left to avoid obstacles", and "The road ahead is safe, you can continue to move forward." This content directly supports the command generation of the voice broadcast module. (3) Structured target information (target category + coordinates): In addition to text output, the module can also decode the location and category of key entities in the scene, such as "pedestrian (x = 120, y = 200)", "obstacle (x = 85, y = 320)", "zebra crossing (x = 150, y = 95)", etc. This information can be used as input for downstream modules such as path planning and collision warning. Within the Transformer Decoder, through the multi-head self-attention mechanism, the model can model the contextual dependencies between each position in the decoding sequence, thereby ensuring that the generated scene description has linguistic fluency and logical coherence. For example, after describing "there is an obstacle ahead", the model can automatically continue with reasonable instructions such as "it is recommended to detour to the left" without generating semantic jumps or contradictory content.

[0066] Combining the above four modules, we can build a multimodal visual model, such as Figure 5 As shown, the entire multimodal vision model adopts a typical dual-encoder architecture. In this structure, the image and language modalities are independently encoded and subsequently deeply fused in a shared space, preserving the autonomous nature of the modalities while enhancing cross-modal interaction. The multimodal vision model process is as follows: the image encoding module and the semantic prompt encoding module encode the image and prompt respectively, and semantic fusion and alignment are performed in the visual language fusion module. Finally, the scene analysis and semantic decoding module obtains the scene description and prompt, as well as the objects and coordinates within the scene, ultimately forming the multimodal vision model.

[0067] Considering that the existing model optimization only optimizes the target detection loss and ignores the joint optimization of semantic generation and coordinate positioning, the output information is incoherent and the positioning accuracy is low. This step uses a multi-task loss function to simultaneously optimize the target category, coordinate position, and scene description to improve the overall performance of the model. The training set in the dataset is sent to the multimodal vision large model, and the training method adopts warmup as the learning rate change strategy, and training is repeated for 50 rounds. Among them, the target category loss function in the scene adopts the cross entropy loss function, which is calculated as follows: Among them, p(class i ) is the true target category, q(class i ) is the predicted target category probability, and n is the number of categories. The target coordinate loss function in the scene uses the IOU loss function, which is calculated as follows: The labeled box is the actual location of the target in the dataset, and the predicted box is the location of the target inferred by the model. The scene description and prompt loss function uses cosine similarity as the loss function, and the calculation formula is: Among them, T s is the real scene description in the dataset, s is the length of the real scene description, T t is the scene description predicted by the model, and t is the length of the scene description predicted by the model; combining the above loss functions, we can get the total loss function of the multimodal vision model: Loss = Loss class +Loss palce +Loss scenes , the test set D in the dataset is fed into the model, and the multimodal vision model is tested and tuned to verify its recognition accuracy, semantic generation effect and generalization ability in different scenarios.

[0068] Finally, a blind guide assistance system is constructed, such as Figure 6As shown, image acquisition uses a high-precision camera (such as a depth camera) to capture images of the guidance scene in real time under different environments. The camera position and angle are appropriately selected based on user needs. Autofocus and dynamic camera adjustments ensure clear and stable image input in all environments. Users input voice input, and then speech-to-text technology is used to generate task prompts. Visual understanding utilizes the resulting multimodal visual model and scene analysis derived from image and task prompt input. Combined with the depth camera, the position data of each object in the scene in the world coordinate system (3D space) is obtained to implement tasks such as obstacle detection, scene interpretation, and passage judgment. Natural language prompts (e.g., "Obstacle ahead, please avoid") are generated from the output semantic data and converted into speech signals using text-to-speech (TTS) technology. This module supports multiple languages and personalized voice selection to provide users with clear and accurate voice feedback. Finally, multi-scenario, multi-user testing was conducted to verify the system's performance in various environments, such as crowded streets and quiet indoor spaces. The system is adjusted based on user feedback to ensure the accuracy of voice prompts, the precision of image recognition, and the clarity of voice feedback, ultimately achieving an efficient and convenient blind guidance function.

[0069] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any person skilled in the art may utilize the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes for application in other fields. However, any simple modification, equivalent change, and modification of the above embodiments made in accordance with the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for blind-guiding scene recognition based on a multimodal visual large model, characterized in that: The following steps are involved: S1. Data acquisition: collecting image or video data in a variety of typical blind guide scenarios, including urban roads, crosswalks, traffic signals, and temporary obstacle environment information; S2. Build a multimodal visual model, including an image encoding module, a semantic prompt encoding module, a visual language fusion module (VLM), and a scene analysis semantic decoding module; S3. Test and optimize the multimodal vision model to verify its recognition accuracy, semantic generation effect, and generalization ability in different scenarios. S4. Build a guide assistance system that integrates image acquisition, visual understanding, and voice broadcasting functions to achieve a complete closed-loop process from image to semantics to voice feedback.

2. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The specific implementation of step S1 is: S1.

1. Use a wearable camera or mobile terminal to capture images and video data of typical blind guidance scenes; S1.

2. After acquiring the data, annotate the data to generate image semantic label data object = top_left, down_right, class, where top_left is the coordinate of the top left corner of the bounding box of the target in the image, down_right is the coordinate of the bottom right corner of the bounding box of the target in the image, and class is the category of the target in the image; S1.

3. Construct a multimodal visual large-scale scene semantic description dataset, pair each image segment with the corresponding scene semantic prompt description, and the language training data txt corresponding to the image; S1.

4. Introduce an image prompt mechanism to guide the multimodal visual model for tasks, and design corresponding semantic prompt word templates based on different blind guidance tasks; S1.

5. Get the final dataset D = img, prompt, object, txt, and divide the dataset into 80% training set and 20% test set.

3. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The steps of building the image encoding module in step S2 specifically include: S2.1.

1. Use the YOLOv8 backbone network as the basic framework for image feature extraction, input the image to obtain the image feature map; S2.1.

2. Divide the feature map into image blocks of fixed size, each image block representing a local region in the image; S2.1.

3. Perform patch encoding on each image block, mapping the original image block into a feature vector of uniform dimension through linear transformation; S2.1.

4. Introduce a relative position encoding mechanism to maintain the spatial order of image blocks in the image, ensure that the image feature vector has spatial distinctiveness and directionality when performing semantic alignment, and obtain an image feature vector representation suitable for semantic alignment.

4. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The steps of building the semantic hint encoding module in step S2 specifically include: S2.2.

1. Introduce a task-directive prompt mechanism to express user requirements or system default tasks in natural language as semantic prompts, which serve as control information to guide visual perception. S2.2.

2. Perform token-level word segmentation on the prompt text to split a complete semantic instruction into several semantic minimum units; S2.2.

3. Encode each token and map it into a fixed-dimensional word vector representation using a lookup table to preserve its semantic information. S2.2.

4. Introducing a rotational position encoding mechanism to combine position information with word vectors, enabling the model to understand the word order relationship between words; S2.2.

5. A 4-layer Transformer Encoder structure is used as the main module of the Prompt text encoder. Each layer consists of a multi-head self-attention mechanism and a feedforward neural network to model the global dependency relationship between words in the text and obtain the Prompt latent space feature vector.

5. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The steps of building the visual language fusion module VLM in step S2 specifically include: S2.3.

1. Based on the Transformer architecture, we construct a four-layer fused Transformer Encoder stack, where each layer consists of a multi-head cross-attention mechanism and residual connections. S2.3.

2. Use a multi-head cross-attention mechanism to achieve dynamic perception of language tokens by image features, and directional attention of language features to visual patches; S2.3.

3. The residual connection mechanism is used to ensure the stable transmission of features during the fusion process, and to obtain a context-sensitive, task-driven, and structurally robust multimodal semantic expression.

6. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The steps of building the scene analysis semantic decoding module in step S2 specifically include: S2.4.

1. A four-layer Transformer Decoder structure is used, each layer consists of a multi-head self-attention mechanism to adaptively model the fused representation and gradually generate content; S2.4.

2. Decoding outputs three types of key information, including natural language description of the scene, instruction-type semantic prompts, and structured target information.

7. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The construction process of the multimodal visual large model in step S2 adopts a dual-tower structure. The image encoding module and the semantic prompt encoding module encode the image and prompt respectively, and perform semantic fusion and alignment in the visual language fusion module. Finally, the scene analysis and semantic decoding module obtains the scene description and prompt, the target and coordinates in the scene, and finally obtains the multimodal visual large model.

8. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The specific operation of testing and tuning the multimodal vision large model in step S3 is as follows: S3.1, sending the training set in the data set to the multimodal vision large model, using warmup as the training method and learning rate change strategy, and training for 50 rounds. S3.2, among which: The target category loss function in the scene uses the cross entropy loss function, which is calculated as follows: Among them, p(class i ) is the true target category, q(class i ) is the predicted target category probability, n is the number of categories; S3.

3. The target coordinate loss function in the scene adopts the IOU loss function, which is calculated as follows: The annotation box is the actual location of the target in the dataset, and the prediction box is the location of the target inferred by the model; S3.4, scene description and prompt loss function uses cosine similarity as the loss function, and the calculation formula is: Among them, T s is the real scene description in the dataset, s is the length of the real scene description, T t is the scene description predicted by the model, and t is the length of the scene description predicted by the model; S3.5, combined with the loss functions of S3.2, S3.3, and S3.4, we get the total loss function of the multimodal vision model: Loss = Loss class +Loss palce +Loss scenes ; S3.

6. Feed the test set in the dataset into the model to test and tune the multimodal vision model to verify its recognition accuracy, semantic generation effect, and generalization ability in different scenarios.

9. The method for blind-guiding scene recognition based on a multimodal visual large model according to claim 1, characterized in that: The construction of the blind guide assistance system in step S4 is as follows: S4.

1. First, collect images of the blind-guiding scene; S4.

2. The user inputs the task prompt text using speech-to-text technology; S4.

3. Using a large multimodal visual model, the scene is analyzed after inputting images and task prompts, and the position data of each target in the scene in the world coordinate system is obtained to achieve obstacle detection, scene interpretation, and passage judgment tasks; S4.

4. Generate natural language prompts from the output semantic data and convert them into speech signals through text-to-speech technology, supporting multi-language and personalized voice selection; S4.

5. Conduct multi-scenario and multi-user testing to verify the system’s performance in different environments and adjust the system based on user feedback.

Images