Multi-modal end-to-end guide dog model training method based on blind path scene data distillation

Abstract

The application discloses a kind of multi-modal end-side guide blind large model training methods based on blind road scene data distillation.The method is through the coordination distillation of domain multi-modal perception large model and depth estimation large model, automatically generates blind road scene annotation data for training end-side guide blind large model.In target detection data distillation and training, construct blind road scene image dataset, design structured prompt template, using Seed-1.5-VL model annotation, after artificial verification correction, based on Qwen2-VL-7B-Instruct model carries out LoRA efficient fine-tuning;When depth perception enhancement data distillation and training, use Depth-AnythingV2 model to generate depth map, construct depth contrast training dataset, depth perception LoRA fine-tuning is carried out to the model that completes target detection training;The model trained can be deployed to mobile device, real-time obstacle recognition, depth understanding and generate voice navigation prompt.The application effectively solves the problem that end-side multi-modal large model performs poorly in blind road scene, can significantly improve guide blind accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, and natural language processing, and in particular to a method for optimizing tactile paving scenarios using multimodal data processing and model training techniques to achieve efficient and accurate training of a large-scale end-side guide model for the visually impaired. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models have demonstrated powerful capabilities in many fields. Multimodal Large Language Models (MLLMs) combine large language models (LLMs) and large visual models (LVMs), enabling them to process and understand various types of data such as text and images, and demonstrating outstanding performance in cross-modal tasks. However, current powerful multimodal large models are often around 200 bytes in size, making them difficult to implement on edge devices.

[0003] In applications involving tactile paving, existing multimodal large-scale models with parameter sizes ranging from 0.5B to 8B have significant shortcomings. In tactile paving scenarios, these models exhibit poor generalization ability for object recognition and localization, failing to accurately identify various obstacles or clearly understand scene depth information. For example, in complex tactile paving environments, the models cannot accurately identify and locate obstacles such as randomly parked electric vehicles or small stone blocks, resulting in a lack of reliable navigation information for visually impaired individuals. This is because such models lack extensive and high-quality data training specifically for tactile paving scenarios, making it difficult for them to learn the characteristics and patterns of objects within these environments.

[0004] Meanwhile, there is currently no large-scale dataset specifically designed for training end-to-end multimodal large models in tactile paving scenarios. This lack of vertical domain datasets makes training end-to-end multimodal large models in tactile paving applications extremely difficult, hindering the effective improvement of model accuracy and reliability in this scenario.

[0005] Invention content

[0006] To address the aforementioned issues, this invention proposes a multimodal end-side guidance model training method based on tactile paving scene data distillation. By co-distilling a generalized multimodal model and a depth estimation model, high-quality labeled tactile paving scene data is automatically generated, overcoming the bottleneck of missing vertical domain datasets. Combined with LoRA efficient fine-tuning technology, a lightweight end-side model is trained in multiple stages, ensuring model generalization ability while achieving accurate identification, localization, and depth perception of obstacles in tactile paving scenes, effectively improving guidance accuracy and meeting the application needs of end-side devices with limited resources.

[0007] This invention aims to address the problems of poor performance of existing end-side multimodal large models in tactile paving scenarios and the lack of dedicated datasets, and to achieve high-precision end-side guidance functions.

[0008] The objective of this invention is achieved as follows:

[0009] A multimodal end-side navigation model training method based on tactile paving scene data distillation is proposed. This method automatically generates high-quality labeled data for target detection and depth perception enhancement through co-distillation of a general-domain multimodal model (e.g., Seed-1.5-VL) and a depth estimation model (e.g., Depth-AnythingV2). The end-side multimodal model (e.g., Qwen2-VL-7B-Instruct) is efficiently fine-tuned in two stages using LoRA technology. First, the model is trained based on target detection distillation data to identify obstacles in tactile paving scenes. Then, the model is trained using depth perception distillation data for depth regression and distance judgment. The finally trained model can be deployed on mobile devices to perform obstacle recognition, depth understanding, and voice navigation prompts in tactile paving scenes, providing accurate path guidance for visually impaired users. The method specifically includes the following steps:

[0010] Step 1: Object detection data distillation and training, specifically including:

[0011] 1.1: Construct a dataset of images of tactile paving scenes, collecting real images of tactile paving scenes containing various obstacles and path environments;

[0012] 1.2: Design a structured object detection prompt template, preset prompt statements for the needs of tactile paving navigation, and require the multimodal large model to identify objects that obstruct walking and output the category name and bounding box coordinates in JSON format;

[0013] 1.3: Automatic annotation of tactile paving scene images is performed using the pervasive multimodal large model (Seed-1.5-VL) to generate detection labels containing object category and location information;

[0014] 1.4: Manually verify and correct the labeled data generated by distillation, and iteratively optimize the prompt template based on the labeling effect;

[0015] 1.5: LoRA fine-tuning training is performed based on the edge multimodal large model (Qwen2-VL-7B-Instruct) to enable the model to output structured object detection results;

[0016] Step 2: Depth-aware augmented data distillation and training, specifically including:

[0017] 2.1: Generate corresponding depth maps for tactile paving scene images using the depth estimation model (Depth-AnythingV2);

[0018] 2.2: Based on the bounding boxes of the detected objects, extract the average depth value of the central region of each object as the depth representation of the object;

[0019] 2.3: Construct a depth contrast training dataset, combining n objects in a single image into n×(n-1) / 2 depth contrast sample pairs;

[0020] 2.4: Based on the preset depth perception Prompt, the edge model that has completed object detection training is fine-tuned using LoRA for depth perception, so that it can regress the object depth and make distance comparison judgments.

[0021] Furthermore, the target detection data distillation specifically includes:

[0022] a1: The large-scale multimodal model in the general domain uses the Seed-1.5-VL model for image understanding and object detection annotation distillation;

[0023] a2: The output format is structured JSON data containing object category and location coordinates;

[0024] a3: Manually verify the image annotation data generated by distillation and establish an error correction mechanism; use the following formula to calculate automated annotations.

[0025] Error values ​​of the data:

[0026]

[0027] Where l represents the data number of the image, j represents the j-th object in the image, and k represents the category of the bounding box coordinates. The bounding box coordinates output by the model. The coordinates are manually labeled, and M is the number of objects detected in the image;

[0028] The depth-sensing enhanced data distillation specifically includes:

[0029] b1: Use the Depth-AnythingV2 model to extract pixel-level depth information from images of tactile paving scenes;

[0030] b2: Based on the bounding box of the object detection, extract the average depth value within a 7×7 pixel range of its central region as the depth feature of the object;

[0031] b3: For n objects in a single image, construct n×(n-1) / 2 depth contrast pairs to form training samples <(object A, depth value A), (object B, depth value B)>.

[0032] Furthermore, the efficient LoRA fine-tuning training based on the edge-side multimodal large model specifically includes:

[0033] 1) The basic model adopts Qwen2-VL-7B-Instruct as the end-side multimodal large model;

[0034] 2) First stage: LoRA fine-tuning is performed using object detection distillation data to train the model to output the category and location information of obstacles in the tactile paving scene. The loss function used is:

[0035] L det =αL cls +(1-α)L box

[0036] Among them, L cls For the cross-entropy loss of class prediction, L box The smoothed L1 loss is for bounding box regression, and α is the balancing parameter;

[0037] 3) Second Stage: Based on the training results of the first stage, a second LoRA fine-tuning is performed using depth-aware enhanced distillation data. The model is then trained to perform depth regression and distance comparison judgment. The loss function used to optimize the model is:

[0038] L total =L MSE +λ·L cls

[0039] Among them, L MSE For depth difference regression loss:

[0040]

[0041] The depth difference predicted by the model. Let S be the true depth difference, S be the set of sample pairs, and |S| be the number of sample pairs; L cls Loss classification based on proximity:

[0042]

[0043] For sample pairs Real Labels y k Defined as:

[0044]

[0045] p k For obj a Compared to obj b A closer probability, through Calculate, where σ is the sigmoid activation function. λ represents the object depth value predicted by the model; λ is a weighting parameter used to balance the importance of the two losses.

[0046] This invention proposes a multimodal end-side navigation model training method based on tactile paving scene data distillation, solving the problems of current multimodal large-scale models being difficult to apply at the end-side, small-scale models performing poorly in tactile paving scenes, and the lack of dedicated datasets. This method automatically generates target detection and depth perception enhancement training data through collaborative distillation of a generalized multimodal large-scale model (Seed-1.5-VL) and a depth estimation large-scale model (Depth-AnythingV2). After manual verification and optimization, the end-side multimodal large-scale model (Qwen2-VL-7B-Instruct) is fine-tuned in two stages using LoRA technology. The finally trained model can be deployed on high-performance mobile devices to achieve real-time obstacle recognition, depth understanding, and voice navigation in tactile paving scenes, significantly improving navigation accuracy. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the process of the present invention;

[0048] Figure 2 and Figure 3 A schematic diagram of the distillation results for target detection data;

[0049] Figure 4 and Figure 5 This is a schematic diagram of the distillation results of depth map data. Detailed Implementation

[0050] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0051] Example

[0052] See Figure 1 Perform two-stage data distillation and training of the multimodal guide vanishing model according to the following steps:

[0053] I. Target Detection Data Distillation and Model Training

[0054] The generation of target detection data based on large model data distillation and the training of edge models specifically include the following steps:

[0055] 1) Construction of the Target Detection Dataset for Tactile Pavement Scenes. A multi-source acquisition strategy was adopted to construct an image dataset for tactile pavement scenes. Images of tactile pavement scenes were acquired using portable high-definition cameras (resolution no less than 1920×1080), panoramic cameras, and other devices at different times (daytime, nighttime), under different weather conditions (sunny, rainy, foggy), and in different environments (city streets, parks, shopping malls, etc.). Let the dataset be... Where I iLet N represent the i-th image of the tactile paving scene, and N be the size of the dataset. In practice, N ≥ 10000 to ensure the richness of the dataset. After acquisition, the images are preprocessed, including but not limited to unifying the image size to a fixed size (e.g., 640×480 pixels) and removing noise, to meet the needs of subsequent model processing.

[0056] 2) Structured prompt template design

[0057] To address the needs of tactile paving navigation, a structured natural language prompt template was designed. The template uses an instruction-based text structure, with the core prompt being: "In this tactile paving environment, please identify objects that may obstruct walking and output all detected objects as a dictionary list in JSON format: [{"category": "obstacle_type", "bbox": " <bbox> x1y1 x2 y2< / bbox> "}, {"categories":"Obtacle_type","bbox":" <bbox> x1 y1 x2 y2< / bbox> To enhance template adaptability, in actual implementation, the template can be fine-tuned according to the characteristics of different scenarios. For example, when there are many small obstacles in the scene, the prompt statement can be supplemented with content such as "pay special attention to identifying obstacles of smaller size" to guide the multimodal large model in the general domain to generate more accurate labeled data.

[0058] 3) Automatic annotation of large-scale multimodal models in the common domain. The Seed-1.5-VL large-scale multimodal model proposed by ByteDance was selected as the automatic annotation engine. For each image in the dataset... Combine it with the prompt template Combine to form the input sequence The input is then fed into the Seed-1.5-VL model. Leveraging its powerful image understanding and natural language generation capabilities, the model processes the input sequence and outputs a list of object categories. with bounding box coordinates detection label L i Where M is the number of objects detected in the image, b j =[x j,min ,y j,min ,x j,max ,y j,max [] represents the bounding box coordinates of the j-th object. The output detection label L i Strictly adhering to JSON format specifications facilitates subsequent data storage, management, and model training. The generated results are available for reference. Figure 2 , Figure 3 .

[0059] 4) Manual verification and template optimization of labeled data. This involves manually verifying the labeled data L output by the model. iManual verification is performed. Professional annotators meticulously check the object categories and bounding box coordinates generated by the model based on the actual content of the image. To quantify annotation errors, the mean absolute error (MAE) of the bounding box coordinates is used. box The calculation formula for this evaluation indicator is as follows:

[0060]

[0061] in, The bounding box coordinates output by the model. These are manually labeled coordinates. (When MAE) box When the error exceeds a preset threshold (e.g., 0.05, which can be adjusted according to actual needs), the prompt template P is iteratively optimized based on the error analysis results. For example, the instruction description is modified, the output format requirements are adjusted, etc., and the accuracy and consistency of the annotation data are continuously improved through multiple optimizations.

[0062] 5) Efficient LoRA fine-tuning of end-side multimodal large models

[0063] Using the Qwen2-VL-7B-Instruct model as the baseline multimodal model, LoRA (Low-Rank Adaptation) technique is employed for efficient fine-tuning. During fine-tuning, the original model parameters θ are frozen, and a low-rank adapter parameter Δθ is introduced. The fine-tuned model parameters θ' can be expressed by the formula θ' = θ + A·Δθ·B. T The calculation shows that A and B are low-rank matrices. During training, the tactile paving scene image I is used. i Using preset prompts as input, and manually verified annotation data L... i To provide the monitoring signal, a loss function L is constructed. det Loss function L det Based on cross-entropy loss L cls With bounding box regression loss L box The formula is as follows:

[0064] L det =αL cls +(1-α)L box

[0065] Among them, L cls Cross-entropy loss for category prediction is used to measure the accuracy of the model in predicting object categories; L boxα is the smoothed L1 loss for bounding box regression, used to optimize the accuracy of the bounding box coordinates predicted by the model; α is a balancing parameter, whose optimal value can be determined experimentally in practice, typically ranging from 0.3 to 0.7. Based on freezing most parameters of the Qwen2-VL-7B-Instruct model, only the low-rank parameters embedded in key network layers are trained. This is combined with the AdamW optimization algorithm (initial learning rate 1e-4, β1 = 0.9, β2 = 0.999, epsilon = 1e-8, gradient clipping threshold 1.0), and a cosine annealing learning rate scheduling strategy (initial learning rate 1e-4, minimum learning rate 1e-6) is used to dynamically adjust the learning rate.

[0066] II. Deep Perception Enhancement Data Distillation and Training

[0067] The process of deep perception-enhanced data distillation and edge model training is as follows:

[0068] 1) Depth Map Generation: The Depth-AnythingV2 depth estimation model is used as the core processing unit to extract depth information from the constructed tactile paving scene image dataset. This model is based on the Transformer architecture and multi-scale feature fusion technology, and it can extract depth information from the input tactile paving scene image. (Where H is the image height, W is the image width, and 3 represents the RGB three channels) After that, the corresponding depth map is output through a multi-layer feature encoding and decoding module. In actual computation, the model weight parameter θ is optimized through pre-training on large-scale natural scene data, effectively capturing the spatial geometric relationships of obstacles, the ground, and other objects in tactile paving scenes, providing high-precision pixel-level depth data for subsequent depth analysis. The generated results can be found in the documentation. Figure 4 , Figure 5 .

[0069] 2) Object depth representation: Based on the boundingbox set B = {b1, b2, ..., b} output from the object detection stage. n} (where n is the number of detected objects, and each b) i Includes object category, top-left corner coordinates (x) min ,y min ), lower right corner coordinates (x max ,y max For each bounding box, depth features are extracted. Specifically, the bounding box center coordinates are used for depth feature extraction. Based on this, a 7×7 pixel central region is defined. The average pixel depth within this region is calculated using the following formula, which serves as the depth feature of the corresponding object:

[0070]

[0071] in, d represents the depth value of the i-th object. j Let be the depth value of the j-th pixel within the 7×7 region in the depth map D. This method utilizes the statistical characteristics of local regions to effectively filter out abnormal depth values ​​caused by factors such as sensor noise and textured similar regions, thereby improving the stability and reliability of object depth representation.

[0072] 3) Construction of the depth contrast training dataset: For n objects detected in a single image of a tactile paving scene, a set S of depth contrast sample pairs is constructed through permutations and combinations based on the principles of combinatorics. The formula for calculating the number of sample pairs |S| is:

[0073]

[0074] Each sample pair s k ∈S consists of two distinct objects and their depth values, in the form of Among them obj a obj b For different object categories, These represent the depth values ​​of the corresponding objects. In the actual construction process, a parallel computing strategy is employed, using GPU acceleration to efficiently generate large-scale sample pairs, providing rich comparative learning data for depth perception training.

[0075] 4) Edge Model Depth Perception Fine-tuning: Based on the Qwen2-VL edge model trained for object detection, LoRA (Low-Rank Adaptation) technology is used to optimize the parameters for the depth perception task. The preset depth perception prompt example is: Given a tactile paving (blind path) scene image containing detected objects, analyze the depth relationships between all object pairs. For each pair of objects in the scene, determine their relative depth positions and calculate the depth difference. Input format: You will receive an object detection result (JSON format) containing the object category and bounding box, as well as the depth information for each object. The output depth comparison results (JSON format) are as follows: [{"Object Pair":["Object A Category","Object B Category"],"Object A Depth":Depth Value A,"Object B Depth":Depth Value B,"Depth Difference":|Depth Value A - Depth Value B|,"Closer Object":"Object A Category" or "Object B Category","Depth Relationship":"Object A is closer / farther than Object B"},{"Object Pair":["Object C Category","Object D Category"],"Object C Depth":Depth Value C,"Object D Depth":Depth Value D,"Depth Difference":|Depth Value C - Depth Value D|,"Closer Object":"Object C Category" or "Object D Category","Depth Relationship":"Object C is closer / farther than Object D"}] The key is to understand the spatial depth relationships between obstacles and their relative distances to the camera / observer's viewpoint to help visually impaired people navigate. During training, depth comparison sample pairs are used as input, and the model calculates the predicted depth difference through forward propagation. The loss function is the mean squared error (MSE) of the difference between the depth and the true depth ΔD. A new binary cross-entropy loss function is also added to determine the distance relationship between objects. The total loss function is as follows:

[0076] L total =L MSE +λ·L cls

[0077] Among them, L MSE For depth difference regression loss:

[0078]

[0079] The depth difference predicted by the model. This represents the actual depth difference. L cls Loss classification based on proximity:

[0080]

[0081] For sample pairs Real Labels y k Defined as:

[0082]

[0083] model predicted p k For obj a Compared to obj b A closer probability, through Calculate, where σ is the sigmoid activation function. λ represents the object depth value predicted by the model. λ is a weight parameter used to balance the importance of the two losses. The LoRA module parameters are updated with a learning rate η using backpropagation, while keeping the core parameters of the base model frozen. Key LoRA parameters are set as follows: rank = 16 to control the dimension of the low-rank matrix, alpha = 32 to balance fine-tuning weights, and dropout = 0.1 to prevent overfitting. This ensures efficient adaptation to depth perception tasks in tactile paving scenarios while maintaining model generalization ability, enabling the model to accurately regress object depth and perform distance comparison judgments. During training, the specific settings are the same as for the object detection task: most parameters of the Qwen2-VL-7B-Instruct model are frozen, and only the low-rank parameters embedded in key network layers are trained. The AdamW optimization algorithm (initial learning rate 1e-4, β1 = 0.9, β2 = 0.999, epsilon = 1e-8, gradient clipping threshold 1.0) is used, and a cosine annealing learning rate scheduling strategy (initial learning rate 1e-4, minimum learning rate 1e-6) is used to dynamically adjust the learning rate.

[0084] III. Deployment of a large-scale end-side guide model

[0085] 1) Hardware deployment environment

[0086] The device utilizes a portable mobile device (such as a processing terminal for guide glasses or a smartphone) equipped with a Snapdragon 8 Gen3 processor. This hardware configuration supports real-time inference of 8B parameter-level models, meeting the requirement of low latency (≤200ms) on the edge. The device integrates a high-definition RGB camera (1080P resolution, 30fps) to capture images of the tactile paving scene, and a built-in bone conduction headphone module to provide voice prompts, ensuring that visually impaired users can clearly receive navigation information in noisy environments.

[0087] 2) Model Deployment Optimization

[0088] The Qwen2-VL-7B-Instruct edge model, which underwent two-stage LoRA fine-tuning, was quantized (using 4-bit quantization). While maintaining an accuracy loss of ≤3%, the model size was compressed to 1 / 4 of its original size, reducing memory usage to under 6GB. Inference was accelerated using the TensorRT framework, the computation graph structure was optimized, and operator fusion was enabled, increasing image processing throughput to 30 frames per second to match the camera's frame rate.

[0089] 3) Real-time inference process

[0090] Image acquisition: The device's camera captures images of the tactile paving scene in real time. Each frame of the image is preprocessed (size adjusted to 640×480 pixels, normalized to the range of [0,1]) and then input into the model.

[0091] Obstacle detection: The model receives an image and a preset target detection prompt, and outputs the target detection results in JSON format;

[0092] Depth determination: Based on the detected obstacle bounding boxes and the depth perception prompt, the model predicts the relative depth relationship between objects (e.g., "the stone block is closer than the bicycle") and outputs the absolute depth value of the key objects (error range ±0.1 meters).

[0093] Navigation decision: Based on the location and depth information of obstacles, the system generates path guidance instructions (such as "There is a stone block 1.2 meters ahead, it is recommended to detour to the left by 0.5 meters"), which are then converted into concise speech text by the natural language processing module;

[0094] Voice output: The bone conduction headphones play voice prompts at 1.5 times the speech speed and update navigation information every 500ms to ensure that users are aware of the dynamic environment in real time.

[0095] This invention proposes a multimodal end-side navigation model training method based on tactile paving scene data distillation. By co-distilling a permeable multimodal model and a depth estimation model, target detection and depth perception labeled data are generated. Combined with LoRA for efficient fine-tuning training of the end-side model, the method can ultimately achieve obstacle recognition, depth understanding and voice navigation in tactile paving scenes, thereby improving the accuracy of navigation.

[0096] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for training a multimodal end-side guidance system large model based on tactile paving scene data distillation, characterized in that, By co-distilling a large-scale multimodal model and a large-scale depth estimation model, labeled data for tactile paving scenes is automatically generated for training an end-side guidance model. The specific steps include: Step 1: Object detection data distillation and training, specifically including: 1.1: Construct a dataset of images of tactile paving scenes, collecting real images of tactile paving scenes containing various obstacles and path environments; 1.2: Design a structured object detection prompt template, preset prompt statements for the needs of tactile paving navigation, and require the multimodal large model to identify objects that obstruct walking and output the category name and bounding box coordinates in JSON format; 1.3: Use a multimodal large model to automatically annotate images of tactile paving scenes, generating detection labels that include object category and location information; 1.4: Manually verify and correct the labeled data generated by distillation, and iteratively optimize the prompt template based on the labeling effect; 1.5: LoRA finger fine-tuning training based on edge multimodal large model enables the model to output structured object detection results; Step 2: Depth-aware augmented data distillation and training, specifically including: 2.1: Generate corresponding depth maps for tactile paving scene images using a depth estimation model; 2.2: Based on the bounding boxes of the detected objects, extract the average depth value of the central region of each object as the depth representation of the object; 2.3: Construct a depth contrast training dataset, combining n objects in a single image into n×(n-1) / 2 depth contrast sample pairs; 2.4: Based on the preset depth perception prompt, the edge model that has completed object detection training is fine-tuned using LoRA for depth perception, enabling it to regress object depth and perform distance comparison judgment; where: The efficient LoRA fine-tuning training based on the edge-side multimodal large model specifically includes: 1) The basic model adopts Qwen2-VL-7B-Instruct as the end-side multimodal large model; 2) First stage: LoRA fine-tuning is performed using object detection distillation data to train the model to output the category and location information of obstacles in the tactile paving scene. The loss function used is: L det =αL cls +(1-α)L box ; in, Cross-entropy loss for class prediction, For bounding box regression, a smoothed L1 loss is used. For balance parameters; 3) Second Stage: Based on the training results of the first stage, a second LoRA fine-tuning is performed using depth-aware enhanced distillation data. The model is then trained to perform depth regression and distance comparison judgment. The loss function used to optimize the model is: L total L MSE +λ·L cls in, Cross-entropy loss for class prediction, For bounding box regression, a smoothed L1 loss is used. For balance parameters; The depth difference predicted by the model. Let S be the true depth difference, S be the set of sample pairs, and |S| be the number of sample pairs. Loss classification based on proximity: For sample pairs Authentic Labels Defined as: for Compare A closer probability, through Calculation, where The sigmoid activation function is used. , The object depth value predicted by the model; is a weighting parameter used to balance the importance of the two losses.

2. The multimodal end-side guide vanity large model training method according to claim 1, characterized in that, The target detection data distillation specifically includes: a1: The large-scale multimodal model in the general domain uses the Seed-1.5-VL model for image understanding and object detection annotation distillation; a2: The output format is structured JSON data containing object category and location coordinates; a3: Manually verify the image annotation data generated by distillation and establish an error correction mechanism; use the following formula to calculate the error value of the automated annotation data: in, This indicates the data number of the image. Indicates the first in the image One object, The category representing the bounding box coordinates, The bounding box coordinates output by the model. The coordinates are manually labeled, and M is the number of objects detected in the image; The depth-sensing enhanced data distillation specifically includes: b1: Use the Depth-AnythingV2 model to extract pixel-level depth information from images of tactile paving scenes; b2: Based on the bounding box of the object detection, extract the average depth value within a 7×7 pixel range of its central region as the depth feature of the object; b3: For n objects in a single image, construct n×(n-1) / 2 depth contrast pairs to form training samples <(object A, depth value A), (object B, depth value B)>.

Images