A multi-path recall method and device for automatic driving data

Abstract

The application provides a multi-path recall method and device for automatic driving data, wherein the method comprises the following steps: first, receiving multi-source driving data returned by a vehicle end according to a sample confidence degree and a confidence threshold matching relationship; performing rule pre-screening on the received data in a cloud end to obtain a candidate data subset; inputting the candidate data subset into a coarse recall model, a full-amount fine-tuning fine recall model and a multi-modal retrieval recall model in parallel for synchronous processing, and outputting multi-path recall results; and fusing the multi-path recall results and classifying to construct a multi-class training data set, which can be used to train at least one of an automatic driving algorithm model, a vehicle-mounted lightweight model, a coarse recall model or a full-amount fine-tuning fine recall model. That is, the bandwidth consumption is reduced by the hierarchical return of the vehicle end, the scene is fully covered by relying on the three-way parallel recall of the cloud end, the training set is constructed in layers to continuously iterate various models, and the data recall coverage, recall accuracy and cloud processing cost are considered.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and autonomous driving technology, and in particular relates to a multi-path recall method and apparatus for autonomous driving data. Background Technology

[0002] The perception, prediction, and planning capabilities of autonomous driving systems are highly dependent on large-scale, high-quality training data. As the size of autonomous driving fleets continues to expand, the amount of raw data collected by vehicles daily is growing exponentially. This necessitates efficiently filtering data from massive amounts of data to identify those relevant to the target scenario, target category, or areas where the model is weak, and constructing high-quality training sets.

[0003] Currently, the main methods for data screening are: 1) rule-based data screening; 2) recall based on manual annotation and active learning; 3) similarity retrieval based on image feature embedding; and 4) labeling and recall based on fine-tuning of a single model.

[0004] However, the coverage of a single recall method is limited. Whether it is rule-based recall, similarity retrieval, or single model labeling, it is difficult to fully cover the diverse and complex scenarios in autonomous driving. When it is necessary to build recall capabilities for new categories or new scenarios, if the sample size is extremely small, the model cannot be effectively trained by fine-tuning the entire sample, while pure rule-based methods are difficult to accurately describe complex semantics, resulting in difficulties in cold start.

[0005] There is currently no effective solution for how to efficiently, cost-effectively, and accurately retrieve tiered data from autonomous driving systems. Summary of the Invention

[0006] The purpose of this application is to provide a multi-path recall method and apparatus for autonomous driving data, which can achieve efficient, low-cost, and high-precision hierarchical recall of raw autonomous driving data.

[0007] This application provides a multi-path recall method and apparatus for autonomous driving data, which is implemented as follows: A multi-path recall method for autonomous driving data, the method comprising: The receiving vehicle receives multi-source data that is graded and transmitted back based on the relationship between sample confidence and confidence threshold; The cloud performs pre-screening based on rules on the received multi-source data to obtain a subset of candidate data. A subset of candidate data is fed in parallel into a coarse recall model, a full-scale fine-tuning recall model, and a multimodal retrieval recall model for processing to obtain multi-path recall results; The multi-path recall results are fused and classified to construct a multi-class training dataset, which is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full fine-tuning recall model.

[0008] In one implementation, the multi-source data that the vehicle-mounted terminal transmits in a tiered manner based on the relationship between sample confidence and confidence threshold includes: The vehicle's multi-source sensors collect real-time driving images and vehicle body data as multi-source data; A lightweight model, obtained by distilling a large model from the cloud, is deployed locally on the vehicle for real-time inference, outputting the label prediction result and corresponding sample confidence for each data point. Based on preset first and second thresholds, a three-stage tiered processing of confidence scores is performed: data with confidence scores higher than the first threshold are locally labeled, and only the label prediction results are uploaded; data with confidence scores between the second and first thresholds are asynchronously uploaded to the cloud for review; data with confidence scores lower than the second threshold are identified as weak samples in the model, and the original data is immediately uploaded first. In one implementation, a subset of candidate data is fed in parallel into a coarse recall model, a full-scale fine-tuning recall model, and a multimodal retrieval recall model for processing to obtain multi-path recall results, including: Valid samples are obtained through a coarse recall model; High-confidence pseudo-label samples were obtained by fine-tuning the full-scale recall model. Highly semantically relevant samples are obtained through a multimodal retrieval and recall model; The valid samples, high-confidence pseudo-labeled samples, and highly semantically relevant samples are sent to the multi-path recall fusion layer for aggregation processing. The sample source branches and confidence scores are uniformly labeled, sample identification and deduplication are completed, duplicate samples are merged and the highest comprehensive confidence record is retained, and the integrated multi-path recall results are output.

[0009] In one implementation, effective samples are obtained through a coarse recall model, and high-confidence pseudo-label samples are obtained through a full-scale fine-tuning recall model, including: A subset of candidate data is simultaneously fed into the coarse recall model, and the full-scale fine-tuning of the fine recall model is performed in parallel inference. Collect suspected relevant samples from the coarse recall model, and remove invalid samples that are falsely recalled through manual review or cloud-based high-precision model verification, while retaining valid samples that truly match the target scenario to form a reusable incremental labeled sample pool. The reusable incremental labeled sample pool is added to the full fine-tuning training set to supplement the sample quantity required for training the full fine-tuning fine-recall model. After the full fine-tuning of the fine recall model is completed, high-confidence pseudo-label samples are obtained, and the coarse recall training set is expanded using the high-confidence pseudo-label samples. The valid samples and the high-confidence pseudo-label samples are sent to the multi-path recall fusion layer for aggregation processing.

[0010] In one implementation, highly semantically relevant samples are obtained through a multimodal retrieval recall model, including: The multimodal retrieval and recall model extracts image embeddings from a subset of candidate data and calculates similarity. For differentiated images with similarity below a threshold, the visual language model is triggered; the visual language model generates structured long-text scenes containing lighting, roads, traffic participants, vehicle status, and right-of-way relationships according to the autonomous driving-specific dimensions; Differentiated images and generated scene text are input into the same multimodal embedding model and mapped to a shared vector space to generate image vectors and text vectors respectively, which are then stored in a vector library. At the same time, driving keywords are extracted from the long text through word segmentation, and a BM25 inverted text index is built to form two retrieval systems: vector retrieval and keyword retrieval. During the recall phase, the query text and images are converted into query vectors, and vector retrieval and keyword retrieval are performed simultaneously. The two-way retrieval scores are merged and a semantic conflict penalty term is introduced to filter samples that are similar in appearance but irrelevant in driving logic. After sorting, highly semantically relevant samples are output and sent to the multi-way recall fusion layer for aggregation.

[0011] In one implementation, merging duplicate samples and retaining the record with the highest overall confidence score includes: Combine duplicate samples using the following formula and calculate the overall confidence level of each sample:

[0012] in, To assess the overall confidence level, Dynamic weights for rule-based recall. For coarse recall dynamic weights, To fine-tune the dynamic weights for full-scale recall, Fixed base weights for multimodal retrieval recall. To retrieve the matching score for the rule recall, For the coarse recall matching score, To fine-tune the full-scale recall matching score, For multimodal retrieval recall matching score, The preset hyperparameters are used, where, when the sample size is sufficient, the improvement is... The value of .

[0013] In one implementation, the multimodal retrieval recall matching score is calculated according to the following formula:

[0014] in, The cosine similarity is used to calculate the difference between the image and the generated scene long text multimodal vector. The score for keyword retrieval in driving scenarios. The penalty term for semantic conflicts in driving is increased when there are conflicts in right-of-way, conflicting scene types, or conflicting driving states of the main vehicle. This is the balance adjustment coefficient.

[0015] A multi-path recall device for autonomous driving data includes: The receiving module is used to receive multi-source data returned by the vehicle and perform rule-based pre-screening to obtain a subset of candidate data. The vehicle obtains the sample confidence of the multi-source data by distilling the large model in the cloud through local vehicle deployment and obtaining a lightweight model. Based on the relationship between the sample confidence and the confidence threshold, the multi-source data is returned to the cloud in a hierarchical manner. The fusion recall module is used to feed a subset of candidate data into the coarse recall model, the full fine-tuning recall model, and the multimodal retrieval recall model in parallel for processing, so as to obtain multi-path recall results; The training set generation module is used to fuse and classify the multi-path recall results to construct a multi-class training dataset, wherein the multi-class training dataset is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full-scale fine-tuning fine recall model.

[0016] An electronic device includes a processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method described above.

[0017] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0018] The multi-path recall method for autonomous driving data provided in this application reduces the bandwidth requirements for uploading raw data from the vehicle and the pressure on cloud storage by tiering data transmission based on confidence thresholds and differentiating between high, medium, and low confidence samples. Pre-screening based on rules completes the first layer of data compression, reducing the computational overhead of subsequent model processing. Coarse recall, full-scale fine-tuning recall, and multimodal retrieval are processed in parallel and collaboratively. The complementarity between models covers cold-start long-tail scenarios, high-precision mature scenarios, and high-level driving semantic scenarios, thereby simultaneously improving recall coverage and accuracy. The fusion of multi-path results produces multiple training datasets, which can simultaneously supply autonomous driving business algorithms, lightweight vehicle models, and two types of recall models for iterative training, forming a self-driven data closed loop and continuously optimizing autonomous driving scene recognition and data mining capabilities. Through these methods, efficient, low-cost, and high-precision tiered recall of raw autonomous driving data can be achieved, thereby improving the accuracy of autonomous driving models. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of one embodiment of the multi-path recall method for autonomous driving data provided in this application; Figure 2 This is the cloud-based funnel-style multi-path recall flowchart provided in this application; Figure 3 This is a flowchart of the multimodal embedding retrieval and recall process provided in this application; Figure 4 This is a diagram illustrating the synergistic relationship between LoRA fine-tuning recall and full fine-tuning recall provided in this application; Figure 5 This is a flowchart of the confidence grading process for the vehicle-side distillation model provided in this application; Figure 6 This is a flowchart of the cloud-vehicle data closed-loop iteration process provided in this application; Figure 7 This is a hardware structure block diagram of an electronic device for a multi-path recall method for autonomous driving data provided in this application; Figure 8 This is a schematic diagram of the module structure of one embodiment of the multi-path recall device for autonomous driving data provided in this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0022] It should be noted that the information and data related to users involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by the relevant parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws, regulations, and standards, and necessary confidentiality measures have been taken. They do not violate public order and good morals, and corresponding operation entry points are provided for users or relevant parties to choose to authorize or refuse.

[0023] It should also be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0024] Figure 1 This is a flowchart of one embodiment of the multi-path recall method for autonomous driving data provided in this application. Although this application provides method operation steps or device structures as shown in the following embodiments or figures, more or fewer operation steps or module units may be included in the method or device based on conventional or non-inventive effort. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments and figures of this application. When the method or module structure is applied in actual devices or terminal products, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or figures (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0025] Specifically, such as Figure 1 As shown, the above-mentioned multi-path recall method for autonomous driving data may include the following steps: Step 101: Receive multi-source data from the vehicle terminal based on the hierarchical relationship between sample confidence and confidence threshold; In one example, the multi-source data that the vehicle-side transmits in a tiered manner based on the relationship between sample confidence and confidence threshold may include: S1: The vehicle's multi-source sensors collect real-time driving images and vehicle body data as multi-source data; S2: Real-time inference is performed using a lightweight model obtained by distilling a large model from the cloud and deployed locally on the vehicle, to output the label prediction result and the corresponding sample confidence for each data point; S3: Based on the preset first and second thresholds, the confidence level is processed in a three-stage classification: data with a confidence level higher than the first threshold are locally labeled and only the label prediction results are uploaded; data with a confidence level between the second and first thresholds are asynchronously uploaded to the cloud for review; data with a confidence level lower than the second threshold are identified as weak samples in the model and the original data is uploaded immediately.

[0026] Step 102: The cloud performs pre-filtering of the received multi-source data according to rules to obtain a subset of candidate data; Step 103: Feed the candidate data subsets into the coarse recall model, the full fine-tuning recall model, and the multimodal retrieval recall model in parallel for processing to obtain multi-path recall results; Step 104: The multi-path recall results are fused and classified to construct a multi-class training dataset, wherein the multi-class training dataset is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full fine-tuning fine recall model.

[0027] The above-mentioned process of feeding a subset of candidate data into a coarse recall model, a full-scale fine-tuning recall model, and a multimodal retrieval recall model in parallel to obtain multi-path recall results can include: obtaining effective samples through the coarse recall model; obtaining high-confidence pseudo-label samples through the full-scale fine-tuning recall model; obtaining highly semantically relevant samples through the multimodal retrieval recall model; feeding the effective samples, high-confidence pseudo-label samples, and highly semantically relevant samples into a multi-path recall fusion layer for aggregation processing, uniformly labeling the sample source branch and confidence score, completing sample identification deduplication, merging duplicate samples and retaining the record with the highest comprehensive confidence score, and outputting the integrated multi-path recall results.

[0028] Specifically, obtaining effective samples through a coarse recall model and obtaining high-confidence pseudo-label samples through full-scale fine-tuning of the recall model can include: S1: Simultaneously feed a subset of candidate data into the coarse recall model and perform full-scale fine-tuning of the fine recall model for parallel inference; S2: Collect suspected relevant samples from the coarse recall model, and remove invalid samples that are falsely recalled through manual review or cloud-based high-precision model verification, while retaining valid samples that truly match the target scenario to form a reusable incremental labeled sample pool. S3: Supplement the reusable incremental labeled sample pool into the full fine-tuning training set to make up for the amount of samples required for training the full fine-tuning fine-recall model. S4: After the full fine-tuning of the fine recall model is completed, high-confidence pseudo-label samples are obtained, and the coarse recall training set is expanded through the high-confidence pseudo-label samples; S5: Send the valid samples and the high-confidence pseudo-label samples into the multi-path recall fusion layer for aggregation processing.

[0029] In other words, effective samples are generated through inference using a coarse-grained recall model (e.g., a LoRA fine-tuning model): a subset of candidate data, filtered by pre-defined rules, is input into the trained LoRA coarse-grained recall model; this model only fine-tunes a small number of low-rank adaptation parameters to adapt to cold-start, long-tail, and low-sample scenarios. Then, the model infers frame by frame, outputting the confidence score of each data point matching the target scenario. A basic low-confidence threshold is set to filter out completely irrelevant samples, retaining data with the required confidence level and suspected matching the target scenario as effective coarse-grained recall samples. These samples can cover a large number of long-tail, scarce scenarios, but there is a certain amount of false recall; they are only used as incremental material to be verified.

[0030] Full-scale fine-tuning of the refined recall model generates high-confidence pseudo-label samples: The same batch of candidate data is simultaneously fed into the full-scale fine-tuning recall model. This model completes all parameter training based on a sufficient number of labeled samples, resulting in higher recognition accuracy. The model inference outputs scene classification results and corresponding confidence scores, with a confidence screening threshold higher than that of coarse recall. Only samples with confidence scores exceeding the high standard threshold are retained, and scene category labels are automatically generated, resulting in high-confidence pseudo-label samples. These samples have fewer false recalls and higher label credibility, and can be directly used for model training, thus significantly reducing the workload of manual review.

[0031] Two types of samples are reused in a collaborative manner to form a two-way iterative support: after the effective samples of coarse recall are verified by manual or high-precision models, they are added to the training set of the full fine-tuning model, which can solve the problem of insufficient samples in the cold start stage; the high-confidence pseudo-label samples produced by full fine-tuning are fed back to expand the training data of the coarse recall model, which can continuously reduce the false recall rate of coarse recall and realize the mutual optimization of coarse and fine models.

[0032] In one example, obtaining highly semantically relevant samples through a multimodal retrieval recall model can include: S1: The multimodal retrieval and recall model extracts image embeddings from a subset of candidate data and calculates similarity. S2: Trigger the visual language model for differential images with similarity below a threshold; The visual language model generates structured long-text scenes containing lighting, roads, traffic participants, vehicle status, and right-of-way relationships according to the autonomous driving-specific dimensions; S3: Input the differentiated images and the generated scene long text into the same multimodal embedding model, map them to the shared vector space, generate image vectors and text vectors respectively and store them in the vector library; at the same time, segment the long text to extract driving keywords, build the BM25 inverted text index, and form two retrieval systems: vector retrieval and keyword retrieval. S4: During the recall phase, convert the query text and images into query vectors, and simultaneously perform vector retrieval and keyword retrieval; S5: Combine the scores from two retrieval paths and introduce a semantic conflict penalty term to filter samples that are similar in appearance but irrelevant to driving logic. After sorting, output highly semantically relevant samples and send them to the multi-path recall fusion layer for aggregation.

[0033] That is, the VLM visual language model can be triggered on demand to generate structured long text of driving scene, and the image and scene long text are encoded separately and mapped to the same multimodal vector space. Two sets of retrieval libraries, vector retrieval index and BM25 text inverted index, are built simultaneously. Then, the vector similarity retrieval and keyword BM25 retrieval are fused and scored to achieve accurate recall of high-level driving semantics.

[0034] For example, it may include the following steps: Step 1: Image similarity pre-screening enables on-demand generation of long texts to address the issue of excessive computational power in full VLM; Input: The set of candidate images after rule-based pre-recall filtering; First, a visual encoder is used to generate a base image embedding vector for each image; Calculate the cosine similarity of each pair of image vectors and set a similarity threshold: If the similarity between two images is greater than the threshold, they are judged as nearly duplicate images. The VLM long text generation is skipped, and the text vectors of similar images that have already been generated are directly reused to avoid repeatedly calling the large model. If the similarity is less than or equal to the threshold, it is determined to be a completely new and differentiated scenario, and only then will VLM be activated to generate a complete long text of driving semantics. That is, VLM is not used for all images, and high-cost text generation is only performed on differentiated images, which can significantly reduce the computing power overhead in the cloud.

[0035] Step 2: Visual Language Model (VLM) generates multi-dimensional driving semantic long text: By invoking VLM and combining it with autonomous driving-specific prompts, structured long text is generated, which contains multiple types of high-level semantics specific to driving: Ambient lighting dimensions: weather (sunny / rainy / foggy), lighting (backlight / night / dusk), visibility; Road surface dimensions: road type (highway / intersection / non-motorized vehicle lane), number of lanes, water accumulation / damage on the road surface, and the condition of road markings; Traffic participant dimension: number and location of motor vehicles, pedestrians, non-motorized vehicles, and obstacles; Main vehicle driving context: current speed, straight / turning / lane changing, braking / acceleration status; Spatial interaction relationships: target occlusion, interception, close-range conflict, passing / crossing, and other interactive behaviors; Semantic dimensions of right-of-way: pedestrian priority, vehicle straight-through priority, conflicting right-of-way scenarios.

[0036] The final output is a complete, natural, long text that integrates all dimensions, fully conveying the driving logic hidden within the image, rather than just describing the objects in the picture.

[0037] Step 3: Unify multimodal vectorization of images and long text, and construct a cross-modal vector index: Dual-branch encoding: Image branch: The original image is input into the multimodal embedding model, and the output is a unified image vector; Text branch: The long driving text generated by VLM is input into the same multimodal embedding model, and the output is a text vector. Spatial alignment: Image vectors and text vectors are mapped to the same high-dimensional vector space, ensuring that images and text can be retrieved from each other: inputting a text can find the corresponding image, and inputting an image can find other images with similar meanings; Vector input: Image vectors and text vectors are stored in a vector database to build an approximate nearest neighbor retrieval index for semantic similarity matching.

[0038] Step 4: Build a BM25 keyword inverted index from long text segmentation: The long driving text generated by VLM is segmented and stop words are removed to extract driving-specific keywords such as pedestrians crossing the road at intersections, backlighting, waterlogged roads, and non-motorized vehicles going against the flow of traffic. A BM25 inverted index is constructed based on the word segmentation results, recording all sample IDs corresponding to each keyword; Leveraging the BM25's strength in accurately matching specific keywords and scene categories, it can solve the problem of vector retrieval encountering images that are similar but semantically unrelated to driving.

[0039] Step 5: Integrate vector retrieval and BM25 retrieval to achieve precise recall of driving semantics: Recall phase: User inputs target scene description / keywords / reference image: The query content (text / reference image) is encoded into a query vector, and vector similarity retrieval is performed to obtain a list of semantically similar samples; Perform a BM25 keyword search on the query keywords to obtain a list of exact match samples; The two sets of search results are weighted, merged, and reordered, while introducing an innovative semantic conflict penalty term for driving in this scheme: If sample vectors are similar, but there are contradictions in the right-of-way, vehicle status, and scene type in the text, the overall score of the sample will be reduced, and invalid recall results that are similar in appearance and completely unrelated to driving logic will be filtered out. The sorted highly relevant samples are output as the recall results of the multimodal embedding retrieval branch and sent to the multi-path recall fusion layer for unified processing.

[0040] In one example, merging duplicate samples and retaining the record with the highest overall confidence score could include: Combine duplicate samples using the following formula and calculate the overall confidence level of each sample:

[0041] in, To assess the overall confidence level, Dynamic weights for rule-based recall. For coarse recall dynamic weights, To fine-tune the dynamic weights for full-scale recall, Fixed base weights for multimodal retrieval recall. To retrieve the matching score for the rule recall, For the coarse recall matching score, To fine-tune the full-scale recall matching score, For multimodal retrieval recall matching score, The preset hyperparameters are used, where, when the sample size is sufficient, the improvement is... The value of .

[0042] The multimodal retrieval recall matching score can be calculated using the following formula:

[0043] in, The cosine similarity is used to calculate the difference between the image and the generated scene long text multimodal vector. The score for keyword retrieval in driving scenarios. The penalty term for semantic conflicts in driving is increased when there are conflicts in right-of-way, conflicting scene types, or conflicting driving states of the main vehicle. This is the balance adjustment coefficient.

[0044] That is, invalid samples that are similar in appearance but have no relation to driving semantics are filtered out by a semantic conflict penalty term.

[0045] The above method will be described below with reference to a specific embodiment. However, it should be noted that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0046] This example presents a multi-path data retrieval method for autonomous driving scenarios. It enables efficient, low-cost, and high-precision hierarchical retrieval of raw autonomous driving data based on rule-based filtering, lightweight fine-tuning of the model for coarse retrieval, multimodal embedding retrieval, full-scale fine-tuning for high-precision retrieval, and vehicle-side confidence-based hierarchical data transmission. The retrieved data can provide high-quality, continuous training data for downstream autonomous driving algorithms such as end-to-end driving models and VLA (Vision-Language-Action) large models, while also supporting iterative updates of the labeled model, forming a complete data closed-loop mechanism.

[0047] The method described in this example can solve at least the following problems: 1) Multi-path collaborative recall problem: How to organically integrate rule-based recall, model-based recall, and multimodal retrieval recall to construct a hierarchical multi-path recall system with high coverage and controllable accuracy, overcoming the shortcomings of insufficient coverage of single recall methods. 2) Recall problem under cold start conditions: When the sample size of the target category or target scene is extremely small, how to utilize LoRA fine-tuning to quickly build a lightweight model with initial recall capabilities to support data discovery in the cold start phase, and gradually transition to high-precision recall through full-scale fine-tuning as the sample size accumulates. 3) Retrievable representation of complex semantics in driving scenarios: How to convert the high-level driving semantics implicit in images (including ambient lighting conditions, road structure, traffic participant interaction relationships, vehicle motion context, key spatial relationships, and right-of-way analysis) into a unified representation that is indexable, searchable, and quantifiable, to support accurate recall for complex semantics. 4) Resource consumption control for multimodal processing: How to avoid high-cost long text generation and multimodal vectorization of all images while ensuring retrieval effectiveness? This can be achieved by using an image similarity pre-screening mechanism to trigger long text descriptions on demand, reducing overall resource consumption. 5) Efficiency and cost balance in cloud-based and vehicle-based collaborative retrieval: How to utilize the distillation model deployed on the vehicle to pre-screen and pre-label data, and determine the data processing method based on confidence levels, thereby reducing the cost of uploading invalid data to the cloud while promptly identifying and returning weak samples to the model. 6) Continuous iteration of the data closed loop: How to construct a complete closed loop driven by data retrieval, supported by model training, and aimed at using training results to improve retrieval, ensuring that cloud-based and vehicle-based models continuously improve their retrieval capabilities as data accumulates.

[0048] Specifically, this example presents a multi-path recall method for closed-loop autonomous driving data, comprising a cloud-based recall link and a vehicle-based recall link. These two links work together through label data, model parameters, confidence information, and training feedback to form a complete closed-loop multi-path recall system for autonomous driving data. The cloud-based recall link and the vehicle-based recall link are explained in detail below: Cloud-based recall chain such as Figure 2 As shown, a funnel-shaped hierarchical processing architecture is adopted, including: The first layer: rule-based pre-recall, which uses the output results of sensor and algorithm modules to quickly filter and narrow down the range of candidate data; The second layer (parallel recall) is used for categories with small sample sizes or cold start. A recall model is quickly built through lightweight low-rank adaptation fine-tuning to perform coarse recall. The LoRA fine-tuning recall results can supplement the full fine-tuning training set to form progressive enhancement.

[0049] In practice, LoRA fine-tuning recall and full fine-tuning recall run in parallel, serving recall needs of different categories or different stages respectively; the results of LoRA fine-tuning recall can further supplement the training set of full fine-tuning.

[0050] Furthermore, multimodal embedding retrieval recall can be configured, such as... Figure 3 As shown, a visual language model is used to generate long-form text for a scene as needed. Images and long-form text are then uniformly vectorized into multimodal vectors, and combined with BM25 retrieval, accurate recall based on driving semantics is achieved.

[0051] In implementation, multimodal embedding retrieval, LoRA fine-tuning retrieval, and full fine-tuning retrieval can be run in parallel. Long-text scenes are generated through a visual language model and combined with images for multimodal vectorization. Semantic-level fine-grained retrieval is achieved by combining BM25 retrieval.

[0052] The third layer is the result fusion layer, which aggregates, deduplicates, filters confidence scores, and constructs training sets for the recall results from the above channels.

[0053] The vehicle-side recall link is deployed locally in the vehicle to obtain a lightweight model by distilling the large model in the cloud, enabling real-time pre-labeling of vehicle-side data; based on the labeling confidence, different processing strategies are triggered, such as local labeling, asynchronous upload, or immediate upload, to balance bandwidth cost and the ability to discover weak samples.

[0054] The aforementioned first-layer rule-based pre-recall mechanism serves as the first funnel in the cloud-based recall chain. Before any model inference or high-cost processing, it performs rule labeling and filtering on massive amounts of raw data based on sensor outputs and algorithm module outputs. The sensors mentioned above can output, but are not limited to, one of the following: Camera: image quality indicators (exposure status, sharpness, occlusion rate, etc.); LiDAR: point cloud density, target detection box information, obstacle type and distance; Millimeter-wave radar: target velocity, acceleration, and relative position; High-precision positioning and inertial navigation unit (GNSS / IMU): vehicle position, velocity, and attitude information; Vehicle status signals (CAN bus): vehicle speed, acceleration, brake pedal position, steering angle, emergency braking events, etc.

[0055] The outputs of the aforementioned algorithm modules may include, but are not limited to, at least one of the following: the perception module outputs the target detection category (vehicles, pedestrians, non-motorized vehicles, static obstacles, etc.), target confidence, and target quantity; the tracking module outputs the target trajectory status, target disappearance and addition events; the prediction module outputs the prediction results of high-risk traffic participant behavior; the planning and control module outputs trajectory anomalies, safety takeover events, and emergency obstacle avoidance trigger events; and the anomaly detection module outputs various fault codes and perception anomaly trigger signs.

[0056] By transforming the definition of the target category or target scenario into the corresponding rule condition expression, the original data size can be reduced to a subset of candidate data, thereby reducing the overall cost of subsequent model inference, vectorization computation, and long text generation.

[0057] The aforementioned second-layer parallel recall link, such as Figure 4 As shown, LoRA (Low-Rank Adaptation) fine-tuning achieves task adaptation with very few trainable parameters by injecting a low-rank adaptation matrix into only a portion of the attention weight matrix in the base model. Compared to full fine-tuning, LoRA fine-tuning requires fewer samples, has shorter training time, and lower computational cost, making it suitable for quickly building recall models with initial recognition capabilities under conditions of a small number of labeled samples.

[0058] When only a small number of labeled samples have been accumulated for a certain target category or target scenario, LoRA fine-tuning is prioritized for coarse recall. Through the coarse recall results, more potentially relevant samples can be mined from the candidate data. These samples are then reviewed manually or automatically to expand the labeled set before the next round of iterative fine-tuning, forming a gradual enhancement process from a small number of samples to a large number of samples.

[0059] The LoRA fine-tuning pipeline and the full fine-tuning pipeline are parallel and can run simultaneously for different categories or scenarios. The recall results, newly confirmed samples, and candidate data generated by the LoRA fine-tuning pipeline can be directly incorporated into the construction process of the full fine-tuning training set, thereby effectively making up for the lack of samples in cold start and long-tail categories in full fine-tuning.

[0060] Once the training samples for the target category or scene have accumulated to a certain scale, a full fine-tuning process can be triggered to obtain a more accurate recognition model for high-quality data labeling and high-precision recall. By updating all parameters through full fine-tuning, the model can more fully learn the feature distribution of the target task, resulting in higher classification accuracy, lower false positive recall, and stronger generalization ability. Full fine-tuning is suitable for: mature categories with a sufficient number of labeled samples; scenarios with high recall accuracy requirements and low tolerance for false positives; and tasks requiring batch high-precision labeling of large-scale candidate data.

[0061] The full-scale fine-tuning model does not perform a broadcast scan of all the original data. Instead, it serves as a high-precision supplementary layer to the funnel-shaped architecture, prioritizing the processing of candidate data that has been pre-filtered by rules, and performing fine-grained review and secondary confirmation of the multimodal retrieval recall results, thereby controlling the overall inference cost while ensuring accuracy.

[0062] LoRA fine-tuning quickly starts and mines candidate samples, providing a supplementary training set for full fine-tuning. After achieving higher accuracy in full fine-tuning, it can further provide high-quality pseudo-labels or data augmentation sources for the LoRA fine-tuning link, forming a bidirectional enhancement mechanism between the two recall links.

[0063] To address the issue that pure image vectors cannot fully express high-level driving semantics, this example constructs a joint retrieval and recall link based on selective image deduplication, long text generation from a visual language model, and multimodal embedding, such as... Figure 3 As shown, it may include the following steps: Step 1: Image Embedding and Similarity Deduplication: Image embedding is performed on the images in the candidate data to obtain a vector representation of each image. The similarity between images (e.g., cosine similarity) is calculated to identify nearly duplicate images or highly similar image groups. Furthermore, a preset threshold is set; an image is considered sufficiently different only when the similarity difference between images exceeds this threshold, thus triggering the subsequent long text generation step. Images whose similarity difference does not exceed the preset threshold are determined to be nearly duplicate samples and are not given repeated long text descriptions, effectively controlling computational and storage resource consumption.

[0064] Step 2: Scene-based long text generation based on Visual Language Model (VLM): For images that meet the difference criteria, the Visual Language Model (VLM) is invoked in conjunction with common prompt words to automatically generate a long text description of the scene that covers multi-dimensional driving semantics.

[0065] Long texts should include at least the following dimensions: Environment and Lighting: Describe weather conditions (sunny, cloudy, rainy, foggy, etc.), lighting conditions (front lighting, back lighting, nighttime, dusk, etc.), and visibility. Roads and road surfaces: Describe the road type (highway, urban expressway, ordinary road, intersection, etc.), number and width of lanes, road surface condition (dry, wet, waterlogged, damaged, etc.) and the integrity of road markings; Traffic and Scene: Describe the types of traffic participants (motor vehicles, pedestrians, non-motor vehicles, special vehicles, etc.), traffic density, and traffic signal status in the scene; Main vehicle context: Describes the current driving status of the main vehicle (speed range, straight / turning / lane changing / parking), relative distance to surrounding targets, and interaction status; Key details and spatial relationships: Describe key targets in the image (such as truncated targets, occluded targets, and irregularly shaped targets) and the spatial relationships between them; Right-of-way analysis: describes the right-of-way allocation of each traffic participant, whether there are right-of-way conflicts, and the priority status of pedestrians or vehicles; That is, by combining the above information, a natural language scene description for the image is generated, providing a complete context for semantic search.

[0066] Step 3: Multimodal Unified Embedding and Index Construction: Images and corresponding long-form scene text are input into a multimodal embedding model to obtain image vectors and text vectors, which are then mapped to a unified vector space. Simultaneously, word segmentation is performed on the long-form scene text, and an inverted index based on the BM25 algorithm is constructed. Finally, the image vectors, text vectors, and BM25 index are stored together in a database, forming a data indexing system that supports multimodal joint retrieval.

[0067] Step 4: Multimodal joint recall.

[0068] During the recall phase, the target tags, scene description text, keywords, or query statements to be searched are converted into query vectors. At the same time, vector similarity retrieval and BM25 keyword retrieval are performed. The two search results are then combined and sorted according to a weighted fusion or re-ranking strategy to obtain the final recall results.

[0069] In this example, the results of rule-based pre-recall, LoRA fine-tuning recall, full fine-tuning recall, and multimodal embedding retrieval recall are aggregated, deduplication and confidence merging are performed, priority sorting is applied, and the following classification dataset is constructed: 1) Downstream algorithm training set: High-quality data that has been recalled, filtered and confirmed, used as continuous training data for downstream autonomous driving algorithms such as end-to-end driving models and VLA (Vision-Language-Action) large models; 2) High-confidence automatic inclusion set: Samples that have been hit by multiple recalls and have high confidence can be directly labeled and included in the training library; 3) Manual review of the pending confirmation set: Samples from single-channel recalls or with confidence levels in the middle range are submitted for manual review; 4) LoRA Fine-tuning Lightweight Training Set: A LoRA fine-tuning sample set for cold-start or long-tail categories; 5) Full-scale fine-tuning of high-quality training set: A high-confidence confirmed sample is gathered to form a high-quality training set for full-scale fine-tuning; 6) Weak sample set: Samples with low confidence, missed by multiple recalls, or with high uncertainty in model prediction, used for long-tail expansion.

[0070] That is, to form a closed-loop data system in which recall drives training and training feeds back into recall.

[0071] In this example, a lightweight model is deployed on the vehicle side to work in conjunction with the cloud-based recall chain, forming a complete multi-path recall system. The model deployed on the vehicle side is obtained from the large cloud model (such as the recall model after full fine-tuning) through knowledge distillation, ensuring that its parameter count and operator complexity meet the operational constraints of the vehicle-side computing platform (embedded processor or in-vehicle AI chip), while retaining sufficient recognition and classification capabilities.

[0072] During vehicle operation, the vehicle-side model performs real-time or near-real-time label prediction on image frames or video clips captured by cameras, and outputs a corresponding confidence score for each predicted label. Based on the confidence score output by the vehicle-side model, a first threshold and a second threshold are set in this example (the first threshold is higher than the second threshold), such as... Figure 5 As shown, the hierarchical processing is performed according to the following rules: Scenario 1 (Confidence level higher than the first threshold): The vehicle-side model is considered to have reliable annotation results for the data. The vehicle directly performs local labeling and records the labels on the data, without immediately uploading the original data. Only the annotation results are uploaded to the cloud for subsequent cloud-based filtering, data storage, and quality inspection. This approach minimizes network transmission costs.

[0073] Scenario 2 (Confidence level between the second and first thresholds): The annotation results from the vehicle-side model are considered to have some uncertainty, requiring verification using a cloud-based model. An asynchronous upload method is used, sending the data and vehicle-side label results to the cloud. The cloud model verifies the data and returns the final labels, after which the cloud performs subsequent filtering and data entry operations. This method ensures the quality of the final annotation while reducing real-time bandwidth consumption through asynchronous processing.

[0074] Scenario 3 (Confidence level below the second threshold): The vehicle-side model is considered to have high uncertainty regarding this data, belonging to a weak area of ​​the model. An immediate upload is triggered, prioritizing the sending of the raw data to the cloud for high-precision processing and annotation. Simultaneously, this type of data is automatically included in the weak sample set for subsequent training set expansion, gradually improving the recognition and recall capabilities of both the cloud-based and vehicle-side models in weak scenarios.

[0075] In one implementation, the first threshold can be set to 85%, and the second threshold can be set to 70%. These thresholds are not restrictive values ​​and can be flexibly configured in practice according to specific business objectives, model capability levels, communication bandwidth constraints, and acceptable false positive and false negative rates.

[0076] To further construct a complete cloud-vehicle data closed-loop iteration mechanism, ensuring that the multi-path recall system can continuously improve its effectiveness with data accumulation and model iteration, the complete closed-loop process can be as follows: Figure 6 As shown, it includes: S1: Data Acquisition: The vehicle collects raw driving data through various sensors; S2: Vehicle-side preprocessing: The vehicle-side distillation model labels the data in real time and determines the processing method (local labeling / asynchronous upload / immediate upload) based on the confidence level. S3: Cloud aggregation: The cloud receives data from various sources, including raw data and tag information uploaded by the vehicle. S4: Cloud-based multi-path recall: Execute rule-based pre-recall, LoRA fine-tuning recall, full fine-tuning recall, and multimodal embedding retrieval recall, aggregate results and build training set; S5: Model Training and Update: The training set is distributed to two uses: 1) as continuous training data for downstream autonomous driving algorithms such as end-to-end driving models and VLA large models (core use); 2) for iterative updates of LoRA fine-tuning models and full fine-tuning models to continuously improve recall accuracy (auxiliary use). S6: Model Distillation and Deployment: Distill the updated cloud model into a new vehicle-side model and deploy it to the vehicle-side for update and deployment; S7: Next iteration: The vehicle-side continues to perform data pre-labeling with stronger model capabilities, iterating in a loop to form a positive cycle where the more data is accumulated, the stronger the model becomes, and the more accurate the recall becomes.

[0077] In the example above, multi-path collaboration significantly improves recall coverage: through the organic collaboration of four cloud-based recall links—rule-based recall, LoRA fine-tuning recall, full-scale fine-tuning recall, and multimodal embedding retrieval recall—with the vehicle-side recall link, different recall methods complement each other in terms of coverage scenarios, semantic granularity, and accuracy levels, which can significantly improve the recall coverage of autonomous driving data in complex and long-tail scenarios.

[0078] The funnel-style hierarchical architecture effectively controls processing costs: rule-based recall is used as a pre-funnel to significantly reduce the range of data candidates before triggering more expensive model inference or multimodal processing, so that the overall computation and storage costs of a single recall task can be significantly lower than the solution of performing model scanning on the full data.

[0079] Improving recall accuracy in complex scenes by explicitly defining driving semantics: By generating structured long texts containing ambient lighting, road structure, traffic participant relationships, main vehicle context, spatial relationships, and right-of-way analysis through a visual language model, and embedding them into images in a unified multimodal manner, the high-level driving semantics implicit in the images can be quantitatively expressed, thereby significantly improving the semantic recall accuracy of complex and fine-grained scenes.

[0080] Image deduplication can reduce the cost of generating long texts: by first performing embedding similarity calculations on images, and then determining whether to trigger long text generation based on a preset threshold, it avoids a large number of repetitive descriptions of nearly identical images, thereby effectively reducing the resource consumption of visual language model inference and vectorization, and has cost feasibility for large-scale engineering implementation.

[0081] Combining vector retrieval with BM25 can improve retrieval robustness: multimodal vector retrieval can capture semantically similar scenarios, while BM25 keyword retrieval can accurately match specific words and category descriptions. The combined use of the two can balance semantic generalization and keyword accuracy, improving the overall effect of target scenario retrieval.

[0082] A progressive enhancement mechanism is formed by the collaboration of LoRA and full fine-tuning: LoRA fine-tuning quickly responds to cold start and long-tail scenarios, while full fine-tuning provides high-precision recall capability after sample accumulation; LoRA recall results supplement the full fine-tuning training set in reverse, so that the two recall links promote each other and gradually form a progressive enhancement effect from coarse to fine.

[0083] The vehicle-side confidence grading mechanism can balance cost and weak sample discovery: a three-stage processing strategy based on the first and second thresholds enables high-confidence data to be locally labeled in a low-cost manner, medium-confidence data to be uploaded asynchronously for verification, and low-confidence weak samples to be prioritized for back transmission and processing, thereby maximizing the recovery of data value under limited bandwidth and computing power constraints.

[0084] The high-quality training sets continuously generated by the multi-path recall system are directly fed into downstream autonomous driving algorithms such as end-to-end driving models and VLA large models for continuous training. On the other hand, through the discovery of weak samples on the vehicle side, iterative updates of the labeled model, and expansion of the training set, a positive feedback data closed-loop iterative mechanism is constructed, enabling the recall model's capabilities to continuously improve with the accumulation of data, ultimately achieving a positive cycle where the more data accumulates, the stronger the algorithm becomes, and the more accurate the recall becomes.

[0085] The methods and embodiments provided in the above-described embodiments of this application can be executed in a mobile terminal, computer terminal, or similar computing device. Taking its operation on an electronic device as an example... Figure 7 This is a hardware structure block diagram of an electronic device for a multi-path recall method for autonomous driving data provided in this application. Figure 7 As shown, the electronic device 10 may include one or more (only one is shown in the figure) processors 02 (processors 02 may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 04 for storing data, and a transmission module 06 for communication functions. Those skilled in the art will understand that... Figure 7 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, electronic device 10 may also include... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown.

[0086] The memory 04 can be used to store software programs and modules of application software, such as the program instructions / modules corresponding to the multi-path recall method for autonomous driving data in the embodiments of this application. The processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, thereby realizing the aforementioned multi-path recall method for autonomous driving data of the application. The memory 04 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 04 may further include memory remotely located relative to the processor 02, and these remote memories can be connected to the electronic device 10 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] The transmission module 06 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 10. In one example, the transmission module 06 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 06 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0088] At the software level, the aforementioned multi-path recall device for autonomous driving data can, as follows: Figure 8 As shown, it includes: The receiving module 801 is used to receive multi-source data that is hierarchically transmitted back from the vehicle end based on the relationship between sample confidence and confidence threshold; The filtering module 802 is used to perform pre-filtering of the received multi-source data according to rules to obtain a subset of candidate data; The fusion recall module 803 is used to feed a subset of candidate data into the coarse recall model, the full fine-tuning recall model, and the multimodal retrieval recall model in parallel for processing, so as to obtain multi-path recall results. The training set generation module 804 is used to fuse and classify the multi-path recall results to construct a multi-class training dataset, wherein the multi-class training dataset is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full-scale fine-tuning fine recall model.

[0089] In one implementation, the multi-source data that the vehicle-side transmits hierarchically based on the relationship between sample confidence and confidence threshold may include: real-time driving images and vehicle body data collected by multi-source sensors on the vehicle-side as multi-source data; real-time inference using a lightweight model obtained by distilling a large model from the cloud and deployed locally on the vehicle-side to output the label prediction result and corresponding sample confidence for each data point; and a three-stage hierarchical processing of the confidence based on a preset first threshold and a second threshold: data with sample confidence higher than the first threshold are locally labeled and only the label prediction result is uploaded; data with sample confidence between the second threshold and the first threshold are asynchronously uploaded to the cloud for verification; and data with sample confidence lower than the second threshold are identified as weak samples in the model and the original data is immediately uploaded first.

[0090] In one implementation, the fusion recall module 803 is specifically used to obtain effective samples through a coarse recall model; obtain high-confidence pseudo-label samples through full-scale fine-tuning of the recall model; obtain highly semantically relevant samples through a multimodal retrieval recall model; send the effective samples, high-confidence pseudo-label samples, and highly semantically relevant samples into the multi-path recall fusion layer for aggregation processing, uniformly label the sample source branches and confidence scores, complete sample identification deduplication, merge duplicate samples and retain the record with the highest comprehensive confidence score, and output the integrated multi-path recall result.

[0091] In one implementation, obtaining effective samples through a coarse recall model and obtaining high-confidence pseudo-label samples through full-scale fine-tuning of a fine-tuning recall model may include: simultaneously feeding a subset of candidate data into the coarse recall model and parallel inference of the full-scale fine-tuning recall model; collecting suspected relevant samples from the output of the coarse recall model, and eliminating falsely recalled invalid samples through manual review or cloud-based high-precision model verification, retaining effective samples that truly match the target scenario to form a reusable incremental labeled sample pool; supplementing the reusable incremental labeled sample pool into the full-scale fine-tuning training set to supplement the sample quantity required for training the full-scale fine-tuning recall model; obtaining high-confidence pseudo-label samples after the full-scale fine-tuning recall model is completed, and expanding the coarse recall training set with the high-confidence pseudo-label samples; and sending the effective samples and the high-confidence pseudo-label samples into a multi-path recall fusion layer for aggregation processing.

[0092] In one implementation, obtaining highly semantically relevant samples through a multimodal retrieval and recall model may include: the multimodal retrieval and recall model extracting image embeddings from a subset of candidate data and calculating similarity; triggering a visual language model for differentiated images with similarity below a threshold; the visual language model generating structured long-text scenes containing lighting, roads, traffic participants, vehicle status, and right-of-way relationships according to autonomous driving-specific dimensions; inputting the differentiated images and the generated long-text scenes into the same multimodal embedding model, mapping them to a shared vector space, generating image vectors and text vectors respectively, and storing them in a vector library; simultaneously segmenting the long-text to extract driving keywords, building a BM25 inverted text index, forming two retrieval systems: vector retrieval and keyword retrieval; converting the query images and text into query vectors during the recall stage, and simultaneously performing vector retrieval and keyword retrieval; fusing the scores from the two retrieval paths and introducing a semantic conflict penalty term to filter samples that are visually similar but logically irrelevant, sorting them, and outputting highly semantically relevant samples, which are then sent to a multimodal recall fusion layer for aggregation.

[0093] In one implementation, merging duplicate samples and retaining the record with the highest overall confidence score may include: Combine duplicate samples using the following formula and calculate the overall confidence level of each sample:

[0094] in, To assess the overall confidence level, Dynamic weights for rule-based recall. For coarse recall dynamic weights, To fine-tune the dynamic weights for full-scale recall, Fixed base weights for multimodal retrieval recall. To retrieve the matching score for the rule recall, For the coarse recall matching score, To fine-tune the full-scale recall matching score, For multimodal retrieval recall matching score, The preset hyperparameters are used, where, when the sample size is sufficient, the improvement is... The value of .

[0095] In one implementation, the multimodal retrieval recall matching score is calculated according to the following formula:

[0096] in, The cosine similarity is used to calculate the difference between the image and the generated scene long text multimodal vector. The score for keyword retrieval in driving scenarios. The penalty term for semantic conflicts in driving is increased when there are conflicts in right-of-way, conflicting scene types, or conflicting driving states of the main vehicle. This is the balance adjustment coefficient.

[0097] This application also provides a specific implementation of an electronic device capable of implementing all steps of the multi-path recall method for autonomous driving data in the above embodiments. The electronic device specifically includes: a processor, a memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other via the bus; the processor is used to call a computer program in the memory, and when the processor executes the computer program, it implements all steps of the multi-path recall method for autonomous driving data in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: Step 1: Receive multi-source data from the vehicle terminal, which is hierarchically transmitted based on the relationship between sample confidence and confidence threshold; Step 2: The cloud performs pre-filtering based on rules on the received multi-source data to obtain a subset of candidate data; Step 3: Feed the candidate data subsets into the coarse recall model, the full fine-tuning recall model, and the multimodal retrieval recall model in parallel for processing to obtain multi-path recall results; Step 4: Merge and classify the multi-path recall results to construct a multi-class training dataset, wherein the multi-class training dataset is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full fine-tuning fine recall model.

[0098] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multi-path recall method for autonomous driving data in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multi-path recall method for autonomous driving data in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: Step 1: Receive multi-source data from the vehicle terminal, which is hierarchically transmitted based on the relationship between sample confidence and confidence threshold; Step 2: The cloud performs pre-filtering based on rules on the received multi-source data to obtain a subset of candidate data; Step 3: Feed the candidate data subsets into the coarse recall model, the full fine-tuning recall model, and the multimodal retrieval recall model in parallel for processing to obtain multi-path recall results; Step 4: Merge and classify the multi-path recall results to construct a multi-class training dataset, wherein the multi-class training dataset is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full fine-tuning fine recall model.

[0099] As described above, this application embodiment achieves differentiated transmission of data from the vehicle based on confidence thresholds, distinguishing between high, medium, and low confidence samples. This reduces the bandwidth required for uploading raw data from the vehicle and the pressure on cloud storage. Pre-screening based on rules completes the first layer of data compression, reducing computational overhead for subsequent model processing. The parallel collaborative processing of three models—coarse recall, full-scale fine-tuning recall, and multimodal retrieval—covers cold-start long-tail scenarios, high-precision mature scenarios, and high-level driving semantic scenarios, thereby simultaneously improving recall coverage and accuracy. The fusion of multiple result classifications produces multiple training datasets, which can simultaneously supply the autonomous driving business algorithm, the vehicle-mounted lightweight model, and the two recall models for iterative training, forming a self-driven data loop and continuously optimizing autonomous driving scene recognition and data mining capabilities. Through these methods, efficient, low-cost, and high-precision hierarchical recall of raw autonomous driving data can be achieved, thereby improving the accuracy of autonomous driving models.

[0100] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0101] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0102] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0103] While this specification provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0104] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0105] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0106] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0109] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0110] The above description is merely an embodiment of the present specification and is not intended to limit the embodiments of the present specification. For those skilled in the art, various modifications and variations can be made to the embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present specification should be included within the scope of the claims of the embodiments of the present specification.

Claims

1. A multi-path recall method for autonomous driving data, characterized in that, The method includes: The receiving vehicle receives multi-source data that is graded and transmitted back based on the relationship between sample confidence and confidence threshold; The cloud performs pre-screening based on rules on the received multi-source data to obtain a subset of candidate data. A subset of candidate data is fed in parallel into a coarse recall model, a full-scale fine-tuning recall model, and a multimodal retrieval recall model for processing to obtain multi-path recall results; The multi-path recall results are fused and classified to construct a multi-class training dataset, which is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full-scale fine-tuning recall model; The process involves feeding a subset of candidate data in parallel into a coarse recall model, a full-scale fine-tuning recall model, and a multimodal retrieval recall model to obtain multi-path recall results. These results include: obtaining valid samples through the coarse recall model; obtaining high-confidence pseudo-label samples through the full-scale fine-tuning recall model; and obtaining highly semantically relevant samples through the multimodal retrieval recall model. The valid samples, high-confidence pseudo-label samples, and highly semantically relevant samples are then fed into a multi-path recall fusion layer for aggregation processing. This layer uniformly labels the sample source branches and confidence scores, performs sample identification deduplication, merges duplicate samples, retains the record with the highest overall confidence score, and outputs the integrated multi-path recall results. The process of obtaining highly semantically relevant samples through a multimodal retrieval and recall model includes: the multimodal retrieval and recall model extracting image embeddings from a subset of candidate data and calculating similarity; triggering a visual language model for differentiated images with similarity below a threshold; the visual language model generating structured long-text scenes containing lighting, roads, traffic participants, vehicle status, and right-of-way relationships according to autonomous driving-specific dimensions; inputting the differentiated images and the generated long-text scenes into the same multimodal embedding model, mapping them to a shared vector space, generating image vectors and text vectors respectively, and storing them in a vector library; simultaneously extracting driving keywords from the long-text by word segmentation, building a BM25 inverted text index, and forming two retrieval systems: vector retrieval and keyword retrieval; converting the query images and text into query vectors during the recall stage, and simultaneously executing vector retrieval and keyword retrieval; fusing the scores from the two retrieval paths and introducing a semantic conflict penalty term to filter samples that are similar in appearance but irrelevant to driving logic, sorting them, and outputting highly semantically relevant samples, which are then sent to the multi-path recall fusion layer for aggregation.

2. The method according to claim 1, characterized in that, The vehicle-side data is hierarchically transmitted from multiple sources based on the relationship between sample confidence and confidence threshold, including: The vehicle's multi-source sensors collect real-time driving images and vehicle body data as multi-source data; A lightweight model, obtained by distilling a large model from the cloud, is deployed locally on the vehicle for real-time inference, outputting the label prediction result and corresponding sample confidence for each data point. Based on preset first and second thresholds, the confidence level is processed in three stages: data with a confidence level higher than the first threshold are locally labeled and only the label prediction results are uploaded; data with a confidence level between the second and first thresholds are asynchronously uploaded to the cloud for review; data with a confidence level lower than the second threshold are identified as weak samples in the model and the original data is uploaded immediately.

3. The method according to claim 1, characterized in that, Valid samples are obtained through a coarse recall model, and high-confidence pseudo-label samples are obtained through a full-scale fine-tuning recall model, including: A subset of candidate data is simultaneously fed into the coarse recall model, and the full-scale fine-tuning of the fine recall model is performed in parallel inference. Collect suspected relevant samples from the coarse recall model, and remove invalid samples that are falsely recalled through manual review or cloud-based high-precision model verification, while retaining valid samples that truly match the target scenario to form a reusable incremental labeled sample pool. The reusable incremental labeled sample pool is added to the full fine-tuning training set to supplement the sample quantity required for training the full fine-tuning fine-recall model. After the full fine-tuning of the fine recall model is completed, high-confidence pseudo-label samples are obtained, and the coarse recall training set is expanded using the high-confidence pseudo-label samples. The valid samples and the high-confidence pseudo-label samples are sent to the multi-path recall fusion layer for aggregation processing.

4. The method according to claim 1, characterized in that, Merge duplicate samples and retain the record with the highest overall confidence level, including: Combine duplicate samples using the following formula and calculate the overall confidence level of each sample: in, To assess the overall confidence level, Dynamic weights for rule-based recall. For coarse recall dynamic weights, To fine-tune the dynamic weights for full-scale recall, Fixed base weights for multimodal retrieval recall. To retrieve the matching score for the rule recall, For the coarse recall matching score, To fine-tune the full-scale recall matching score, For multimodal retrieval recall matching score, The preset hyperparameters are used, where, when the sample size is sufficient, the improvement is... The value of .

5. The method according to claim 4, characterized in that, The multimodal retrieval recall matching score is calculated using the following formula: in, The cosine similarity is used to calculate the difference between the image and the generated scene long text multimodal vector. The score for keyword retrieval in driving scenarios. The penalty term for semantic conflicts in driving is increased when there are conflicts in right-of-way, conflicting scene types, or conflicting driving states of the main vehicle. This is the balance adjustment coefficient.

6. A multi-path recall device for autonomous driving data, characterized in that, include: The receiving module is used to receive multi-source data that is hierarchically transmitted back from the vehicle based on the relationship between sample confidence and confidence threshold; The filtering module is used to perform pre-filtering of received multi-source data according to rules to obtain a subset of candidate data; The fusion recall module is used to feed a subset of candidate data into the coarse recall model, the full fine-tuning recall model, and the multimodal retrieval recall model in parallel for processing, so as to obtain multi-path recall results; The training set generation module is used to fuse and classify the multi-path recall results to construct a multi-class training dataset, wherein the multi-class training dataset is used to train at least one of the following models: autonomous driving algorithm model, lightweight model, coarse recall model, and full-scale fine-tuning fine recall model; The process involves feeding a subset of candidate data in parallel into a coarse recall model, a full-scale fine-tuning recall model, and a multimodal retrieval recall model to obtain multi-path recall results. These results include: obtaining valid samples through the coarse recall model; obtaining high-confidence pseudo-label samples through the full-scale fine-tuning recall model; and obtaining highly semantically relevant samples through the multimodal retrieval recall model. The valid samples, high-confidence pseudo-label samples, and highly semantically relevant samples are then fed into a multi-path recall fusion layer for aggregation processing. This layer uniformly labels the sample source branches and confidence scores, performs sample identification deduplication, merges duplicate samples, retains the record with the highest overall confidence score, and outputs the integrated multi-path recall results. The process of obtaining highly semantically relevant samples through a multimodal retrieval and recall model includes: the multimodal retrieval and recall model extracting image embeddings from a subset of candidate data and calculating similarity; triggering a visual language model for differentiated images with similarity below a threshold; the visual language model generating structured long-text scenes containing lighting, roads, traffic participants, vehicle status, and right-of-way relationships according to autonomous driving-specific dimensions; inputting the differentiated images and the generated long-text scenes into the same multimodal embedding model, mapping them to a shared vector space, generating image vectors and text vectors respectively, and storing them in a vector library; simultaneously extracting driving keywords from the long-text by word segmentation, building a BM25 inverted text index, and forming two retrieval systems: vector retrieval and keyword retrieval; converting the query images and text into query vectors during the recall stage, and simultaneously executing vector retrieval and keyword retrieval; fusing the scores from the two retrieval paths and introducing a semantic conflict penalty term to filter samples that are similar in appearance but irrelevant to driving logic, sorting them, and outputting highly semantically relevant samples, which are then sent to the multi-path recall fusion layer for aggregation.

7. An electronic device comprising a processor and a memory for storing processor-executable instructions, characterized in that, When the processor executes the instructions, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

Images