Long-tail scene data screening method and device and readable storage medium

Abstract

The invention discloses a long-tail scene data screening method and device and a readable storage medium, and the method comprises the steps: inputting intelligent driving data and prompt words into a visual language model, so that the visual language model carries out the long-tail scene recognition of the intelligent driving data according to a task set by the prompt words, the recognition result is output according to an output format set by the cue word, the output format comprises long-tail scene information, the long-tail scene information comprises a long-tail scene category and a long-tail scene confidence coefficient, the long-tail scene category is an enumeration field, and the long-tail scene confidence coefficient is an enumeration field. The visual language model carries out identification item by item according to enumeration values included in the enumeration fields and outputs corresponding enumeration values; and if the type of the long-tail scene in the recognition result is a set enumeration value and the confidence coefficient of the long-tail scene is greater than a first preset confidence coefficient, taking the intelligent driving data corresponding to the recognition result as long-tail scene data. According to the method and the device, the accuracy of screening the long-tail scene data is greatly improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus and readable storage medium for filtering long-tail scene data. Background Technology

[0002] The field of autonomous driving is witnessing rapid advancements in environmental perception technologies, resulting in increasingly better user experiences. As intelligent driving algorithms mature in routine scenarios, improving their coverage of long-tail scenarios becomes increasingly crucial. Therefore, identifying valuable long-tail scenario data from massive amounts of intelligent driving data has become a pressing issue. Long-tail scenarios (corner cases) refer to infrequent or extreme scenarios.

[0003] Relying on manual screening of long-tail scene data results in high accuracy at the cost of extremely low efficiency. Therefore, improving the efficiency of long-tail scene data screening while maintaining accuracy becomes crucial. By designing scene recognition algorithms using CNN (Convolutional Neural Network) or Transformer deep learning models (Transformer is a deep learning model architecture based on self-attention mechanisms), the input autonomous driving data can be classified according to different scenes, significantly improving efficiency while ensuring a certain level of accuracy.

[0004] However, the accuracy of using CNN or Transformer deep learning models to filter long-tail scene data for intelligent driving data is not high enough, and the generalization of intelligent driving data recognition for different types (such as different times, different weather and different vehicle models) is poor. Summary of the Invention

[0005] This application provides a method, apparatus, and readable storage medium for filtering long-tail scene data, aiming to solve the technical problems that the accuracy of filtering long-tail scene data of intelligent driving data using CNN or Transformer deep learning models is not high enough, and the generalization of the recognition of intelligent driving data of different types (such as different times, different weather and different vehicle models) is poor.

[0006] In a first aspect, embodiments of this application provide a method for filtering long-tail scene data, the method comprising: Intelligent driving data and prompts are input into a visual language model, which then performs long-tail scene recognition on the intelligent driving data according to the task set by the prompts, and outputs the recognition results according to the output format set by the prompts. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, which allows the visual language model to recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompts also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. If the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then the intelligent driving data corresponding to the recognition result will be used as the long-tail scene data.

[0007] Optionally, before inputting the intelligent driving data and prompt words into the visual language model, so that the visual language model can perform long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and output the recognition result according to the output format set by the prompt words, the following steps are included: The visual language model is configured according to preset hyperparameters to control the speed and accuracy of semantic generation by the visual language model. The preset hyperparameters include temperature, which is lower than the default temperature of the visual language model to reduce the randomness of the visual language model's output.

[0008] Optionally, if the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then after using the intelligent driving data corresponding to the recognition result as long-tail scene data, the following steps are included: If a new enumeration value appears in the long-tail scene category in the recognition results, and the confidence level of the long-tail scene is greater than the second preset confidence level, then the new enumeration value is added to the long-tail scene category.

[0009] Optionally, the output format may further include scene basic information and driving suggestions, and the long-tail scene information may further include long-tail scene descriptions. The scene basic information, driving suggestions, and long-tail scene descriptions are used to provide explanations for the recognition results.

[0010] Optionally, the scenario basic information and driving suggestions are enumerated fields, so that the visual language model can identify each enumerated value included in the enumerated fields and output the corresponding enumerated value.

[0011] Optionally, the prompt words may also include examples of recognition results.

[0012] Optionally, the visual language model is an InternVL3-78B model, which is deployed locally using the lmdeploy tool.

[0013] Secondly, embodiments of this application provide a method for filtering long-tail scene data, applied to a visual language model, the method comprising: The visual language model acquires intelligent driving data and prompt words, performs long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and outputs the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, which allows the visual language model to recognize and output the corresponding enumeration value item by item according to the enumeration value included in the enumeration field. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. If the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then the intelligent driving data corresponding to the recognition result will be used as the long-tail scene data.

[0014] Thirdly, embodiments of this application provide a long-tail scene data filtering device, the long-tail scene data filtering device comprising: The recognition module is used to input intelligent driving data and prompt words into the visual language model, so that the visual language model can perform long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and output the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, so that the visual language model can recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. The filtering module is used to select the intelligent driving data corresponding to the recognition result as long-tail scene data if the long-tail scene category in the recognition result is a set enumeration value and the confidence of the long-tail scene is greater than the first preset confidence.

[0015] Fourthly, embodiments of this application provide a readable storage medium storing a long-tail scene data filtering program, wherein when the long-tail scene data filtering program is executed by a processor, it implements the steps of the long-tail scene data filtering method as described above.

[0016] The beneficial effects of the technical solutions provided in this application include: In this embodiment, intelligent driving data and prompt words are input into a visual language model, which then performs long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and outputs the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, which allows the visual language model to recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. If the long-tail scene category in the recognition result is a set enumeration value, and the long-tail scene confidence is greater than a first preset confidence, then the intelligent driving data corresponding to the recognition result is taken as long-tail scene data. Through the embodiments of this application, a visual language model is enabled to perform long-tail scene recognition on input intelligent driving data by setting tasks through prompt words. The prompt words also set a unified output format for the recognition results of the visual language model, thereby improving the versatility of the visual language model in recognizing different types of intelligent driving data. By setting the long-tail scene category in the output format as an enumeration field, the visual language model is forced to recognize and output the corresponding enumeration values ​​item by item according to the enumeration values ​​included in the enumeration field. Through constraints on the non-empty, exact, traffic rule, and confidence levels of the output content, the illusions that the visual language model is prone to can be greatly suppressed, and the accuracy of long-tail scene data filtering is significantly improved. Furthermore, the visual language model is an artificial intelligence model that integrates visual (image / video) and linguistic (text) modalities. Through cross-modal semantic alignment and joint reasoning, it achieves deep understanding and generation of complex scenes. The visual language model greatly improves generalization, increases the coverage of long-tail scene data recognition, and reduces the adaptation cost for recognizing different types of intelligent driving data. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the first process of an embodiment of the long-tail scenario data filtering method of this application; Figure 2 This is a schematic diagram of the second process of an embodiment of the long-tail scenario data filtering method of this application; Figure 3 This is a schematic diagram of the third process of an embodiment of the long-tail scenario data filtering method of this application; Figure 4 This is a schematic diagram of the functional modules of an embodiment of the long-tail scene data filtering device of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0020] Firstly, embodiments of this application provide a method for filtering data in long-tail scenarios.

[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a schematic diagram of the first process of an embodiment of the long-tail scenario data filtering method of this application, as shown below. Figure 1 As shown, the data filtering methods for long-tail scenarios include: Step S10: Input the intelligent driving data and prompt words into the visual language model so that the visual language model can perform long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and output the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field so that the visual language model can recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints.

[0022] In this embodiment, intelligent driving data refers to various data collected by various sensors during the vehicle's autonomous driving process (such as high-definition images captured by cameras) and various driving data of the vehicle itself (such as vehicle speed). The intelligent driving data and designed prompts are input into a deployed visual language model. Intelligent driving data is typically massive, and this massive amount of data is continuously input into the visual language model in a sequential manner. The visual language model is expected to identify the required long-tail scene data according to the set prompts. The prompts can be configured to perform long-tail scene recognition on the intelligent driving data, and a unified output format can be set for different types of intelligent driving data (such as different times, different weather conditions, and different vehicle models), thereby improving the versatility of the visual language model in recognizing different types of intelligent driving data. Specifically, an example of a prompt written in JSON (an open standard file and data exchange format) format is as follows: Prompt = """ { Instruction: "You are a vision-language model (VLM) specialized in autonomous driving perception and decision-making. Analyze the front-facing camera image and generate a structured JSON response with three components: scene description, corner cases, and driving recommendations. Ensure all fields are populated with concrete, actionable insights based on visual cues and traffic rules. **Special Note: In scenarios with common traffic signs (such as speed limits and stop signs), confidence should be below 0.5; in complex or unusual scenarios (such as construction zones, low visibility, and sudden obstacles), confidence should be above 0.7.** "input": "high-resolution front-facing camera image captured in real-time driving scenario", "output_format": { "scene_description": { "road_type": "urban road | highway | residential | rural | parkinglot | construction zone", "weather": "clear | foggy | rainy | snowy | cloudy | low visibility", "traffic_status": "heavy | moderate | light | gridlocked", "traffic_participants": ["vehicles", "pedestrians", "cyclists", "emergency vehicles", "road workers", "animals"], "notable_objects": ["traffic lights", "road signs", "lane markings","obstacles", "potholes", "road closures", "unique features"] }, "corner_cases": [ { "type": "blind_spot | occlusion | rare_sign | erratic_behavior | sudden_obstacle | low-visibility | complex_intersection | construction_zone | sudden_cut-in | accident", "description": "A concise, context-specific explanation of the detected corner case (eg, 'Pedestrian partially occluded by parked vehicle at crosswalk')", "confidence": "0.1-1.0 (Numerical confidence levels must strictly differentiate between scene complexity: less than 0.5 for ordinary traffic sign scenes, and greater than 0.7 for complex / rare scenes)" } ], "driving_recommendation": { "action": "slow_down | stop | proceed_with_caution | change_lane |overtake", "avoid_lane_change": true | false, "yield_to": ["pedestrians", "emergency_vehicles", "cyclists", "school_zones"], "human_intervention": true | false, "risk_zones": ["blind_spot", "crosswalk", "merge_lane", "school_zone", "construction_area"] } }, "constraints": { "no_empty_fields": "All JSON fields must be filled with valid values;use 'unknown' only if critical information is undetectable", "precision": "Use specific terms (eg, 'school zone speed limitsign' instead of 'sign')", "safety_priority": "Prioritize safety-critical recommendations (eg,'stop' for jaywalking pedestrians)", "traffic_rules": "Adhere to traffic regulations (eg, 'yield to emergency vehicles')", "confidence_guidance": "For ordinary traffic sign scenarios (such as speed limit signs and stop signs), confidence must be ≤0.5; for complex / rare scenarios (such as construction zones, low visibility, sudden obstacles, and sudden cut-ins), confidence must be ≥0.7." }, "example_output": { "scene_description": { "road_type": "urban road", "weather": "rainy", "traffic_status": "moderate", "traffic_participants": ["vehicles", "pedestrians", "cyclists"], "notable_objects": ["school zone sign", "wet road surface"] }, "corner_cases": { "type": "occlusion", "description": "Child partially hidden behind parked car near schoolzone", "confidence": 0.85 / / Complex scenario example (high confidence) }, { "type": "rare_sign", "description": "Unfamiliar temporary construction sign detected", "confidence": 0.4 / / Ordinary traffic sign scenario (low confidence) } , "driving_recommendation": { "action": "slow_down", "avoid_lane_change": true, "yield_to": ["pedestrians"], "human_intervention": false, "risk_zones": ["school_zone"] } } } Referring to the above prompt example, the output format includes long-tail scene information, namely the content included in the "corner_cases": [...] section. This information includes long-tail scene categories (i.e., "type" in "corner_cases") and long-tail scene confidence (i.e., "confidence" in "corner_cases"). The long-tail scene category is an enumerated field; specific enumerated values ​​in the example include "blind_spot | occlusion | rare_sign | erratic_behavior | sudden_obstacle | low-visibility | complex_intersection | construction_zone | sudden_cut-in | accident". The visual language model is required to identify each of these enumerated values ​​and output the identified values. The prompt also includes output content constraints, namely "constraints": The {...} section includes output content constraints such as non-empty constraints, precision constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. The non-empty constraint "no_empty_fields" forces the visual language model to fill in all fields even when using 'unknown', only using 'unknown' when key information cannot be detected, preventing the model from "skipping" or "making up stories" regarding field output. The precision constraint "precision" requires the model to use more specific terms, such as 'school zone speed limit sign' instead of... The model is constrained by several output content constraints. The "safety_priority" constraint requires the model to prioritize outputting safety-critical suggestions (e.g., outputting "stop" for pedestrians crossing the road). The "traffic_rules" constraint requires the model to comply with traffic regulations (e.g., "yield to emergency vehicles"). The "confidence_guidance" constraint requires the model to have a confidence level ≤0.5 for common traffic sign scenarios (e.g., speed limits, stop signs) and a confidence level ≥0.7 for complex / rare scenarios (e.g., construction zones, low visibility, sudden obstacles, sudden intersections). These output content constraints significantly suppress the illusions that visual language models are prone to, greatly improving the accuracy of filtering long-tail scene data.

[0023] It should be noted that the execution subject in this embodiment can be a third-party device, vehicle, or human with data processing capabilities. The third-party device, vehicle, or human inputs the intelligent driving data and the designed prompt words into the deployed visual language model. After the visual language model completes the recognition processing, it obtains the recognition result output by the visual language model and further filters out long-tail scene data from the intelligent driving data based on the recognition result.

[0024] Step S20: If the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then the intelligent driving data corresponding to the recognition result is taken as the long-tail scene data.

[0025] In this embodiment, the system parses the recognition results output by the visual language model in JSON format and examines each element in the `corner_cases` array. If the `type` field of `corner_cases` matches a preset long-tail scene enumeration value (such as `construction_zone`, `sudden_cut-in`, etc.), and its confidence value is greater than a first preset confidence threshold (such as 0.7), then the intelligent driving data corresponding to that recognition result is added to the long-tail scene dataset as the filtered long-tail scene data. This threshold is set based on extensive experimental verification to ensure high accuracy while avoiding misclassification of ordinary scenes as long-tail scenes. The system also records the confidence level and other relevant information for each long-tail scene and ordinary scene for subsequent scene analysis and algorithm optimization.

[0026] In this embodiment, intelligent driving data and pre-designed prompts are input into a deployed visual language model. The intelligent driving data is typically massive, and this massive amount of data is continuously input into the visual language model in a sequential manner. The goal is for the visual language model to identify the required long-tail scene data based on the pre-defined prompts. The prompts can be configured to perform long-tail scene recognition on the intelligent driving data, and a unified output format can be set for different types of intelligent driving data (e.g., different times, different weather conditions, and different vehicle models), thereby improving the versatility of the visual language model in recognizing different types of intelligent driving data. The system parses the recognition results output by the visual language model in JSON format and checks each element in the `corner_cases` array. If the `type` field of `corner_cases` matches a preset long-tail scene enumeration value (e.g., `construction_zone`, `sudden_cut-in`, etc.), and its confidence value is greater than a first pre-set confidence threshold (e.g., 0.7), then the intelligent driving data corresponding to this recognition result is considered as the selected long-tail scene data and added to the long-tail scene dataset. The threshold setting is based on extensive experimental verification to ensure high accuracy while avoiding misclassifying ordinary scenes as long-tail scenes. The system simultaneously records confidence levels and other relevant information for each long-tail scene and ordinary scene, used for subsequent scene analysis and algorithm optimization. Furthermore, the visual language model is an AI model that integrates visual (image / video) and linguistic (text) modalities. Through cross-modal semantic alignment and joint reasoning, it achieves deep understanding and generation of complex scenes. The visual language model significantly improves generalization, increases the coverage of long-tail scene data recognition, and reduces the adaptation cost for recognizing different types of autonomous driving data.

[0027] Furthermore, in one embodiment, reference is made to Figure 2 , Figure 2 This is a second flowchart illustrating an embodiment of the long-tail scenario data filtering method of this application, as shown below. Figure 2 As shown, before step S10, the following steps are included: Step S00: The visual language model is set according to preset hyperparameters to control the speed and accuracy of semantic generation by the visual language model. The preset hyperparameters include temperature, which is lower than the default temperature of the visual language model to reduce the randomness of the visual language model output.

[0028] In this embodiment, the visual language model is configured according to preset hyperparameters to control the speed and accuracy of semantic generation. By precisely setting the hyperparameters and carefully designing the prompt word structure, the accuracy and efficiency of the visual language model in filtering long-tail scene data can be greatly improved. After obtaining the designed preset hyperparameters, they can be transmitted to the visual language model via a program interface, and the hyperparameters of the visual language model can be set according to the preset hyperparameters. The preset hyperparameters include temperature, max_tokens, top_p, top_k, frequency_penalty, presence_penalty, stop, n, stream, logprobs, and seed. The temperature can be set to 0.3, which is much lower than the default temperature of 0.7 for the visual language model. This reduces the randomness of the visual language model's output, ensures stable output, and reduces the illusion of jumps. The specific recommended values ​​for other hyperparameters and their corresponding mechanisms of action are shown in Table 1 below.

[0029] Table 1.

[0030] Furthermore, in one embodiment, reference is made to Figure 3 , Figure 3 This is a schematic diagram of the third process of an embodiment of the long-tail scenario data filtering method of this application, as shown below. Figure 3 As shown, after step S20, the following steps are included: Step S30: If a new enumeration value appears in the long-tail scene category in the recognition result, and the confidence of the long-tail scene is greater than the second preset confidence, then the new enumeration value is added to the long-tail scene category.

[0031] In this embodiment, after processing all intelligent driving data, the system automatically analyzes all type values ​​appearing in the corner_cases array. The visual language model has a certain probability of outputting new long-tail scene categories. Therefore, when a new enumeration value (i.e., not belonging to the existing preset enumeration value of the long-tail scene category "type") is detected, and the confidence value of this long-tail scene is greater than a second preset confidence threshold (e.g., 0.7), the system automatically adds this new enumeration value to the enumeration value array of the long-tail scene category in the prompt words. Furthermore, the enumeration value array of the long-tail scene category in the prompt words can also be added as needed. This mechanism achieves automatic expansion of scene categories without retraining the model, significantly reducing the cost and time of scene expansion. The system also records the characteristics and frequency of occurrence of new scenes, providing data support for subsequent optimization of the scene library.

[0032] Furthermore, in one embodiment, the output format further includes scene basic information and driving suggestions, and the long-tail scene information further includes a long-tail scene description. The scene basic information, driving suggestions, and long-tail scene description are used to provide an explanation of the recognition results.

[0033] In this embodiment, continuing to refer to the above prompt example, the output format includes not only long-tail scene information (corner_cases) but also basic scene information (scene_description) and driving suggestions (driving_recommendation). Scene_description contains detailed information such as road type, weather, traffic conditions, traffic participants, and key objects. This information is strictly limited through enumerated fields to ensure the completeness and consistency of the model output. Driving_recommendation provides specific driving suggestions, including action instructions, avoiding lane changes, yielding objects, whether manual intervention is needed, and risk areas. This information together constitutes a complete "scene recognition - risk assessment - decision suggestion" reasoning chain, enabling engineers to clearly understand the basis of the model's judgments and greatly improving the system's interpretability and reliability.

[0034] Furthermore, in one embodiment, the scene basic information and driving suggestions are enumeration fields, so that the visual language model can identify each enumeration value included in the enumeration field and output the corresponding enumeration value.

[0035] In this embodiment, both the scene description and driving recommendation are designed as enumerated fields. The road type enumeration values ​​in scene description include urban road, highway, residential, rural, parking lot, construction zone, etc., while the weather enumeration values ​​include clear, foggy, rainy, snowy, cloudy, low visibility, etc. The action enumeration values ​​in driving recommendation include slow_down, stop, proceed_with_caution, change_lane, overtake, etc. By designing these fields as enumeration fields, the model must select from predefined enumeration values, thus avoiding inconsistent outputs caused by the model's free choice, ensuring the structured and predictable nature of the output results, and significantly reducing the probability of model illusions.

[0036] Furthermore, in one embodiment, the prompt word also includes an example of the recognition result.

[0037] In this embodiment, the prompt words also include carefully designed examples of recognition results, namely the content of the "example_output": {...} section in the prompt word examples mentioned above. These examples demonstrate the correct output format and the processing methods for typical scenarios. For example, an example of a common traffic sign scenario (confidence=0.4) and an example of a complex scenario (confidence=0.85) are provided, enabling the model to understand the confidence requirements of different scenarios through contextual learning. These examples not only guide the model to understand the task but also help the model establish a quantitative understanding of scenario difficulty, enabling the model to accurately distinguish between common scenarios and long-tail scenarios. The introduction of the contextual learning strategy significantly improves the model's generalization ability on unseen scenarios, achieving accurate scenario classification without the need for a large amount of labeled data.

[0038] Furthermore, in one embodiment, the visual language model is an InternVL3-78B model, which is deployed locally using the lmdeploy tool.

[0039] In this embodiment, the visual language model uses the InternVL3-78B model, an open-source, multimodal large-scale model released by Shanghai AI Lab in collaboration with SenseTime and several top universities in China. Local deployment using the lmdeploy tool avoids the cost and latency issues associated with frequent calls to cloud APIs. The InternVL3-78B model performs exceptionally well in visual question answering (VQA) capabilities, demonstrating strong performance in multiple benchmark tests, particularly scoring 906 points on OCRbench and 96.0 / 98.6 on VCR, significantly outperforming similar models. After local deployment, the system can fully leverage the general visual understanding capabilities gained from training on massive amounts of multimodal data, achieving efficient recognition of long-tail scene data from different data sources (highways, urban areas, rural areas, extreme weather, different vehicle types, etc.). The recognition accuracy is improved to over 92%, the false positive rate is reduced by 40%, and system maintenance costs are significantly reduced.

[0040] Secondly, embodiments of this application provide a method for filtering long-tail scene data, applied to a visual language model.

[0041] In one embodiment, the long-tail scenario data filtering method includes: The visual language model acquires intelligent driving data and prompt words, performs long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and outputs the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, which allows the visual language model to recognize and output the corresponding enumeration value item by item according to the enumeration value included in the enumeration field. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. If the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then the intelligent driving data corresponding to the recognition result will be used as the long-tail scene data.

[0042] In this embodiment, the execution entity is a visual language model. The program can be developed based on the existing visual language model to enable the visual language model to automatically acquire intelligent driving data and prompt words, as well as to filter long-tail scene data according to the recognition results. After filtering out the long-tail scene data, the visual language model automatically adds it to the long-tail scene dataset for subsequent scene analysis and algorithm optimization.

[0043] Thirdly, embodiments of this application also provide a long-tail scene data filtering device.

[0044] In one embodiment, reference is made to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the long-tail scene data filtering device of this application, as shown below. Figure 4 As shown, the long-tail scene data filtering device includes: The recognition module 10 is used to input intelligent driving data and prompt words into the visual language model, so that the visual language model can perform long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and output the recognition result according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, so that the visual language model can recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints and confidence guidance constraints. The filtering module 20 is used to select the intelligent driving data corresponding to the recognition result as long-tail scene data if the long-tail scene category in the recognition result is a set enumeration value and the confidence of the long-tail scene is greater than the first preset confidence.

[0045] Furthermore, in one embodiment, the long-tail scene data filtering device further includes a setting module, used for: The visual language model is configured according to preset hyperparameters to control the speed and accuracy of semantic generation by the visual language model. The preset hyperparameters include temperature, which is lower than the default temperature of the visual language model to reduce the randomness of the visual language model's output.

[0046] Furthermore, in one embodiment, the long-tail scene data filtering device further includes an addition module, used for: If a new enumeration value appears in the long-tail scene category in the recognition results, and the confidence level of the long-tail scene is greater than the second preset confidence level, then the new enumeration value is added to the long-tail scene category.

[0047] Furthermore, in one embodiment, the output format further includes scene basic information and driving suggestions, and the long-tail scene information further includes a long-tail scene description. The scene basic information, driving suggestions, and long-tail scene description are used to provide an explanation of the recognition results.

[0048] Furthermore, in one embodiment, the scene basic information and driving suggestions are enumeration fields, so that the visual language model can identify each enumeration value included in the enumeration field and output the corresponding enumeration value.

[0049] Furthermore, in one embodiment, the prompt word also includes an example of the recognition result.

[0050] Furthermore, in one embodiment, the visual language model is an InternVL3-78B model, which is deployed locally using the lmdeploy tool.

[0051] The functions of each module in the long-tail scene data filtering device correspond to the steps in the long-tail scene data filtering method embodiment, and their functions and implementation processes will not be described in detail here.

[0052] Fourthly, embodiments of this application also provide a readable storage medium.

[0053] The present application has a long-tail scene data filtering program stored on a readable storage medium, wherein when the long-tail scene data filtering program is executed by a processor, it implements the steps of the long-tail scene data filtering method described above.

[0054] The method implemented when the long-tail scene data filtering program is executed can be referred to in various embodiments of the long-tail scene data filtering method of this application, and will not be repeated here.

[0055] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0056] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0057] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0058] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0059] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0061] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for filtering data in long-tail scenarios, characterized in that, The long-tail scenario data filtering method includes: Intelligent driving data and prompts are input into a visual language model, which then performs long-tail scene recognition on the intelligent driving data according to the task set by the prompts, and outputs the recognition results according to the output format set by the prompts. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, which allows the visual language model to recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompts also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. If the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then the intelligent driving data corresponding to the recognition result will be used as the long-tail scene data.

2. The long-tail scene data filtering method as described in claim 1, characterized in that, Before inputting intelligent driving data and prompt words into the visual language model, so that the visual language model can perform long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and output the recognition result according to the output format set by the prompt words, the process includes: The visual language model is configured according to preset hyperparameters to control the speed and accuracy of semantic generation by the visual language model. The preset hyperparameters include temperature, which is lower than the default temperature of the visual language model to reduce the randomness of the visual language model's output.

3. The long-tail scene data filtering method as described in claim 1, characterized in that, If the long-tail scene category in the identification result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then after taking the intelligent driving data corresponding to the identification result as long-tail scene data, the following steps are included: If a new enumeration value appears in the long-tail scene category in the recognition results, and the confidence level of the long-tail scene is greater than the second preset confidence level, then the new enumeration value is added to the long-tail scene category.

4. The long-tail scene data filtering method as described in claim 1, characterized in that, The output format also includes basic scene information and driving suggestions. The long-tail scene information also includes a long-tail scene description. The basic scene information, driving suggestions, and long-tail scene description are used to provide an explanation of the recognition results.

5. The long-tail scene data filtering method as described in claim 4, characterized in that, The basic scene information and driving suggestions are enumerated fields, which are used by the visual language model to identify and output the corresponding enumerated values ​​according to the enumerated values ​​included in the enumerated fields.

6. The long-tail scene data filtering method as described in claim 1, characterized in that, The prompt words also include examples of recognition results.

7. The long-tail scene data filtering method as described in claim 1, characterized in that, The visual language model is the InternVL3-78B model, which is deployed locally using the lmdeploy tool.

8. A method for filtering long-tail scene data, applied to visual language models, characterized in that, The long-tail scenario data filtering method includes: The visual language model acquires intelligent driving data and prompt words, performs long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and outputs the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, which allows the visual language model to recognize and output the corresponding enumeration value item by item according to the enumeration value included in the enumeration field. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. If the long-tail scene category in the recognition result is a set enumeration value, and the confidence level of the long-tail scene is greater than the first preset confidence level, then the intelligent driving data corresponding to the recognition result will be used as the long-tail scene data.

9. A long-tail scene data filtering device, characterized in that, The long-tail scene data filtering device includes: The recognition module is used to input intelligent driving data and prompt words into the visual language model, so that the visual language model can perform long-tail scene recognition on the intelligent driving data according to the task set by the prompt words, and output the recognition results according to the output format set by the prompt words. The output format includes long-tail scene information, which includes long-tail scene category and long-tail scene confidence. The long-tail scene category is an enumeration field, so that the visual language model can recognize each enumeration value included in the enumeration field and output the corresponding enumeration value. The prompt words also include output content constraints, which include non-empty constraints, precise output constraints, safety priority constraints, traffic rule constraints, and confidence guidance constraints. The filtering module is used to select the intelligent driving data corresponding to the recognition result as long-tail scene data if the long-tail scene category in the recognition result is a set enumeration value and the confidence of the long-tail scene is greater than the first preset confidence.

10. A readable storage medium, characterized in that, The readable storage medium stores a long-tail scene data filtering program, wherein when the long-tail scene data filtering program is executed by a processor, it implements the steps of the long-tail scene data filtering method as described in any one of claims 1 to 8.

Images