Image data mining method and system for corner cases in automatic driving field

Abstract

The invention discloses an image data mining method and system for corner cases in the field of automatic driving, and belongs to the technical field of data processing. A data mining demand corresponding to an image data set is determined by using a natural language; the method comprises the following steps of: disassembling a data mining demand through a large language model to obtain demand elements of a quantity requirement and a plurality of image content requirements; based on the historical mining data, determining a mining tool of each demand element; the method comprises the following steps: determining a mining priority corresponding to a mining tool of each demand element by acquiring the proportion of the number of pictures with each demand element feature in an image data set to the total number of the pictures; the smaller the proportion is, the higher the mining priority is; and performing data mining according to the mining priority to obtain a data mining result. According to the method, the demand elements related to the extreme scenes can be accurately positioned through the natural language determination demand and the large language model disassembly demand, so that the mining process is more targeted, blind acquisition is avoided, and the acquisition difficulty is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to an image data mining method and system for corner cases in the field of autonomous driving. Background Technology

[0002] In the research and application of autonomous driving technology, the key to moving the system from "usable" to "reliable" lies in improving the robustness and generalization ability of the algorithm. Edge cases, as low-probability but high-risk scenarios in autonomous driving systems, are key factors affecting system safety and reliability. Edge case data mining refers to the systematic approach of accurately identifying, extracting, and analyzing rare, complex, or extreme scenarios from massive amounts of driving data. These scenarios often exceed the scope of conventional testing but are crucial for optimizing the performance of autonomous driving systems.

[0003] Key aspects of autonomous driving systems, such as perception, decision-making, and control, all require a large amount of driving scenario images that meet specific conditions as data support, especially various peripheral cases that deviate from conventional driving scenarios. By collecting, mining, and labeling this peripheral case data, the algorithm's ability to cope with complex situations can be effectively improved, thereby ensuring the safety of autonomous driving systems in actual operation.

[0004] However, traditional data mining methods mainly rely on manual collection, a process that is not only time-consuming but also susceptible to inaccurate understanding due to differences in individual interpretations. This is particularly true in the field of autonomous driving technology, where niche cases need to cover various extreme scenarios, such as rare weather, unusual lighting conditions, and abnormal traffic situations. The rarity and unpredictability of these extreme scenarios, coupled with the complexity of scenario simulation, significantly increases the difficulty of manual data collection. Summary of the Invention

[0005] To address the problems existing in the above-mentioned fields, this invention proposes an image data mining method and system for edge and corner cases in the field of autonomous driving. By decomposing the data mining requirements and planning the mining tools and data mining in sequence according to the decomposed requirements, the mining process becomes more targeted, avoids blind collection, and reduces the difficulty of collection.

[0006] To address the aforementioned technical problems, this invention discloses an image data mining method for corner cases in the field of autonomous driving, comprising the following steps: We acquire image datasets of edge cases to be mined, and based on the potential demand for image datasets in corresponding image processing tasks in the field of autonomous driving, we use natural language to determine the data mining requirements corresponding to the image datasets. We then decompose the data mining requirements through a large language model to obtain the requirement elements of quantity requirements and multiple image content requirements. Based on historical mining data, determine the mining tool for each demand element; by obtaining the proportion of the number of images with the characteristics of each demand element in the image dataset to the total number of images, determine the mining priority of the mining tool for each demand element, where the smaller the proportion, the higher the mining priority. Data mining is performed based on the mining priority of the mining tool corresponding to each demand element to obtain data mining results; and a mining dataset is formed based on the quantity requirements and the data mining results.

[0007] Preferably, the data mining requirements are broken down using a large language model to obtain multiple requirement elements; wherein, the requirement elements are instructions executed by the mining tool, specifically including: The large language model, which serves as the inference engine, is trained based on the few-shot learning paradigm in the prompt word engineering. The data mining requirements are broken down into quantity requirements and multiple image content requirements using a well-trained large language model. The data mining requirements are broken down into factors such as the number of images, environmental conditions, target objects, and density by using a large language model. The number of images is considered as the quantity requirement, while environmental conditions, target objects, and density are considered as the requirements for image content.

[0008] Preferably, the tool for determining each demand element based on historical mining data specifically includes: Based on historical mining data, a small number of examples are provided to guide the large language model to understand the intent of the input data mining needs. Multiple demand elements are obtained by decomposing the data, including generating corresponding text search keywords, image search keywords, model validation instructions, and counter configurations. The mining tools corresponding to the text search keywords, image search keywords, model verification instructions, and counter configurations include text search models, image search models, verification models, and counters.

[0009] Preferably, the step of determining the mining priority of each required element by obtaining the proportion of images with each required element feature in the image dataset relative to the total number of images, wherein the smaller the proportion, the higher the mining priority, specifically includes: Based on the mining tools determined for each requirement element, and combining the functions and characteristics of different mining tools, the number of images in the image dataset that possess the characteristics of each requirement element is obtained through mining. The proportion of images containing each required feature in the image dataset to the total number of images is used as the mining efficiency. Based on the mining efficiency, the mining priority corresponding to the mining tool for each demand element is determined. The smaller the proportion, the smaller the mining efficiency value, indicating a higher mining priority.

[0010] Preferably, the step of performing data mining based on the mining priority corresponding to the mining tool for each demand element to obtain data mining results; and forming a mining dataset based on the quantity requirements and the data mining results, specifically including: When the requirements include the number of images, environmental conditions, target objects, and density, and the mining priority of the corresponding mining tools is ranked as follows: text search models for semantic search of environmental conditions have the highest priority; image search models and counters for target object recognition and image count have the second highest priority; and verification models for density verification have the lowest priority. A text search model is used to filter high-priority environmental condition semantic relevance, generating an image-level vector for each image in the image dataset; based on the image-level vector, images that are semantically related to environmental conditions are identified, and the image dataset is then filtered with high priority. An image search model is used to identify medium-priority target objects and count the number of images, generating object-level vectors corresponding to high-priority filtered images. Based on the object-level vectors, unsupervised or semi-supervised learning is used for detection and recognition to obtain the bounding boxes and category information of each target object in the high-priority filtered images. At the same time, a counter is used to count the number of target objects detected in each initially filtered image. When the number of target objects reaches a preset threshold, a medium-priority image is obtained. A verification model is used to verify the density of the acquired low-priority images and determine the relative positions of each target object. When the relative positions reach a preset threshold of density, images that meet the quantity requirements are output as the mining dataset.

[0011] Preferably, the step of using a text search model to filter high-priority environmental condition semantic relevance and generating an image-level vector for each image in the image dataset specifically involves generating a text embedding vector related to environmental condition semantics using the CLIP model.

[0012] Preferably, the step of using an image search model to identify medium-priority target objects and count the number of images to generate object-level vectors corresponding to high-priority filtered images specifically involves generating image embedding vectors corresponding to target object identification using the DINO model.

[0013] Preferably, the step of verifying the density of the acquired low-priority images using a verification model further includes: Obtain the similarity between the text embedding vector and all image embedding vectors; Based on the verification model, logical consistency and knowledge verification are performed on images that meet the similarity threshold requirements.

[0014] Preferably, the output of images that meet the quantity requirement further includes: External feedback is provided to the output images that meet the stated quantity requirement, wherein: When the number of images output meets the preset image number threshold, the number of images corresponding to the mining priority is reviewed. When the number of output images that meet the quantity requirements does not meet the preset image quantity threshold, the required elements are re-decomposed, the mining tools are planned, and the data mining is carried out again.

[0015] Preferably, it also includes an image data mining system for corner cases in the field of autonomous driving, comprising: The requirement element decomposition module is used to obtain the image dataset of the peripheral cases to be mined. Based on the potential demand for image datasets in the corresponding image processing tasks in the field of autonomous driving, it uses natural language to determine the data mining requirements corresponding to the image dataset. The data mining requirements are decomposed through a large language model to obtain the requirement elements of quantity requirements and multiple image content requirements. The mining priority determination module is used to determine the mining tool for each demand element based on historical mining data. By obtaining the proportion of the number of images with the characteristics of each demand element in the image dataset to the total number of images, the mining priority of the mining tool corresponding to each demand element is determined. The smaller the proportion, the higher the mining priority. The data mining module is used to perform data mining according to the mining priority of the mining tools corresponding to each demand element, and to obtain data mining results; based on the quantity requirements and the data mining results, a mining dataset is formed.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes an image data mining method for edge cases in the field of autonomous driving. It uses natural language to determine the data mining requirements corresponding to the image dataset, and decomposes these requirements into multiple requirement elements using a large language model. Based on historical mining data, it determines the mining tool for each requirement element. By obtaining the proportion of images in the image dataset that possess the characteristics of each requirement element out of the total number of images, it determines the mining priority corresponding to the mining tool for each requirement element; the smaller the proportion, the higher the mining priority. Data mining is then performed according to the mining priority. Specifically, for extreme scenarios such as rare weather, special lighting, and abnormal traffic conditions covered in edge cases of autonomous driving, traditional manual data collection suffers from limitations in comprehensive coverage and high costs. This method, by determining requirements through natural language and decomposing requirements using a large language model, can accurately locate the requirement elements related to these extreme scenarios, making the mining process more targeted, avoiding blind collection, and reducing the difficulty of data collection. Mining tools determined based on historical data are more aligned with actual needs and can more accurately extract relevant corner case image data. Data mining is performed according to the mining priority of each required element; the smaller the proportion, the scarcer the feature of that element in the image dataset, and therefore it should be mined first to balance the data efficiency of each layer of the funnel. This priority-based data mining approach makes the mining process more targeted, prioritizing corner case required elements that are difficult or costly to collect using traditional methods, rationally allocating mining resources, and avoiding wasting too much time on easily collected data, thereby improving overall mining efficiency. Attached Figure Description

[0017] Figure 1 This is a flowchart of the image data mining method for corner cases in the field of autonomous driving proposed in this invention; Figure 2 A logical diagram illustrating data mining using a single model, provided for an embodiment of the present invention; Figure 3 A schematic diagram illustrating the logic of data mining using multiple models, provided for an embodiment of the present invention; Figure 4 This is a schematic diagram of the intelligent agent framework provided in an embodiment of the present invention. Detailed Implementation

[0018] The following will refer to the appendices in the embodiments of the present invention. Figures 1-4 The technical solutions in the embodiments of the present invention will be clearly and completely described. It should be understood that the terminology used in the present invention is only for describing particular implementation methods and is not intended to limit the present invention.

[0019] Example like Figure 1As shown, this invention proposes a method for mining image data from corner examples in the field of autonomous driving, comprising the following steps: S1: Obtain the image dataset of the corner cases to be mined. Based on the potential demand for image datasets in the corresponding image processing tasks in the field of autonomous driving, use natural language to determine the data mining requirements corresponding to the image dataset. Decompose the data mining requirements through a large language model to obtain the requirement elements of quantity requirements and multiple image content requirements. S2: Based on historical mining data, determine the mining tool for each demand element; by obtaining the proportion of the number of images with the characteristics of each demand element in the image dataset to the total number of images, determine the mining priority corresponding to the mining tool for each demand element, where the smaller the proportion, the higher the mining priority. S3: Perform data mining according to the mining priority of each demand element's mining tool to obtain data mining results; form a mining dataset based on quantity requirements and data mining results.

[0020] Specifically, in step S1, the image dataset of the corner cases to be mined includes, but is not limited to, real road test data, simulation data, Internet data, etc.

[0021] Based on the potential demand for this image dataset in the field of autonomous driving image processing tasks, natural language is used in the large language model to input the data mining requirements corresponding to this image dataset.

[0022] For example, when the potential demand is for the detection and recognition of specific targets under specific environmental and lighting conditions, the large language model can use natural language to input "mining 2000 images taken in low light or complete darkness that contain a large number of vulnerable road users (VRUs)" or "mining 1500 images taken in heavy rain that contain moving vehicles and road signs".

[0023] A potential demand scenario is the understanding of specific traffic scenarios and emergencies. In a large language model, natural language input can be used to "collect images containing complex traffic scenarios, such as intersections, roundabouts, and congested road sections. The goal could be 500 images of intersections containing multiple interacting vehicles and pedestrians, and labeling the behavioral intentions of each participant (e.g., going straight, turning left, waiting)" or "mining images containing emergencies, such as traffic accidents, road construction, and the sudden appearance of animals."

[0024] For extreme scenarios such as rare weather, special lighting, and abnormal traffic conditions covered in autonomous driving edge cases, traditional manual data collection is difficult to fully cover and is costly. The method proposed in this invention uses natural language to determine the data mining requirements corresponding to the image dataset. With the convenience of natural language processing, the direction of requirements can be quickly and relatively accurately identified, reducing the time invested by humans in the requirements analysis stage, thereby reducing labor costs.

[0025] By using a large language model, data mining requirements are broken down into multiple requirement elements, including quantity requirements and multiple image content requirements. This cleverly transforms the complex requirements expressed by users in natural language into a series of instructions that mining tools can directly understand and execute. The large language model can quickly process large amounts of text information, efficiently decomposing complex requirements into specific, actionable requirement elements. This avoids the tedious process of manually analyzing requirements one by one, greatly shortening the requirement decomposition time, reducing the workload of manual operations, and lowering time and labor costs.

[0026] Among them, the large language model is trained based on the few-shot learning paradigm in the prompt word engineering.

[0027] For example, when the input data mining requirement is "to mine 2000 images taken in low light or complete darkness that contain a large number of vulnerable road users (VRUs)," the large language model can break it down into the following requirement elements: Number of images: at least 2000 Environmental conditions: Night (low light conditions) Target object: Vulnerable road users Denseness: Multiple vulnerable road users exist in the same image and are relatively close to each other, forming a certain density.

[0028] In some embodiments, a large language model serving as the inference engine can be trained based on the few-shot learning paradigm in Prompt Engineering. The trained large language model can then be used to decompose data mining requirements into multiple requirement elements, including quantity requirements and multiple image content requirements.

[0029] Based on historical mining data, by providing a small number of examples, it is possible to guide a large language model to understand the intent of the user's input data mining needs and automatically generate corresponding text search keywords, image search keywords, model validation instructions, counter configurations, etc., as multiple demand elements obtained through decomposition, thereby achieving effective conversion of natural language into machine-executable tasks.

[0030] Among them, the mining tools corresponding to text search keywords, image search keywords, model validation instructions, and counter configurations include text search models, image search models, validation models, and counters.

[0031] A text search model used to perform text retrieval; Image search model for targeted searching of visual content; Verification models are used for logical consistency and knowledge-based verification; and A counter is used to quantize a specific data point.

[0032] Based on historical mining data, the mining tools for each demand element are determined. By utilizing past experience and data, the appropriate mining tools can be matched quickly and accurately for each demand element, improving the efficiency of mining tool selection and thus enhancing the overall data mining efficiency.

[0033] Mining tools determined based on historical data are more aligned with actual needs and can more accurately extract corner and edge case image data that meets the requirements. Compared to manually selecting mining tools arbitrarily, this method improves the accuracy of data mining, making the mining results more suitable for the actual needs of the autonomous driving field for corner and edge case image data.

[0034] Based on the mining tools corresponding to each determined requirement element, and combining the functions and characteristics of different mining tools, the number of images in the image dataset that possess the characteristics of each requirement element is obtained. The proportion of images with the required features in the image dataset to the total number of images is used as the mining efficiency. Based on the magnitude of mining efficiency, the mining priority of each demand element is determined. This strategy is a typical long-tail feature-first optimization approach in scenarios such as feature mining and data filtering in large models. Its core purpose is to improve the overall data efficiency of the funnel.

[0035] The smaller the percentage, the lower the mining effectiveness value, indicating that the characteristic of that demand element is more scarce, and therefore the higher the mining priority. This prioritization method makes the mining process more targeted, enabling priority processing of peripheral demand elements that are difficult or costly to collect using traditional methods, rationally allocating mining resources, avoiding wasting too much time on easily collected data, and thus improving overall mining efficiency.

[0036] Features with a small percentage of representation (corresponding to a small number of images possessing the required feature) are easily overshadowed by mainstream features with a high percentage of representation during model training or data processing, resulting in low recognition accuracy and a high false negative rate for these features. Prioritizing the mining of these features can compensate for the model's weaknesses and avoid "a few features dragging down overall efficiency."

[0037] The core objective of a funnel is to filter effective data layer by layer. If the identification efficiency of low-proportion features is not improved, these data will be mistakenly filtered out in the funnel, resulting in the loss of effective data. Prioritizing the optimization of the mining capability of low-proportion features allows each funnel layer to accurately capture all types of data, rather than just capturing mainstream data, ultimately ensuring the efficiency of the overall data flow.

[0038] In this embodiment, the mining tools mentioned above include, but are not limited to, a text search model for performing precise text retrieval, an image search model for targeted visual content search, a verification model responsible for logical consistency and knowledge verification, and a counter for quantifying specific data points.

[0039] Correspondingly, when the demand elements include the number of images, environmental conditions, target objects, and density, and the mining priority of the corresponding mining tools is ranked as follows: text search models for semantic search of environmental conditions have the highest priority; image search models and counters for target object recognition and image count have the next highest priority; and verification models for density verification have the lowest priority. A text search model is used to filter high-priority environmental condition semantic relevance, generating an image-level vector for each image in the image dataset; based on the image-level vector, images that are semantically related to environmental conditions are identified, and the image dataset is then filtered with high priority. An image search model is used to identify medium-priority target objects and count the number of images, generating object-level vectors corresponding to high-priority filtered images. Based on the object-level vectors, unsupervised or semi-supervised learning is used for detection and recognition to obtain the bounding boxes and category information of each target object in the high-priority filtered images. At the same time, a counter is used to count the number of target objects detected in each initially filtered image. When the number of target objects reaches a preset threshold for the number of images, medium-priority images are obtained. A validation model is used to validate the density of the acquired low-priority images and determine the relative positions of each target object. When the relative positions reach a preset threshold for density, the number of images that meet the mining priority of the mining tool are output as the data mining results.

[0040] The method proposed in this invention comprehensively decomposes requirements using a large language model and mines them sequentially according to priority, ensuring that each requirement element is fully processed and avoiding the omission of important peripheral case information. The data obtained in this way is more comprehensive and complete, has higher practicality, and helps improve the robustness and generalization ability of autonomous driving algorithms.

[0041] In this embodiment, the verification model and the text search model used in the mining tool can be the same model, and the verification model used in the mining tool and the large language model that serves as the inference engine of the agent framework can also be the same model.

[0042] Furthermore, the text search model for performing precise text retrieval, the image search model for targeted visual content search, and the verification model responsible for logical consistency and knowledge verification can all include multiple different models at the same time, so that the large language model, which serves as the inference engine of the intelligent agent framework, can be selected as needed.

[0043] Taking the input data mining requirement of "mining 2000 images containing dense VRUs at night" as an example, the following mining tools can be determined based on the decomposed requirement elements: The Distillation with No Labels (DINO) model can be used to extract visual feature vectors at both the body and image levels, and can also be used as an image search model.

[0044] The Contrastive Language-Image Pre-training (CLIP) model can be used to extract multimodal feature vectors at both the object and image levels, including visual and text vectors. It can also be used as an image search model or a text search model.

[0045] The Qwen2.5-VL model (Alibaba Cloud Tongyi Qianwen Flagship Visual Language Model) has visual understanding capabilities and can be used to determine whether the content of an image matches a given natural language description. It can also be used as a validation model or a text search model.

[0046] A counter can be used to count the number of images that meet specific conditions.

[0047] Based on the established excavation tools, and considering the functions and characteristics of different tools, the priority of each tool can be determined, and the excavation steps can be planned accordingly.

[0048] For example, when performing high-priority initial screening, the image-level vectors generated by the CLIP model can be used to initially screen a large number of images to find images that are semantically related to concepts such as "night", "vehicles", "many people", "pedestrians", "bicycles", "motorcycles", and "scooters".

[0049] The CLIP model performs well in understanding the semantic relevance between text and images, and can quickly filter out irrelevant images.

[0050] When performing fine-grained identification and counting of medium-priority target objects, the object-level vectors generated by the DINO model can be used to detect and identify various objects (such as vehicles, pedestrians, bicycles, etc.) in the images that have been initially screened.

[0051] The DINO model performs well in unsupervised or semi-supervised object detection, providing bounding boxes and category information for objects.

[0052] Simultaneously, a counter can be used to count the number of vulnerable road users detected in each image. If the number of vulnerable road users reaches the "dense" standard (for example, exceeding a certain preset threshold), the image can be considered to have initially met the requirements.

[0053] For the images selected through the first two steps, the Qwen2.5-VL model's validation tool can be used for more detailed evaluation.

[0054] The Qwen2.5-VL model possesses stronger multimodal understanding capabilities, enabling it to better comprehend the meaning of "dense" and the relative positions of objects. This allows for more accurate determination of whether a scene is truly "nighttime" and whether the spatial relationship between vehicles and vulnerable road users matches the description of "many nearby."

[0055] If a more precise determination of the relationships between objects or a more detailed classification is needed, the object-level vector generation function of the CLIP or DINO model can be used again to extract features of specific objects for similarity comparison or more detailed classification. For example, this can be used to differentiate between different types of vulnerable road users.

[0056] Step S4 further includes: providing external feedback to the number of images output that meet the mining priority of the mining tool, wherein: When the number of images output meets the preset image number threshold, the number of images corresponding to the mining priority is reviewed. When the number of output images that meet the quantity requirements does not meet the preset image quantity threshold, the required elements are re-decomposed, the mining tools are planned, and the data mining is carried out again.

[0057] This invention also proposes an image data mining system for corner cases in the field of autonomous driving, comprising: The requirement element decomposition module is used to obtain the image dataset of the peripheral cases to be mined. Based on the potential demand for image datasets in the corresponding image processing tasks in the field of autonomous driving, it uses natural language to determine the data mining requirements corresponding to the image dataset. The data mining requirements are decomposed through a large language model to obtain the requirement elements of quantity requirements and multiple image content requirements. The mining priority determination module is used to determine the mining tool for each demand element based on historical mining data. By obtaining the proportion of the number of images with the characteristics of each demand element in the image dataset to the total number of images, the mining priority of the mining tool corresponding to each demand element is determined. The smaller the proportion, the higher the mining priority. The data mining module is used to perform data mining according to the mining priority of the mining tools corresponding to each demand element, and to obtain data mining results; based on the quantity requirements and the data mining results, a mining dataset is formed.

[0058] In summary, the image data mining method and system proposed in this invention decompose the data mining requirements and plan the mining tools and perform data mining in sequence according to the decomposed requirements, making the mining process more targeted, avoiding blind collection, and reducing the difficulty of collection.

[0059] like Figure 2 As shown, taking the image data mining of corner cases in the field of autonomous driving using a single multimodal model as an example, a large-scale multimodal image dataset can be input, such as the Multimodal Understanding and Generation Evaluation (MUGE) dataset, which can cover a variety of Chinese scene images.

[0060] Furthermore, a multimodal model can be used to extract features from images in an image dataset and generate image embedding vectors; alternatively, a multimodal model can be used to extract features from text embeddings required for data mining and convert them into text embedding vectors.

[0061] After feature extraction, the text embedding vectors and all image embedding vectors can be indexed using a vector database service and stored in the vector database. Utilizing the efficient similarity search function of the vector database, the similarity between the text embedding vectors and all image embedding vectors can be calculated. Furthermore, for images meeting the similarity threshold, further logical consistency and knowledge verification can be performed using a validation model. When the required elements derived from data mining needs include the number of images, a counter can be used to count the images that meet the requirements, thereby outputting image data with a quantity that meets the data mining needs.

[0062] Because each model has different strengths in different types of embeddings, for example, CLIP models focus more on semantics, while Bootstrapping Language-Image Pre-training (BLIP) models focus more on details.

[0063] Therefore, refer to Figure 3 It can simultaneously use the Clip model, Blip model, and Multimodal Pretraining with Language Understanding and Generation (mPLUG model) to process the encoding of image embedding vectors, and simultaneously use the Qwen-VL model and the InternVL model to process the text encoding of text embedding vectors.

[0064] Similarly, the efficient similarity search function of the vector database can be used to calculate the similarity between the text embedding vector and all image embedding vectors, and sort them from high to low similarity to output a preset number of images with the highest similarity ranking (Top K Images).

[0065] like Figure 4 As shown, the image data obtained through mining can also be fed back externally and internally based on the architecture of the intelligent agent framework to ensure that the mined image data strictly meets the data mining requirements proposed in step S1.

[0066] Specifically, it can perform self-reflection based on external feedback from the environment and internal feedback from the estimator (large language model) itself, and store the reflection text in long-term memory. When the executor is executed, it can choose to output the currently mined image or re-mining the data based on the reflection text and feed it back to short-term memory.

[0067] In this embodiment, external feedback includes, for example, reviewing the mined images and, if determined that they do not meet a preset threshold, inputting instructions to re-decompose the requirement elements, plan the mining tools, and perform data mining. Internal feedback includes, for example, re-decomposing the requirement elements, planning the mining tools, and performing data mining when the number of mined images is less than a preset threshold.

[0068] This embodiment can also export the final mining results after verification and reflection as a JSON (JavaScript Object Notation) file and output it to the user.

[0069] This embodiment also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it can implement the above-described image data mining method.

[0070] This embodiment also provides a readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the image data mining method described above.

[0071] The method proposed in this invention uses natural language input to define data mining requirements and performs mining based on those requirements, which reduces the difficulty of use and better meets user needs. By automatically performing data mining based on the input requirements, it can intelligently identify, filter, and extract valuable image data that meets specific needs from massive amounts of existing data. This reduces data acquisition costs and reliance on large-scale manual collection. It also helps accelerate data iteration and achieve coverage of a wider range of scenarios.

[0072] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0073] Furthermore, unless otherwise stated, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. All references to this specification are incorporated by way of citation to disclose and describe methods relating to those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

Claims

1. A method for image data mining of corner cases in the field of autonomous driving, characterized in that, Includes the following steps: Obtain the image dataset of the corner cases to be mined, and use natural language to determine the data mining requirements corresponding to the image dataset based on the potential demand for image datasets in the corresponding image processing tasks in the field of autonomous driving. By breaking down the data mining requirements using a large language model, we can obtain the requirement elements of quantity requirements and multiple image content requirements. Based on historical mining data, determine the mining tool for each demand element; by obtaining the proportion of the number of images with the characteristics of each demand element in the image dataset to the total number of images, determine the mining priority of the mining tool for each demand element, where the smaller the proportion, the higher the mining priority. Data mining is performed based on the mining priority of the mining tool corresponding to each demand element to obtain data mining results; Based on the quantity requirements and data mining results, a mining dataset is formed.

2. The image data mining method for corner cases in the field of autonomous driving according to claim 1, characterized in that, The process involves breaking down data mining requirements using a large language model to obtain multiple requirement elements; these requirement elements are instructions executed by the mining tool, specifically including: The large language model, which serves as the inference engine, is trained based on the few-shot learning paradigm in the prompt word engineering. The data mining requirements are broken down into quantity requirements and multiple image content requirements using a well-trained large language model. The data mining requirements are broken down into factors such as the number of images, environmental conditions, target objects, and density by using a large language model. The number of images is considered as the quantity requirement, while environmental conditions, target objects, and density are considered as the requirements for image content.

3. The image data mining method for corner cases in the field of autonomous driving according to claim 1, characterized in that, The mining tool for determining each demand element based on historical mining data specifically includes: Based on historical mining data, a small number of examples are provided to guide the large language model to understand the intent of the input data mining needs. Multiple demand elements are obtained by decomposing the data, including generating corresponding text search keywords, image search keywords, model validation instructions, and counter configurations. The mining tools corresponding to the text search keywords, image search keywords, model verification instructions, and counter configurations include text search models, image search models, verification models, and counters.

4. The image data mining method for corner cases in the field of autonomous driving according to claim 1, characterized in that, The process involves determining the mining priority of each mining tool for a given requirement element by obtaining the proportion of images with each required feature in the image dataset relative to the total number of images. A smaller proportion indicates a higher mining priority. Specifically, this includes: Based on the mining tools determined for each requirement element, and combining the functions and characteristics of different mining tools, the number of images in the image dataset that possess the characteristics of each requirement element is obtained through mining. The proportion of images containing each required feature in the image dataset to the total number of images is used as the mining efficiency. Based on the mining efficiency, the mining priority corresponding to the mining tool for each demand element is determined. The smaller the proportion, the smaller the mining efficiency value, indicating a higher mining priority.

5. The image data mining method for corner cases in the field of autonomous driving according to claim 4, characterized in that, The data mining results are obtained by performing data mining based on the mining priority corresponding to the mining tool for each demand element. Based on the quantity requirements and data mining results, a mining dataset is formed, specifically including: When the requirements include the number of images, environmental conditions, target objects, and density, and the mining priority of the corresponding mining tools is ranked as follows: text search models for semantic search of environmental conditions have the highest priority; image search models and counters for target object recognition and image count have the second highest priority; and verification models for density verification have the lowest priority. A text search model is used to filter high-priority environmental condition semantic relevance, generating an image-level vector for each image in the image dataset; based on the image-level vector, images that are semantically related to environmental conditions are identified, and the image dataset is then filtered with high priority. An image search model is used to identify medium-priority target objects and count the number of images, generating object-level vectors corresponding to high-priority filtered images. Based on the object-level vectors, unsupervised or semi-supervised learning is used for detection and recognition to obtain the bounding boxes and category information of each target object in the high-priority filtered images. At the same time, a counter is used to count the number of target objects detected in each initially filtered image. When the number of target objects reaches a preset threshold, a medium-priority image is obtained. A verification model is used to verify the density of the acquired low-priority images and determine the relative positions of each target object. When the relative positions reach a preset threshold of density, images that meet the quantity requirements are output as the mining dataset.

6. The image data mining method for corner cases in the field of autonomous driving according to claim 5, characterized in that, The process of using an image search model to identify medium-priority target objects and count the number of images to generate object-level vectors corresponding to high-priority filtered images specifically involves generating text embedding vectors related to environmental condition semantics through the CLIP model.

7. The image data mining method for corner cases in the field of autonomous driving according to claim 5, characterized in that, The process of using a text search model to filter high-priority environmental condition semantic relevance and generating image-level vectors for each image in the image dataset specifically involves using the DINO model to generate image embedding vectors corresponding to target object recognition.

8. The image data mining method for corner cases in the field of autonomous driving according to claim 5, characterized in that, The step of using a verification model to verify the density of the acquired low-priority images also includes: Obtain the similarity between the text embedding vector and all image embedding vectors; Based on the verification model, logical consistency and knowledge verification are performed on images that meet the similarity threshold requirements.

9. The image data mining method for corner cases in the field of autonomous driving according to claim 5, characterized in that, The output of images that meet the quantity requirement also includes: External feedback is provided to the output images that meet the stated quantity requirement, wherein: When the number of images output meets the preset image number threshold, the number of images corresponding to the mining priority is reviewed. When the number of output images that meet the quantity requirements does not meet the preset image quantity threshold, the required elements are re-decomposed, the mining tools are planned, and the data mining is carried out again.

10. An image data mining system for corner cases in the field of autonomous driving, characterized in that, include: The demand element decomposition module is used to obtain the image dataset of the corner cases to be mined. Based on the potential demand of the image dataset for the corresponding image processing tasks in the field of autonomous driving, the data mining demand corresponding to the image dataset is determined using natural language. By breaking down the data mining requirements using a large language model, we can obtain the requirement elements of quantity requirements and multiple image content requirements. The mining priority determination module is used to determine the mining tool for each demand element based on historical mining data. By obtaining the proportion of the number of images with the characteristics of each demand element in the image dataset to the total number of images, the mining priority of the mining tool corresponding to each demand element is determined. The smaller the proportion, the higher the mining priority. The data mining module is used to perform data mining according to the mining priority of the mining tools corresponding to each demand element, and to obtain data mining results; based on the quantity requirements and the data mining results, a mining dataset is formed.

Images