A method of performing an integrity check on an image dataset and a system therefor
Patent Information
- Application Number
- CN202610445241.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-04-04
- Filing Date
- 2026-04-07
- Publication Date
- 2026-10-16
Smart Images
Figure CN122929246A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for performing integrity checks on image datasets. Background Technology
[0002] With the increasing popularity of VLFM (Visual Language Foundation Model) for many downstream tasks, it is important to collect carefully selected datasets that can be used for training, fine-tuning, or validation. For many applications in the computer vision domain, such as image retrieval and visual question answering (VQA), open-source models are trained on general images of everyday "icon" objects (located at the center of the image). To use these VLFM architectures for specific industrial applications, such as automated optical inspection (AOI) or autonomous driving (AD), domain-specific images are needed for training / fine-tuning and validation of the base model. If the domain dataset is incomplete even with reasonable coverage, the VLFM output is prone to producing illusions and giving incorrect results. Since VLFM produces both image and text embeddings, they can be used to describe variations and check the completeness of the dataset.
[0003] In autonomous driving (AD), domain-specific datasets are required that differ from general image datasets and do not share similar characteristics. These requirements include—first, training (or fine-tuning) the base model from scratch using a large dataset, or fine-tuning using a smaller, carefully selected dataset. Second, the requirement includes dataset validation, where a validation set must be selected to cover most real-world scenarios based on the final task. For AOI, having the same type of defects in the validation set is redundant and may not provide a realistic evaluation; therefore, variation in possible defects is necessary. For AD cases, it must cover rare objects that form a very small percentage of the total images collected in real-world scenarios. Finally, the base model has some limitations, such as not being able to provide reliable information on complex attributes and relationships. Because these are particularly important for AD use cases, fine-tuning using such specific cases is necessary. For this, we will need data diversity, as similar attributes and relationships may not be sufficient. To check whether the data is sufficiently diverse, there are Operational Design Domain (ODD) criteria. 1 These are standardized sets of functional specifications that connected autonomous vehicles must be able to handle. This disclosure proposes to examine the integrity of available datasets based on these ODD standards.
[0004] Existing technology Implementing ODD as single point of knowledge to support the development of automated driving It describes the requirements analysis for ODD data and the modeling methods for implementing ODD. Attached Figure Description
[0005] Embodiments of the present invention are described with reference to the following accompanying drawings: Figure 1 An AI system for performing integrity checks on an image dataset according to embodiments of the present disclosure is described.
[0006] Figure 2 A flowchart is depicted illustrating a method for performing an integrity check on an image dataset according to embodiments of the present disclosure. Detailed Implementation
[0007] The invention will now be described by way of example with reference to the accompanying drawings. Throughout the figures, the same or corresponding elements are generally indicated by the same reference numerals. These depicted embodiments should be understood as illustrative of the invention and not as limiting in any way. It should also be understood that the figures are not necessarily drawn to scale, and embodiments are sometimes illustrated by graphic symbols, dashed lines, diagrammatic representations, and partial views. In predetermined examples, details that are unnecessary for understanding the invention or that make other details imperceptible may have been omitted.
[0008] refer to Figure 1 , Figure 1 An AI system (10) for performing integrity checks on an image dataset (1) is disclosed. The AI system (10) includes at least one pre-trained AI model (3) configured to receive the image dataset (1) and at least one Operational Design Domain (ODD) condition (2) as input. The AI model extracts a set of image embeddings (5) from the image dataset (1) and a set of text embeddings (6) from the at least one ODD condition (2). The AI system (10) further includes a scoring module (4) configured to receive the extracted set of image embeddings (5) and the set of text embeddings (6). The scoring module calculates a score to represent the relationship between the set of image embeddings (5) and the set of text embeddings. Based on the score, the scoring module classifies the image dataset (1) into a set of categories (7). The categories in the set of categories (7) correspond to either the presence or absence of at least one ODD condition (2). The AI system performs integrity checks on the image dataset (1) based on the classification. The at least one ODD condition (2) includes (in a non-limiting manner) at least one of urban environment, rural environment, snowy weather, and rainy weather. The AI model (3) is pre-trained from multiple images and multiple text prompts corresponding to the multiple images. According to an embodiment of this disclosure, the pre-trained AI model (3) is a pre-trained Visual Language Foundation Model (VLFM).
[0009] The AI system (10) is described in further detail. It is important to understand some aspects of artificial intelligence (AI) technology and AI-based systems or AI systems. Those skilled in the art will expect to know the capabilities of Visual Language Foundation Models (VLFM). VLFMs are AI models designed to understand and generate both visual and textual data. These models integrate computer vision and natural language processing (NLP) to perform tasks such as image captioning, visual question answering (VQA), image text retrieval, and multimodal inference. These models learn from both images and text, allowing them to generate text based on images or find relevant images from text descriptions. The VLFMs described according to this disclosure can be based on any of the known VLFM architectures, such as CLIP, BLIP, ALIGN, Flamingo, and the like. Additionally, the phrases “training” and “fine-tuning” are limited to the meaning of training and AI models and fine-tuning AI models as known in the terminology of the AI field. The AI system (10) includes an AI model (3) trained using training data, which the AI system uses to analyze real-time data and generate appropriate results. The model can be fine-tuned in real time based on the results. It should further be understood that the term "module" as used in this disclosure refers to a separate hardware or software component that interacts with a larger system. For example, a scoring module, as mentioned below, could be software residing in a system or the cloud, or embodied within an electronic chip. Such a chip is a dedicated silicon chip that incorporates AI technology and is used for machine learning. Other components of this disclosure will become clear upon further reading.
[0010] According to this disclosure, the AI system (10) is implemented in a computing device having dedicated components optimized for parallel processing, memory access, and efficient deep learning computation. The computing device may include a general-purpose processor that handles control logic, pre / post-processing, and scheduling AI tasks, and is adapted for data loading, model orchestration, and running lightweight AI models. The computing device may further include a graphics processing unit (GPU) optimized for parallel processing and matrix operations in deep learning, as well as for training and inference in AI models (3) such as transformers, diffusion models, and segmentation networks. The computing device may further include a tensor processing unit (TPU) for deep learning workloads. The computing device may further include a high-bandwidth memory (HBM) essential for large VLFM-based AI models such as latent diffusion models and transformers, as well as for fast data access for matrix multiplication and attention mechanisms. The computing device may further include matrix multiplication cores (tensor cores, AMX, XMX, etc.) for AI matrix operations such as attention layers, convolutions, and diffusion steps. Additionally, the computing device may have other basic components, such as interconnects, network buses, and the like. All inputs received from the AI system and outputs generated from the AI system are processed through the corresponding input / output interfaces of the computing device. It should be understood that the AI system (10) described according to this disclosure is an aggregation of the AI model (3) and the scoring module (4), which are combined to implement the methods described herein. As an example, these components are executed as program code in known frameworks such as pyTorch, tensorflow, hugging Face Diffusers, labml.ai, and the like.
[0011] The components of the AI system (10) are further described. The AI system (10) includes at least one pre-trained AI model (3). According to an embodiment of this disclosure, the pre-trained model is a VLFM as described above. The VLFM is pre-trained from multiple images and multiple text cues corresponding to the multiple images. The at least one pre-trained AI model (3) is configured to receive an image dataset (1) and at least one Operational Design Domain (ODD) condition (2) as input. The AI model extracts a set of image embeddings (5) from the image dataset (1) and a set of text embeddings (6) from the at least one ODD condition (2). The Operational Design Domain (ODD) condition criterion is that the connected autonomous vehicle must be able to handle 2 (See references) A standardized set of functional specifications.
[0012] The AI system (10) includes a scoring module (4). The scoring module (4) according to this disclosure is a scorer configured to receive an extracted set of image embeddings (5) and a set of text embeddings (6). The scoring module (4) ensures that the image dataset (1) covers a wide range of concepts, objects, and scenes. It measures the distribution of objects, actions, and scenes across the dataset. For example, a dataset containing “traffic signals,” “cars,” and “pedestrians” is diverse, while a dataset containing only “cars” is not. The scoring module (4) is configured to compute scores to represent the relationship between the set of image embeddings (5) and the set of text embeddings. According to this disclosure, the scoring module obtains the dot product between the extracted set of image embeddings (5) and the set of text embeddings (6) to obtain a score. The scoring module classifies the image dataset (1) into a set of categories (7) based on the scores. The categories in the set of categories (7) correspond to either the presence or absence of at least one ODD condition (2), and an integrity check is performed on the image dataset (1) based on the classification.
[0013] The method for performing integrity checks on the image dataset (1) is further described. (See references...) Figure 2 , Figure 2 A flowchart is depicted of a method (100) for performing integrity checks on an image dataset (1). This method is implemented by an AI system (10), which includes at least one pre-trained AI model (3) and a scoring module (4), as described above. Figure 1 As described in [the text].
[0014] According to this disclosure, reference Figure 2 The method comprises several steps from step 101 to step 104, as well as sub-steps (104a) and (104b). The steps are listed below: Step (101) is to input the image dataset (1) into at least one pre-trained AI model (3). Step (102) is to input at least one Operational Design Domain (ODD) condition (2) into the pre-trained AI model (3). Step (103) is to extract an image embedding set (5) from the input image dataset (1) and a text embedding set (6) from at least one ODD condition (2). Step (104) is to input the extracted image embedding set and text embedding set (6) into a scoring module (4) to perform an integrity check on the image dataset (1). Step (104) further includes (104a) calculating a score representing the relationship between the image embedding set (5) and the text embedding set, and (104b) classifying the image dataset (1) into a category set (7) based on the score, wherein the categories in the category set (7) correspond to either the presence or absence of at least one ODD condition (2).
[0015] These steps are described in further detail with reference to working examples. According to this disclosure, step (101) is to input an image dataset (1) into at least one pre-trained AI model (3), and step (102) is to input at least one Operational Design Domain (ODD) condition (2) into the pre-trained AI model (3). According to one example, the VLFM is pre-trained using domain-specific data (both images and captions / text prompts). The fine-tuned model will thus have a better ability to examine images (provided in text form) against the ODD condition. The VLFM can be incorporated into hardware (such as a camera) mounted on a vehicle. Road scenes from moving vehicles are captured by camera sensors. The captured images are sent for inference from the fine-tuned (pre-trained) VLFM, which can be deployed on edge computing software.
[0016] According to this disclosure, step (103) involves extracting an image embedding set (5) from the input image dataset (1) and a text embedding set (6) from at least one ODD condition (2). Step (104) involves inputting the extracted image embedding set and text embedding set (6) into a scoring module (4) to perform an integrity check on the image dataset (1). Step (104) further includes (104a) calculating a score to represent the relationship between the image embedding set (5) and the text embedding set, and (104b) classifying the image dataset (1) into a category set (7) based on the score. According to an example, the image dataset and the ODD condition (as text) are input into a VLFM to obtain image and text embeddings from a pre-trained VLFM. These image and test embeddings are sent to a scorer, which checks whether the data is diverse and complete using the set ODD condition. The proposed method for checking the integrity of an available dataset uses both the text embeddings and the image embeddings (5) to formulate a score (dot product). Based on this score, the dataset can be classified. According to one example, the ODD condition may include conditions with complex attributes describing objects in the scene. According to another example, the ODD condition (2) may specify the semantics used for the description. According to one example, the scoring module (4) may classify the dataset based on a measurable metric of the ODD representation. In other words, the extracted image and text embeddings (6) are fed into the scoring module (4) to evaluate their relationship. A similarity score is calculated between each image embedding and the text embedding of the ODD condition. The dataset is classified based on the calculated score. According to one example, the classification may be the presence or absence of the ODD condition (2). The presence of the ODD condition (2) is if the image dataset (1) contains enough images to represent the ODD condition (2), while the absence of the ODD condition (2) is when the dataset lacks enough images to depict the ODD scene.
[0017] According to another example, if a user wants to check whether there are enough images in the autonomous driving dataset for the condition "foggy road", the method would involve extracting embeddings from all images in the dataset and extracting embeddings for the text "foggy road" set as the ODD condition (2). The scoring module (4) calculates the similarity between the images and the text. It then classifies the dataset as having enough or lacking images of "foggy road". The classification can be threshold-based, where a predefined similarity threshold is set to determine the completeness of the dataset. In another example, a cross-attention mechanism can be used to improve the matching between images and text. According to one example, based on the output, the AI system (10) can suggest to the user that the ODD condition is missing and recommend additional data collection. Once the user knows the missing ODD condition, they can train the VLFM based on the data representing the insufficiency / missing data. This newly trained VLFM (trained on the complete dataset) can be deployed where the required hardware is a camera sensor rig mounted on the vehicle and an edge computing mechanism for inference. The camera for image acquisition and the edge computing hardware system for inference can be mounted on the vehicle, making decision-making possible in near real-time. The results of this disclosure (VLFM trained on the full dataset) can be deployed not only to warn of the presence of obstacles, but also to display complex attributes and relevant information, enabling drivers to avoid collisions in autonomous driving use cases.
Claims
1. A method for performing integrity checks on an image dataset (1), said method being implemented by an AI system (10) comprising at least one pre-trained AI model (3) and a scoring module (4), said method comprising the following steps: - Input the image dataset (1) into the at least one pre-trained AI model (3). Its features are, - Input at least one Operational Design Domain (ODD) condition (2) into the pre-trained AI model (3); - Extract an image embedding set (5) from the input image dataset (1) and extract a text embedding set (6) from the at least one ODD condition (2); - Input the extracted image embedding set and text embedding set (6) into the scoring module (4) to perform an integrity check on the image dataset (1) in the following manner: - Calculate the score representing the relationship between the image embedding set (5) and the text embedding set. - The image dataset (1) is classified into a category set (7) based on the scores, wherein the categories in the category set (7) correspond to either the presence or absence of the at least one ODD condition (2).
2. The method according to claim 1, wherein the at least one ODD condition (2) comprises at least one of the following: -Urban environment, -Rural environment, -Snowy weather, - Rainy weather.
3. The method according to claim 1, wherein, The AI model is trained using multiple images and multiple text prompts corresponding to the multiple images (3).
4. The method according to claim 1, wherein, The pre-trained AI model (3) is a pre-trained visual language foundation model (VLFM).
5. An AI system (10) for performing integrity checks on an image dataset (1), the AI system (10) comprising: - At least one pre-trained AI model (3) is configured as follows: - Receive an image dataset (1) and at least one Operation Design Domain (ODD) condition (2) as input, and - Extract an image embedding set (5) from the image dataset (1) and extract a text embedding set (6) from the at least one ODD condition (2); the AI system (10) is characterized in that, - The scoring module (4) is configured as follows: - Receive the extracted image embedding set (5) and text embedding set (6). - Calculate the score representing the relationship between the image embedding set (5) and the text embedding set (6), - Based on the scores, classify the image dataset (1) into a category set (7), wherein the categories in the category set (7) correspond to either the presence or absence of the at least one ODD condition (2), and - Perform an integrity check on the image dataset (1) based on the classification.
6. The AI system (10) according to claim 5, wherein the at least one ODD condition (2) includes at least one of the following: -Urban environment, -Rural environment, -Snowy weather, - Rainy weather.
7. The AI system (10) according to claim 5, wherein, The AI model (3) is pre-trained using multiple images and multiple text prompts corresponding to the multiple images.
8. The AI system (10) according to claim 5, wherein, The pre-trained AI model (3) is a pre-trained visual language foundation model (VLFM).