Method for developing a machine learning model

Abstract

Method of developing a machine learning model, comprising: training a machine-learning model 110, 112; evaluating performance of the trained model 104 using validation data 114 thereby identifying a semantic data slice where the trained model underperforms; and further training the model based on the semantic data slice 106. Generating training data may comprise generating a first candidate object annotation by an open vocabulary model (212,Fig.2), generating a second candidate object annotation by a closed vocabulary model (220,Fig.2), and determining a final annotated image and bounding box (256,258,Fig.2) based on the first and second candidate annotations. Input labels (204,Fig.2) provided to the open vocabulary model may be generated using prompt text (210,Fig.2) derived from the input image. A semantic segmentation mask (254,Fig.2) may be generated from a first (214,Fig.2) and second (222,Fig.2) candidate semantic segmentation model. Evaluating performance of the trained model may comprise generating a multimodal embedding (404,Fig.4) from a segmented validation image (414,Fig.4) and validation image embedding (410,Fig.4). The multimodal embedding is used to cluster the validation dataset performance scores (426,Fig.4) to produce the semantic data slice (406,Fig.4). Augmented images may be generated to address underperforming categories (Fig.5). Out-of-distribution images may be detected and used for training (Fig.6).

Description

TECHNICAL FIELD

[0001] The present disclosure relates to machine learning (ML) models. Specifically, the present disclosure relates to methods for developing a ML model. BACKGROUND

[0002] ML models may be implemented in a wide range of robotic and automotive products, include autonomous agents, advanced driver assistance systems, and more. These products typically require an extensive and iterative development process where the MLs are trained using large scale datasets. However, maintaining these datasets are time-consuming especially for tasks such as data labelling, augmentation upon model evaluation, and data collection for Out-of-Distribution (OOD) edge cases during deployment.

[0003] The development cycle of ML models involves crucial data science steps that are essential but extremely challenging to automate, particularly when dealing with large-scale training and target data at the terabyte level. The data collection, human labelling and data processing performed by external parties can result in label noise and extended deep learning (DL) development cycles. These processes are often time consuming and expensive.

[0004] It is inherently difficult to collect and label a sufficient number of data samples for long-tail data distributions, which often leads to lower accuracy in deployed DL models especially for less-represented classes or OOD data. Evaluation of model performance is difficult and unintuitive given large diverse datasets that are commonplace in DL. Moreover, such evaluation is often a manual process of visualising model results and manually identifying trends where the model performs poorly.

[0005] The DL development process involving data collection, labelling, processing, model development, model evaluation, model fine-tuning, model compression and model deployment are separated into isolated steps and often handled by individual teams. This results in high latency in delivering the models and an intensive manual interactive process.

[0006] Catching edge cases during production level DL implementations and the rapid pace of AI research demand an accelerated development cycle to quickly develop new models and test out newly introduced methods as soon as possible. However, meeting this demand is challenging given the current maturity level of the deep learning development process. SUMMARY

[0007] According to various embodiments, a computer-implemented method for developing a machine learning model is provided. The method includes generating a training dataset comprising a plurality of labelled images. The method further includes training a machine learning model based on the training dataset, thereby generating a trained model. The method further includes evaluating performance of the trained model based on a validation dataset, by identifying a semantic data slice from the validation dataset where the trained model underperforms. The method further includes further training the trained model based on the identified semantic data slice.

[0008] According to various embodiments, a computer program is provided. The computer program comprises instructions which, when the program is executed by a computer, cause the computer to carry out the above-mentioned method.

[0009] According to various embodiments, a computer-readable storage medium is provided. The computer-readable storage medium comprises instructions which, when executed by a computer, cause the computer to carry out the above-mentioned method.

[0010] According to various embodiments, a data processing device is provided. The data processing device comprises means for carrying out the above-mentioned method.

[0011] According to various embodiments, use of the above-mentioned method for developing a machine learning model, is provided.

[0012] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] These and other features, aspects, and advantages will become better understood with regard to the following description, appended claims, and accompanying drawings where:

[0014] FIG. 1 shows a flow chart of a method for developing a ML model, according to various embodiments.

[0015] FIG. 2 shows a flow chart of an image labelling process according to various embodiments.

[0016] FIG. 3 shows an example ofthe object voting algorithm.

[0017] FIG. 4 shows a flow chart of a model evaluation process according to various embodiments.

[0018] FIG. 5 shows a schematic diagram of the image augmentation module according to various embodiments.

[0019] FIG. 6 shows a flowchart of an OOD detection process according to various embodiments.

[0020] FIG. 7 shows a flow diagram of a computer-implemented method for developing a machine learning model according to various embodiments.

[0021] FIG. 8 is a schematic illustration of a computing system for implementing some embodiments of the present disclosure, in accordance with embodiments of the present disclosure.

[0022] In the drawings, like parts are denoted by like reference numerals. It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown. DETAILED DESCRIPTION

[0023] In the summary above, in this description, in the claims below, and in the accompanying drawings, reference is made to particular features (including method steps) of the disclosure. It is to be understood that the disclosure in this specification includes all possible combinations of such particular features. For example, where a particular feature is disclosed in the context of a particular aspect or embodiment of the disclosure, or a particular claim, that feature can also be used, to tire extent possible, in combination with and / or in the context of other particular aspects and embodiments of the disclosure, and in the disclosure generally.

[0024] In the present document, the word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or implementation of the present subject matter described herein as “exemplary'” is not necessarily be construed as preferred or advantageous over other embodiments.

[0025] In this context, an open vocabulary' model may differ from a closed vocabulary model, in that it may be capable of detecting a wide range of objects beyond the training object categories. A closed vocabulary model may only be able to detect objects that are labelled.

[0026] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however that it is not intended to limit the disclosure to the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternative falling within the scope of the disclosure.

[0027] FIG. 1 shows a flow chart of a method for developing a ML model, according to various embodiments. The method may be performed by running a model development system 100. The integrated model development system 100 may include a plurality of modules. The plurality of modules may be used in a modular fashion to form an automated pipeline that can be applied to any existing model training pipeline 110 in a parallel manner, to limit disruption to existing workflows. The model training pipeline 110 may train a ML model according to known methods, including providing a training dataset 112 to a ML model, generating output based on the training dataset, comparing the generated output to ground truth data, and optimizing the ML model based on the comparison. The plurality of modules may include any combination of at least one from: an image labelling module 102, a model evaluation module 104, an image augmentation module 106 and an out-of-distribution (OOD) detection module 108. The plurality of modules may run at different phases of the ML learning model development process.

[0028] The image labelling module 102 may be configured to automatically generate annotations for images, i.e. labelled images. The labelled images may form part of the training dataset 112. The image labelling module 102 may generate the training data before or during the training phase of the model training process.

[0029] The model evaluation module 104 may be used to assess the performance of the model, after the initial training is completed. The model evaluation module 104 may perform semantic clustering based on a validation dataset 114, and thereby identify the systematic errors of the trained model.

[0030] The image augmentation module 106 may generate the variants of data for which the trained model underperforms, also after the initial training is completed. The image augmentation module 106 may be configured to generate the variants of data, for example, new images, based on the identified systematic errors of the trained model. These new images may be added to the training dataset 112 for further training the trained model.

[0031] The OOD detection module 108 may be used in the deployment stage of the ML model life cycle, to address new scenarios previously unseen in the training and validation datasets. The OOD detection module 108 may be configured to identify uncommon scenes or scenes that were not represented in the training dataset, i.e. out-of-distribution (OOD) data. The trained model may then be finetuned based on the identified OOD data.

[0032] The integrated model development system 100 may automate manual, timeconsuming tasks of labelling data, evaluating performance of a trained model, collecting OOD data and augmentation of the training dataset. The plurality of modules may be selected depending on the user’s requirements. For example, if the user’s priority lies in increased model and dataset understanding, the user can solely implement the model evaluation module 104, to discover the clusters within the dataset where the model underperforms. With this modularity, the integrated model development system 100 may be catered for any existing deep learning model development pipeline with vary ing requirements or priorities.

[0033] According to various embodiments, the model development system 100 may be applied to develop embedded ML products in automated or autonomous agents or drivingassist systems for braking or parking.

[0034] According to various embodiments, the model development system 100 may be suitable for developing ML products in the field of vison-based applications, such as perception of environment, obstacle avoidance etc.

[0035] One embodiment of the invention is a computer program or software module used during any deep learning model development process for increased efficiency in the iterative process of model development.

[0036] Developing a ML model using tire model development system 100 may bring about reduction of development costs due to reduction of man hours required for data labelling, manual model analysis, earlier discovery of model failure modes, targeted data augmentations and generation, identification of OOD data during real-world testing before full production.

[0037] FIG. 2 shows a flow chart of an image labelling process according to various embodiments. The image labelling process may be carried out by tire image labelling module 102. The image labelling module 102 may be configured to generate annotations for images, resulting in labelled images. These labelled images may then form part of a training dataset 112, that may be used to train machine learning models for perception tasks such as semantic segmentation, depth estimation, and object detection tasks with 2D and 3D bounding boxes.

[0038] The image labelling module 102 may cany? out a two-stage learning process, by first generating candidate annotations from both open \ ocabulary and closed vocabulary vision and / or multimodal vision-language models, and subsequently refining the candidates to create the final automatically generated annotations.

[0039] The image labelling module 102 may include a prompt generation model 210. An input image 202 provided to the image labelling module 102 may be received in the prompt generation model 210. The prompt generation model 210 may be configured to translate visual features in the input image 202 into easily comprehensible textual descriptions.

[0040] The image labelling module 102 may further include an open vocabulary model 212. The input image 202 and the textual descriptions generated by the prompt generation model 210 may be provided as inputs to the open vocabulary model 212. Optionally, the descriptions may be further refined with input labels 204, before being input to the open vocabulary model 212. The open vocabulary' model 212 may be configured to produce open vocabulary7 object annotations such as labels and bounding boxes, based on the input image 202 and the textual descriptions. As an example, the prompt generation model 210 may generate textual description of “a young woman wearing a protective mask sits at a desk in a classroom while writing in a notebook" based on the contents of an input image 202. The open vocabulary model 212 may generate bounding boxes over regions in the input image 202 that correspond to keywords in the textual description, such as “young woman”, “protective mask”, “desk”, and “notebook”.

[0041] The image labelling module 102 may further include a mask generation model 214. The mask generation model 214 may be configured to perform semantic segmentation on the input image 202, to generate semantic segmentation masks.

[0042] The image labelling module 102 may further include a general purpose image decoder 216. The general purpose image decoder 216 may be configured to extract visual features from the input image 202. The visual features may be provided to a closed vocabulary visual module 220. The closed vocabulary visual module 220 may generate closed vocabulary annotations, using task-specific decoders, such as a semantic segmentation decoder 222, an object detection decoder 224 and a depth estimation decoder 226. The semantic segmentation decoder 222 may generate semantic segmentation masks based on the visual features extracted by the general purpose image decoder 216. The object detection decoder 224 may generate bounding boxes over objects detected in the input image 202 based on the visual features. The depth estimation decoder 226 may generate depth estimates of objects found in the input image 202, based on the visual features. The depth estimates are also referred herein as depth information 252.

[0043] The output of the open vocabulary model 212 and the output of the object detection decoder 224 of the closed vocabulary model 220, may undergo an object voting process 230. The voting algorithm of the object voting process 230 is shown in FIG. 3.

[0044] The object voting process 230 may result in a decision on which objects to select. The selected objects may be represented by 2D boxes with annotations 256 which may form part of the output annotations 250.

[0045] The output of the object voting process 230 may be provided to a 3D box decoder 218. The 3D box decoder 218 may be configured to generate precise 3D bounding boxes with annotations 258, thereby giving the trained model a deeper understanding of the image’s spatial components. The 3D bounding boxes with annotations 258 may form part of the output annotations 250. The trained model may be a deep learning visual perception system, which may be applied to perception applications for Advanced Driver Assistance System (ADAS).

[0046] The output of the mask generation model 214 and the output of the semantic segmentation decoder 222 may also undergo a semantic voting process 232. The voting process 232 follows the same algorithm as the object voting process, shown in FIG. 3. The semantic voting process 232 may result in a decision on which semantic segmentation masks to select, and the selected semantic segmentation masks 254 may form part of the output annotations 250.

[0047] The object voting process 230 and the semantic voting process 232 may vote for the annotations based on an overlap between the annotations generated by the open vocabulary model and the closed vocabulary model. This may improve the accuracy and correctness of the annotations (including bounding boxes, semantic segmentation masks, text labels etc).

[0048] According to various embodiments, the prompt generation model 210 may be a zeroshot image-to-text generator, an example of which is Blip-2 by Li et al, which is disclosed at httpsi / Zhu^ngf^

[0049] An example of a suitable model for the open vocabulary model 212 is GroundingDino which is disclosed in “Grounding dino: Marrying dino with grounded pretraining for open-set object detection” by Liu et. al.

[0050] An example of a suitable model for the mask generation model 214 is Segment Anything Model (SAM), disclosed in “Segment anything” by Kirillov et. al. and can be found at https: / 7andv.org / abs / 2304.02643.

[0051] An example of a suitable model for the general purpose image decoder 216 is Dinov2, which is disclosed in “Dinov2: Learning robust visual features without supervision” by Oquab et al.

[0052] An example of a suitable model for the 3D box decoder is VoxelNext disclosed in “Voxelnext: Fully sparse voxelnet for 3d object detection and tracking” by Chen et. al, and published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023.

[0053] Automatic labelling poses significant challenges to existing computer vision models, since no published models as of this writing can achieve similar performance comparable to humans. The image labelling process performed by the image labelling module 102 may provide a cost-effective solution to challenge. Further, the image labelling module 102 may be capable of integrating any vision and language models. As such, one or more of the general-purpose image decoder 216, open vocabulary model 212, and the closed vocabulary model 220 may be replaced by task-specific models to optimize performance in a particular context.

[0054] FIG. 3 shows an example of the object voting algorithm 300. The inputs required for the algorithm may include an input image 202 (‘‘inputimage”), an open vocabulary objection model 212 (“openModel”), a closed vocabulary model 220 (“closeModel”), and a voting threshold (“threshold”). The output of the algorithm may include annotations such as image labels (“Labels”) and bounding boxes (“Bboxes”). The algorithm may determine an overlap ratio of the annotations generated by the open vocabulary objection model 212 compared with the annotations generated by the closed vocabulary model 220. The algorithm may select the annotations that have the overlap ratio that exceeds the voting threshold.

[0055] FIG. 4 shows a flow chart of a model evaluation process according to various embodiments. The model evaluation module 104 may be configured to carry out the model evaluation process. The model evaluation process may include providing a validation dataset 114 to the trained model. The trained model may generate outputs based on the validation dataset 114. The model evaluation process may further include semantically clustering the validation dataset 114 to generate semantic data slices 406, assessing the performance of the model for the respective semantic data slices 406 based on the generated outputs, and determining which semantic data slice 406 is associated with poor performance of the trained model.

[0056] The model evaluation module 104 may include an image encoder 410 and a querying transformer 418. The image encoder 410 may be for example, a vision transformer. The image encoder 410 may convert the domain specific image input images 402 to image embeddings. The domain specific image input images 402 may include the validation dataset 114. Tire image embeddings may then be passed to a querying transformer 418. The querying transformer 418 may be connected to a large language model, i.e. the text encoder and decoder 420, with an additional linear layer to bridge the modality gap.

[0057] During the training or fine-tuning phase, supervised dataset comprising training images may be provided to the image encoder 410. The training images may be captioned by a multimodal image captioning tool 422, to generate captions. The supervised dataset may be used to train a finetune layer 412 of the image encoder 410. The finetune layer 412 may include input layer, normalization layer and representation layer of the image encoder 410. The training images and the captions generated by the multimodal image captioned tool 422 may be used to train the image encoder 410 and the querying transformer 418 for representation learning task. The querying transformer 418 may include a text encoder and decoder 420. The querying transformer 418 may be trained until the text encoder and the decoder 420 is folly fine-tuned. Hie querying transformer 418 is used to interact between the text (generated by the multimodal image captioning tool 422) and the image (as encoded by the image encoder 410) and to train or fine-tune a good image embedding used for the mixture model 426. The querying transformer 418 may generate a description in terms of language of the domain specific input images 402. The description generated may be compared with the input captions generated by the image captioning tool 422, to determine losses for the training of the image encoder 410, the fine tune layer 412 and the querying transformer 418.

[0058] During an inference phase, a supervised dataset including annotated domain-specific input images 402 may be input to the image encoder 410.

[0059] The output of the image encoder 410 may include multi-modal representations. The domain specific image input images 402 may also be provided to a segmentation module 414. The multi-modal representations may be fine-tuned by an offline feature fine-tuning module 416 based on the segmentation masks generated by the segmentation module 414, to result in multimodal embedding 404. The multimodal embeddings may be influenced by the textual information received by the text encoder and decoder 420. When the querying transformer is well-trained to bridge the gap between the two modalities of image and text, the output embedding may also include the combined image and text information.

[0060] The multimodal embeddings resulting from the image encoder 410 are postprocessed in combination with advanced zero-shot semantic segmentation models in the segmentation module 414 to enhance the explainability and effectiveness of the embeddings.

[0061] The multimodal embedding 404 may be pushed to the mixture model 426 for clustering. During this phase, the queiying transformer 418, the multimodal image captioning tool 422 and the text summarization tool 424 are not needed. The mixture model 426 may cluster the images based on their multimodal embeddings and their model performance score. The output of the mixture model 426 is semantically similar clusters of images where the model underperforms on, referred herein as the semantic data slice 406. In the end, the captions of the images in each cluster, as generated previously by the multimodal image captioning module 422, may be summarised using the text summarization tool 424 for the description of the entire group of data. The text summarization tool 424 may include large language models. The summarization of the captions may be useful to human developers to understand the semantic clusters where the trained model underperforms.

[0062] The segmentation module 414 may be identical, or similar to, the mask generation model 214.

[0063] Examples of a suitable model for the image encoder 410 include the Swin Transformer disclosed in “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows” by Liu et. al., and the TimeSfonner disclosed in “Is Space-Time Attention All You Need for Video Understanding?” by Bertasius et. al.

[0064] An example of a suitable model for the querying transformer 418 is Q-former, which is disclosed in “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Larger Language Models” by Li et. al.

[0065] An example of the text encoder and decoder 420 for connecting to the querying transfonner 418 is the LLaMA disclosed in “LLaMA: Open and Efficient Foundation Language Models” by Touvron et. al.

[0066] An example of a suitable model for the multimodal image captioning module 422 is Image2Paragraph by Wang et. al., and can be accessed at https: / / github.conVshowlab / lmage2Paragraph.

[0067] By using separate models to encode the image and to generate captions, the model evaluation model 104 may avoid the need to rely on multimodal pre-trained models. Instead, the model evaluation model 104 may be implemented using unimodal pre-trained models, such that the unimodal models may be upgraded to the latest models when needed, without having to label new datasets for fine-tuning or re-training the model evaluation model 104. Also, the model evaluation model 104 uses existing domain-specific unimodal datasets for fine-tuning a limited amount of trainable parameters, and as such, captures subtle differences among homogenous scenes better than general-purpose multimodal pretrained models. General purpose multimodal pretrained models are usually more difficult to train due to the cost and challenges in obtaining large multimodal datasets, and as such, it is more difficult or costly to achieve high accuracy with general-purpose multimodal pretrained models. As the model evaluation model 104 performs offline fine-tuning of the embeddings with a zero-shot segmentation model, the explainability of the clustering process is enhanced. The offline fine-tuning may further improve the quality of the embeddings based on segmentation models, so that the embedding are reflecting the objects of the images in a more semantically understandable manner.

[0068] Given the current limitation that multimodal models are lacking the ability to learn the correlation among multiple image-text pairs in a single sequence, by isolating the process to describe each individual image and then summarising all the captions in a modular way, a more semantically accurate description of a series of images may be achieved.

[0069] The model evaluation module 104 may identify and understand tire underperforming data clusters with respect to the machine learning models, with their corresponding semantic meanings. Accordingly, the model evaluation module 104 may aid in interpreting the performance and behaviors of the machine learning models, providing insights that should subsequently improve model performance by working on methods that identify semantically coherent input data clusters linked to low-performing model predictions. By focusing on these underperforming data clusters, targeted strategies may be applied, for example, data augmentation) to address issues related to data quality, data distribution, and model development. This, in turn, can help to build more robust and effective models, thus elevating tire overall performance and reliability of the machine learning models.

[0070] According to various embodiments, the model evaluation module 104 is suited for vision-based applications, and the input data may include images and videos. Nonetheless, the model evaluation module 104 is not limited to vision-based applications, and may be applied to other types of data sources, such as natural languages and time-series data.

[0071] The model evaluation module 104 is designed in a modular way to allow for the integration of existing unimodal models to achieve the best clustering performance. Each module is focused on a single task. For example, the image encoder 410, the querying transformer 418, the multimodal image captioning tool 422, the segmentation module 414 may each be replaced by other pre-trained models that are trained to perform the same tasks.

[0072] FIG. 5 shows a schematic diagram of the image augmentation module 106 according to various embodiments. The image augmentation module 106 may be configured to generate a set of images that are semantically similar to that of images of a semantic data slice 406. The semantic data slice 406 may be the output of the model evaluation module 104. By doing so, the semantic clusters of data that were lower-performed by the trained model may be adjusted to produce more variations while maintaining their inherent relationships. The image augmentation module 106 may enhance the overall performance of the trained model, in an effective manner from a data-centric point of view. The image augmentation module 106 may generate three types of image augmentations, namely semantic mixing, image variations, inpainting and outpainting targeted at the lower-performing data samples, using generative artificial intelligence (AI) techniques.

[0073] This approach helps to circumvent the traditionally expensive data collection and labelling process carried out by humans. By generating and augmenting the data samples, the original data annotations are preserved, thereby avoiding the need for a re-labelling process.

[0074] The image augmentation module 106 may segment the semantic data slice 406 using a segmentation module 510 to locate the target object for variations and modifications. Tire target object may be any object that is of interest for the trained model to detect, for example, in the context of ADAS applications, the target object may be a vehicle, a traffic sign, a parking lot etc. The target object may be an object that is described in the data annotations. The segmentation module 510 may be identical, or similar to, the segmentation module 414. An image variant module 504 may generate variants of the images of the semantic data slice 406, based on the semantic masks generated by the segmentation module 510. The variants of the data slice, are referred herein as data slice variants 508. In addition, the semantic masks may also be used to vary the size and location of the target objects, by a size and position variant module 512. After the size and location of the target objects have been adjusted, the output of the size and position variant module 512 may be passed on to an inpainting and outpainting module 514. The inpainting and outpainting module 514 may carry out inpainting and outpainting on the image variants while preserving the original data annotations. The inpainting and outpainting module 514 may include an inpainting tool configured to fill in any missing or removed portions of images with plausible content. The image augmentation module 106 may also include an image mixing module 506 configured to blend different image features, thus significantly increasing the variety within the dataset.

[0075] An example of a suitable model for the image variant module 504 is ControlNet, which is disclosed in "‘Adding Conditional Control to Text-to-Image Diffusion Models” by Zhang et. al.

[0076] An example of a suitable inpainting model in the inpainting and outpainting module 514 is Inpaint Anything which is disclosed in “Inpaint anything: Segment anything meets image inpainting” by Yu et al.

[0077] An example of a suitable model in the image mixing module 506 is Image Mixer by Pinkney, which can be accessed at https: / / github.com / justinpinkney / stable-diffusion.

[0078] The model evaluation module 104 and the image augmentation module 106 may work together to offer a cheaper, larger, richer and more adaptable dataset targeted on the lower-performed semantically coherent data clusters, for enhancing the performance of AI models in production.

[0079] FIG. 6 shows a flowchart of an OOD detection process according to various embodiments. The OOD detection process may be carried out by a OOD detection module 108. Data that were not represent in the training dataset may be observed during deployment. Such data may be referred herein as out-of-distribution (OOD) data. In the context of automotive applications, this may present in the form of changing landscapes and traffic infrastructure. The trained model may perform poorly in the presence of such data since the trained model was not exposed to the OOD data in the training stage.

[0080] The OOD module 108 may be used during real-life testing after model deployment, preferably before full deployment. The OOD module 108 may detect OOD images and these newly collected OOD images may be provided as feedback to finetune the model training process iteratively. As a result, the trained model may be better equipped to handle rare data observations, and the amount of updates required post-deployment may be minimized.

[0081] OOD detection may be defined with the specific problem formulation of identifying semantically different inputs from the training set and then deciding to exclude those inputs from model prediction. In the OOD module 108, OOD detection may be defined within the context of novelty detection with the intention of incremental learning thereafter, where post identification of semantically different "‘nover / OOD inputs, those inputs are then treated as positive learning opportunities by being selected for additional processing for labelling and further model training in the image labelling module 102. The OOD module 108 may serve as an incremental learning system that actively detects novel images during deployment, for adding to the training dataset via automatic labelling by the image labelling module. The OOD module 108 may run in parallel to the trained model that is conducting inference, so as to be modular and minimally intrusive for deployment to a target device.

[0082] In other words, OOD detection process may include capturing an image in 602, and providing the image to the trained model to carry out model inference in 604. The trained model may generate a prediction, in 606. The image is also provided to an OOD detector 608 to generate an output. The output of the OOD detector 608 may be compared to the model prediction, to determine if the image is out-of-distribution. Hie comparison may involve determining whether the distribution of the captured image is close to the model prediction output. If no, the captured image may be detennined to be OOD. If the image is out-of-distribution, the image may be provided to the image labelling module 102 for generating annotations. The resulting annotations and the image may then be provided to the model training pipeline 110 to train the model.

[0083] The OOD module 108 may include lightweight OOD detectors, like as disclosed in such as the Multi-level Out-of-Distribution (MOOD) disclosed in “MOOD: Multi-level Out-of-distribution Detection” by Linet, al. (arXiv:2104.14726) or the Neural Mean Discrepancy (NMD) detector disclosed in “Neural Mean Discrepancy for Efficient Out-of-Distribution Detection” by Dong et. al. (arXiv:2104.11408). The on-device OOD detection by means of classification may be conducted with minimal memory and computational overhead.

[0084] FIG. 7 shows a flow diagram of a computer-implemented method 700 for developing a machine learning model according to various embodiments. The method 700 may include method steps carried out by the plurality of modules of the integrated model development system 100, as described above with respect to FIGS. 1 to 6.

[0085] The method 700 may include processes 702, 704, 706 and 708. The process 702 may include generating a training dataset 112. Tire training dataset 112 may include a plurality of labelled images. The process 704 may include training a machine learning model based on the training dataset 112, thereby generating a trained model. The process 706 may include evaluating performance of the trained model based on a validation dataset 114, by identifying a semantic data slice 406 from the validation dataset 114 where the trained model underperforms. The process 708 may include further training the trained model based on tire identified semantic data slice 406.

[0086] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the training dataset 112 may include generating annotations based on the input image 202, by at least one of an open vocabulary model and a closed vocabulary' model 222. The open vocabulary model may include the open vocabulary model 212.

[0087] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating annotations based on the input image 202 may include generating a first candidate annotation by the open vocabulary' model, generating a second candidate annotation by the closed vocabulary model 222, and determining a final annotated bounding box based on the first candidate annotation and the second candidate annotation.

[0088] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the first candidate annotation may include providing input labels 204 to the open vocabulary model, detecting objects in the input image 202 based on the input labels 204, by the open vocabulary model, generating bounding boxes around the detected objects, and appending the respective input labels to the bounding boxes.

[0089] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, providing the input labels 204 may include translating visual features in the input image 202 into textual descriptions, by a prompt generation model 210, and generating the input labels 204 based on tire textual descriptions.

[0090] According to an embodiment which may be combined with any above-described embodiment or with any below descnbed further embodiment, generating annotations based on the input image 202 may further include generating a first candidate semantic segmentation mask based on the input image 202, using a first semantic segmentation model, generating an image embedding based on the input image 202, using an image encoder, and generating a second candidate semantic segmentation mask based on the image embedding, using a second semantic segmentation model. The first semantic segmentation model may include the mask generation model 214. The second semantic segmentation model may include the semantic segmentation decoder 222.

[0091] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating annotations based on the input image 202 may further include determining a final semantic segmentation mask based on the first candidate semantic segmentation mask and the second candidate semantic segmentation mask.

[0092] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, evaluating performance of the trained model may include generating multimodal embedding 404 based on a validation dataset 114, generating outputs based on the validation dataset 114, by the trained model, determining performance scores of the respective generated outputs, and clustering the validation dataset 114 based on the determined performance scores and the multimodal embedding 404, by a mixture model 426.

[0093] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the multimodal embedding 404 may include generating semantic segmentation masks based on the validation dataset 114, using a semantic segmentation model, generating image embedding based on the validation dataset 114, using an image encoder, and generating the multimodal embedding 404 based on the semantic segmentation masks and the image embedding. The semantic segmentation model may include the segmentation module 414.

[0094] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, further training the trained model based on the identified semantic data slice 406 may include generating augmented images based on the identified semantic data slice 406, using an image augmentation module 106, and further training the trained model using the augmented images as input.

[0095] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the augmented images may include generating semantic segmentation masks based on the identified semantic data slice 406, and generating variants of images from the identified semantic data slice 406 based on the semantic segmentation masks, using a generative model.

[0096] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the augmented images may include at least one of inpainting and outpainting images from the identified semantic data slice 406.

[0097] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the augmented images may include blending image features from other images, with image features of the images from the identified semantic data slice 406, to generate variants of the identified semantic data slice 406. The variants of the identified semantic data slice 406 may include the data slice variants 508.

[0098] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, generating the augmented images may further include generating annotations using the image labelling module, based on the variants of the identified semantic data slice 406.

[0099] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the method 700 may further include detecting out-of-distribution images for use in further training the trained model, by an out-of-distribution module 108.

[0100] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, detecting the out-of-distribution images may include monitoring images input into the trained model when the trained model is conducting inference, in order to identify out of distribution images.

[0101] FIG. 8 is a schematic illustration of a computing system for implementing some embodiments of the present disclosure, in accordance with embodiments of the present disclosure. Computing system 800 can be used, for example, for one or more steps of method 700. The computing system 800 may include, or may be part of, a data processing device. System 800 can be a computer connected to a network. System 800 can be a client or a server. System 800 can be any suitable type of processor-based system, such as a personal computer, workstation, server, handheld computing device (portable electronic device) such as a phone or tablet, or an embedded system or other dedicated device. The system 800 can include, for example, one or more processors 810, one or more memory 814, one or more of input device 820, one or more of output device 830, a graphical user interface (GUI) 834, storage 840, and communication device 860. Input device 820 and / or output device 830 can generally either be connectable or integrated with the computing system 800.

[0102] Input device 820 can be any suitable device that provides input, such as a touch screen, keyboard or keypad, mouse, gesture recognition component of a virtual / augmented reality system, or voice-recognition device. Output device 830 can be or include any suitable device that provides output, such as a display, touch screen, haptics device, virtual / augmented reality display, or speaker.

[0103] Storage 840 can be any suitable device that provides storage, such as an electrical, magnetic, or optical memory including a RAM, cache, hard drive, removable storage disk, or other non-transitory computer readable medium. Communication device 860 can include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or device. The components of the computing system 800 can be connected in any suitable manner, such as via a physical bus or wirelessly.

[0104] Processor(s) 810 can be any suitable processor or combination of processors, including any of, or any combination of, a central processing unit (CPU), graphics processing unit (GPU), field programmable gate array (FPGA), and application-specific integrated circuit (ASIC). Software 850, which can be stored in storage 840 and executed by one or more processors 810, can include, for example, the programming that embodies the functionality-- or portions of the functionality of the present disclosure (e.g., as embodied in the devices or methods as described above). For example, software 850 can include one or more programs for execution by one or more processor(s) 810 for performing one or more of the steps of method 700.

[0105] Software 850 can also be stored and / or transported within any non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch instructions associated with the software from the instruction execution system, apparatus, or device and execute the instructions. In the context of this disclosure, a computer-readable storage medium can be any medium, such as storage 840, that can contain or store programming for use by or in connection with an instruction execution system, apparatus, or device.

[0106] Software 850 can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch instructions associated with the software from the instruction execution system, apparatus, or device and execute the instructions. In the context of this disclosure, a transport medium can be any medium that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus, or device. The transport computer readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation medium.

[0107] System 800 may be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communications protocol and can be secured by any suitable security protocol. The network can comprise network links of any suitable arrangement that can implement the transmission and reception of network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.

[0108] System 800 can implement any operating system suitable for operating on the network. Software 850 can be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software embodying the functionality of the present disclosure can be deployed in different configurations, such as in a client / server arrangement or through a Web browser as a Web-based application or Web service, for example.

[0109] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based here on. Accordingly, the embodiments of the present disclosure are intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims

1. A computer-implemented method (700) for developing a machine learning model, the method comprising:generating a training dataset comprising a plurality of labelled images (702);training a machine learning model based on the training dataset, thereby generating a trained model (704);evaluating performance of the trained model based on a validation dataset, by identifying a semantic data slice from the validation dataset where the trained model underperforms (706); andfurther training the trained model based on the identified semantic data slice (708).

2. The method (700) of any preceding claim, wherein generating the training dataset (702) comprisesgenerating annotations based on an input image, by at least one of an open vocabulary model and a closed vocabulary model.

3. Tire method (700) of claim 2, wherein generating annotations based on the input image comprisesgenerating a first candidate annotation by the open vocabulary model,generating a second candidate annotation by the closed vocabulary model, and determining a final annotated bounding box based on the first candidate annotation and the second candidate annotation.

4. The method (700) of claim 3, wherein generating the first candidate annotation comprisesproviding input labels to the open vocabulary model,detecting objects in the input image based on the input labels, by the open vocabulary model,generating bounding boxes around the detected objects, andappending tire respective input labels to the bounding boxes.

5. The method (700) of claim 4, wherein providing the input labels comprisestranslating visual features in the input image into textual descriptions, by a prompt generation model, andgenerating the input labels based on the textual descriptions.

6. The method (700) of any one of claims 2 to 5, wherein generating annotationsbased on the input image further comprisesgenerating a first candidate semantic segmentation mask based on the input image,using a first semantic segmentation model,generating an image embedding based on the input image, using an image encoder, andgenerating a second candidate semantic segmentation mask based on the image embedding, using a second semantic segmentation model.

7. The method (700) of claim 6, wherein generating annotations based on the input image further comprisesdetermining a final semantic segmentation mask based on the first candidatesemantic segmentation mask and the second candidate semantic segmentation mask.

8. The method (700) of any preceding claim, wherein evaluating performance of the trained model comprisesgenerating multimodal embedding based on the validation dataset,generating outputs based on the validation dataset, by the trained model,determining performance scores of the respective generated outputs, andclustering the validation dataset based on the determined performance scores and themultimodal embedding, by a mixture model.

9. The method (700) of claim 8, wherein generating the multimodal embedding comprisesgenerating semantic segmentation masks based on the validation dataset, using athird semantic segmentation model,generating image embedding based on the validation dataset, using an image encoder,andgenerating the multimodal embedding based on the semantic segmentation masks and the image embedding.

10. The method (700) of any preceding claim, wherein further training the trainedmodel based on the identified semantic data slice comprisesgenerating augmented images based on the identified semantic data slice, using an image augmentation module, andfurther training the trained model using the augmented images as input.

11. The method (700) of claim 10, wherein generating the augmented images comprisesgenerating semantic segmentation masks based on the identified semantic data slice, andgenerating variants of images from the identified semantic data slice based on the semantic segmentation masks, using a generative model.

12. The method (700) of any one of claims 10 to 11, wherein generating the augmented images comprisesat least one of inpainting and outpainting images from the identified semantic data slice.

13. The method (700) of any one of claims 10 to 12, wherein generating the augmented images comprisesblending image features from other images, with image features of the images from the identified semantic data slice, to generate variants of the identified semantic data slice.

14. The method (700) of claim 13, wherein generating the augmented images further comprisesgenerating annotations using the image labelling module, based on the variants of the identified semantic data slice.

15. The method (700) of any preceding claim, further comprising:detecting out-of-distribution images for use in further training the trained model, by an out-of-distribution module.

16. The method (700) of claim 15, wherein detecting the out-of-distribution images comprisesmonitoring images input into the trained model when the trained model is conducting inference, in order to identify out-of-distribution images.

17. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (700) of any preceding claims.

18. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method (700) of any one of claims 1 to 16.

19. A data processing device comprising means for carrying out the method (700) of any one of claims 1 to 16.

20. Use of the method (700) of any one of claims 1 to 16, for developing a machine learning model.

Images