A method for multi-task image classification

Abstract

A method for training an image classifier for multi-task image classification. The method includes providing a set of deep learning models pre-trained on an initial image dataset and fine-tuning the set of pre-trained deep learning models on n target datasets of images, wherein each target dataset of images is associated with one task of a plurality of tasks and wherein the set of pre-trained deep learning models is iteratively fine-tuned on one of the target datasets of images from 1 to n and the set of pre-trained deep learning models is re-initialised between each iteration. The method also includes extracting features associated with each of the plurality of tasks from the set of pre-trained deep learning models iteratively fine-tuned on the n target datasets of images, combining the extracted features from each of the fine- tuned set of deep learning models to generate a fused feature set for the plurality of tasks, and training one or more machine learning classifiers on the fused feature set of the extracted features.

Description

A METHOD FOR MULTI-TASK IMAGE CLASSIFICATIONTECHNICAL FIELD[1] The present invention relates to multi-task image classification and training of an image classification model for multi-task image classification. In particular, the present invention relates to, but is not limited to, methods for multi-task image classification and training of an image classification model for multi-task image classification.BACKGROUND[2] Any references to methods, apparatus or documents of the prior art are not to be taken as constituting any evidence or admission that they formed, or form part of the common general knowledge.[3] Multi-task image classification requires the efficient and accurate classification of images into multiple categories simultaneously. Existing solutions often lead to increased computational complexity, data redundancy, and the risk of catastrophic forgetting caused by repeatedly fine-tuning a model for a new task, causing it to forget previously learned knowledge and impacting performance on previous tasks.[4] Additionally, fine-tuning the models from scratch with new tasks can result in overfitting and difficulties in leveraging prior knowledge gained from related domains.[5] The existing solutions attempt to address this problem by employing various techniques such as hard and soft parameter sharing, which involve sharing or partially sharing parameters across tasks. However, these approaches may struggle to strike the right balance between task-specificity and parameter reuse, leading to suboptimal performance and difficulties in capturing unique features for each task.[6] Another challenge in multi-task image classification is handling limited labelled data for certain tasks, which can hinder model performance and generalization.[7] Moreover, the need for explainable deep learning models arises, as the blackbox nature of some methods makes it difficult to understand the model's decisionmaking process, especially in critical applications like healthcare.[8] Given these challenges and issues, there is a need for an improved multi-task image classification.SUMMARY OF INVENTION[9] In an aspect, the invention provides a method for training an image classifier for multi-task image classification, the method comprising: providing a set of deep learning models pre-trained on an initial image dataset; fine-tuning the set of pre-trained deep learning models on n target datasets of images, wherein each target dataset of images is associated with one task of a plurality of tasks and wherein the set of pre-trained deep learning models is iteratively fine-tuned on one of the target datasets of images from 1 to n and the set of pre-trained deep learning models is re-initialised between each iteration, extracting features associated with each of the plurality of tasks from the set of pre-trained deep learning models iteratively fine-tuned on the n target datasets of images; combining the extracted features from each of the fine-tuned set of deep learning models to generate a fused feature set for the plurality of tasks; and training one or more machine learning classifiers on the fused feature set of the extracted features.

[0010] Preferably, after providing two or more deep learning models pre-trained on an initial image dataset, the method further comprises updating the pre-trained deep learning models on a secondary image dataset of images of a same image type as images of a target dataset.

[0011] Preferably, combining the extracted features from each of the fine-tuned set of deep learning models to generate a fused feature set for the plurality of tasks includes first combining the features extracted from the set of pre-trained deep learning models for each task of the plurality of tasks to generate a plurality of fused feature sub-sets each relating to one task of the plurality of tasks, wherein each fused feature sub-set only includes features extracted from the set of pre-trained deep learning models in relation to one of the tasks of the plurality of tasks and subsequently combining the fused feature subsets to generate the fused feature for the plurality of tasks.

[0012] Preferably, the method includes identifying one or more ideal machine learning classifiers from the one or more trained machine learning classifiers based on one or more performance metrics.

[0013] In another the invention provides a method of classifying images comprising: receiving images; and applying the images to one or more pre-trained machine learning classifiers to thereby produce identifications and / or classifications based on a plurality of tasks, wherein the one or more pre-trained machine learning classifiers are: trained on a fused feature set of features extracted from a set of deep learning models that are pre-trained on an initial image dataset; wherein the set of deep learning models are fine-tuned on n target datasets of images, wherein each target dataset of images is associated with one task of theplurality of tasks, wherein the set of pre-trained deep learning models is iteratively fine-tuned on one of the target datasets of images from 1 to n and the set of pretrained deep learning models is re-initialised between each iteration.

[0014] Preferably, the method includes processing the images to generate one or more views. Preferably, the method includes automatic processing of the images to generate the one or more views.

[0015] Preferably, fine-tuning the set of pre-trained deep learning models includes freezing initial layers of the pre-trained models and any remaining layers are trained on the images associated with the task to adapt the two or more pre-trained deep learning models to the task. Preferably, freezing the initial layers includes maintaining weights and parameters of the set of pre-trained deep learning models.

[0016] Preferably, the one or more machine learning classifiers include XGBoost, K- Nearest Neighbour (KNN), Support Vector Machines (SVM), Random Forest and Naive Bayes.

[0017] Preferably, the two or more pre-trained deep learning models include two or more convolutional neural networks (CNNs). More preferably, the set of pre-trained deep learning models include at least two of the Xception model, the lnceptionV3 model and the MobileNetV2 model. Preferably, the set of pre-trained deep learning models are pre-trained on the ImageNet dataset.

[0018] Preferably, the method includes pre-training the set of deep learning models on an initial image dataset.

[0019] Preferably, the method includes identifying one or more ideal machine learning classifiers from the one or more pre-trained machine learning classifiers based on one or more performance metrics.

[0020] Preferably, the method includes combining decisions of the one or more pretrained machine learning classifiers, or the one or more ideal machine learning classifiers, using majority voting to produce the identifications and / or classifications.

[0021] Preferably, the method further includes the steps of: fine-tuning the set of pre-trained deep learning models on an n+1 target dataset of images associated with an additional task; extracting features associated with the additional task from the set of pretrained deep learning models iteratively fine-tuned on the n+1 target dataset of images; combining the extracted features from the set of deep learning models finetuned on the n+1 target dataset of images with the fused feature set of the extracted features to generate a new fused feature set of extracted features; and training one or more machine learning classifiers on the new fused feature set of the extracted features.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Preferred features, embodiments and variations of the invention may be discerned from the following Detailed Description which provides sufficient information for those skilled in the art to perform the invention. The Detailed Description is not to be regarded as limiting the scope of the preceding Summary of the Invention in any way. The Detailed Description will make reference to a number of drawings as follows:

[0023] Figures 1a and 1b illustrate a flowchart for a method of training a classifier to identify and classify content of images across multiple tasks according to an embodiment of the invention;

[0024] Figure 2 is a flowchart for a method of identifying and classifying content of images across multiple tasks using a trained classifier according to an embodiment of the invention;

[0025] Figure 3 is a diagrammatic illustration of training a classifier to identify and classify contents of images across multiple tasks according to an embodiment of the invention;

[0026] Figure 4is a diagrammatic workflow of the transfer learning step of the method shown in Figure 1 ;

[0027] Figure 5 is a diagrammatic workflow of the single task fusion step of the method shown in Figure 1 ;

[0028] Figure 6 is a diagrammatic workflow of the multi-task classification step of the method shown in Figure 1;

[0029] Figure 7 illustrates the majority voting to determine a final classification result;

[0030] Figure 8 illustrates MURA dataset classes where A) Forearm; B) Elbow; C) Shoulder; D) Hands; and E) Humerus;

[0031] Figure 9 illustrates a comparison between two different TL for an X-ray classification task;

[0032] Figures 10a-e illustrate a heatmap of DL models on where the model looks to make a decision;

[0033] Figures 11a-c illustrate confusion metrics of single task fusion the of five tasks with KNN

[0034] Figures 12a-c illustrate a heatmap of DL models where the areas of focus contribute to the model’s decision-making process;

[0035] Figures 13a-c illustrate missed cases by one model and correct by the second one;

[0036] Figures 14a-b illustrate confusion metrics and AUC of combination of two tasks with KNN;

[0037] Figures 15a-c illustrate confusion metrics and AUC of combination of three tasks with KNN;

[0038] Figures 16a-e illustrate confusion metrics and AUC of combination of Five tasks with KNN

[0039] Figure 17 illustrates confusion matrices of a Movie dataset where (a) lnceptionResNetV2 (b) lnceptionV3 and (c) Xception;

[0040] Figure 18 illustrates a heatmap of DL models on where the model looks to make the decision for the movie dataset;

[0041] Figure 19 illustrates a heatmap of DL models on where the model looks to make the decision for the Hockey dataset;

[0042] Figure 20 illustrates a heatmap of DL models on where the model looks to make the decision for the RLVS dataset;

[0043] Figure 21 illustrates missed cases by one model and correct by the second one;

[0044] Figure 22 illustrates the confusion matrices of the Movie dataset with three ML classifiers;

[0045] Figure 23 illustrates the Confusion metrics and AUC of case study #3 on the whole dataset with ML classifiers;

[0046] Figure 24 illustrates the Grad-CAM of four models on a test sample for case study #3 where a) MobileNetv2, b) EfficientNet, c) Xception, d) lnceptionresNetv2;

[0047] Figure 25 illustrates the Grad-CAM of four models shows the cover of the region of interest for case study #3 a) MobileNetv2, b) EfficientNet, c)Xception, d) lnceptionresNetv2; and

[0048] Figure 26 illustrates the Grad-CAM of four models of case study 3# shows the missing cases by two models are fixed by the other two a) MobileNetv2, b) EfficientNet, c) Xception, d) lnceptionresNetv2.DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0049] Embodiments of the invention described herein provide an improved multitask image classification method and method for training an image classifier model for multi-task image classification.

[0050] The method, illustrated in Figures 1 to 7, can be described as using at least three steps, and up to four steps.

[0051] Firstly, the method uses a transfer learning phase 1 including transfer learning for each individual model in a set of models (pre-trained deep learning models). This is referred to as part 1. Every pre-trained deep learning model of the set of pretrained deep learning models must undergo this transfer learning before being trained on a target dataset.

[0052] Each deep learning model of the set of deep learning models is pre-trained on an initial image dataset (such as ImageNet, for example). Pre-trained models known in the art include lnceptionResNetv2 [1], Xception [2], MobileNetv2 [3], and EfficientNet [4],

[0053] The initial image dataset typically includes a diverse collection of natural colour images belonging to thousands of classes.

[0054] In the transfer learning phase 1, there are two approaches to using transfer learning for these models.

[0055] One approach is same domain transfer learning. If the features extracted from the target dataset exhibit similarities to the features extracted from the initial image dataset then employing transfer learning from pre-trained models of the initial image dataset becomes a suitable approach to address the scarcity of training data.

[0056] The resemblance between the target dataset’s features and those of the initial image dataset indicates that the knowledge gained from diverse collection of labelled images in the initial image dataset can be effectively leveraged to enhance the performance of the model on the target dataset. By utilizing the pre-trained models, the model can benefit from the learned representations of the initial image dataset, allowing it to generalize well even with limited training data from the target domain.

[0057] Another approach is different domain transfer learning. If the features extracted from the target dataset differ from those extracted from the initial image dataset, such as in the case of X-ray images, an in-domain transfer learning approach, as depicted in Figure 4 can be used.

[0058] This approach entails updating the learned features of the pre-trained models through a process known as fine-tuning. Fine-tuning involves training the pre-trained models on images that are more similar to the target dataset, allowing the models to adapt and specialize to the specific characteristics of the target domain.

[0059] By fine-tuning domain-specific data, the models can capture relevant patterns and features that are specific to the target dataset, leading to improved performance and better alignment with the target domain’s unique characteristics. This in-domain transfer learning approach enables models to effectively leverage the knowledgegained from ImageNet while tailoring the learned representations to suit the specific requirements and features of the target dataset, such as X-ray images.

[0060] For example, suppose that the target data set focuses on classifying X-ray images of the shoulder into normal and fractured categories. In this case, an appropriate source for transfer learning would be X-ray images of different body parts that share similar features, including colours, shapes, and other relevant characteristics. These body parts may exhibit analogous imaging patterns and structures, making them valuable sources of knowledge for the target task. Using transfer learning from these related X-ray images, the model can benefit from shared features and gain insights into discerning normal and fractured shoulder X-ray images.

[0061] This part offers several advantages in any image classification task.

[0062] Faster Training: Part 1 allows models to be trained faster than training from scratch. By leveraging pretrained models, the initial layers of the network have already learned low-level features, which can be applied to new tasks, reducing the time required for convergence.

[0063] Overfitting: Part 1 can help prevent overfitting, which occurs when the model becomes too specialized in the training data and performs poorly on new data. By utilizing pre-trained models, which have already learned general features, the risk of overfitting is reduced.

[0064] Reduced Data Requirements: DL models often require a large amount of labelled training data. Part 1 helps mitigate this issue by utilizing knowledge from a related domain, allowing the model to generalize better even with limited labelled data.

[0065] Improved Generalization: Part 1 assists in improving the generalization capabilities of the model. The pre-trained models have already learned a wide range of features on large datasets, making them adept at capturing meaningful patterns. The model can better understand and generalize new unseen data by transferring this knowledge.

[0066] Effective Feature Extraction: DL models typically have several layers, where the initial layers learn low-level features, and the later layers learn more abstract representations. Part 1 allows the model to benefit from the learned features in the initial layers, allowing effective feature extraction for the new task.

[0067] Domain Adaptation: Part 1 enables the model to adapt to different domains or tasks. By leveraging knowledge from a source domain, the model can adapt and perform well on a target domain even when the target domain has limited or different data available.

[0068] In the second phase (part 2), the method performs a fusion of single task features from multiple pre-trained deep learning models, as illustrated in Figure 5. This single task fusion 2 of the method focuses on the fusion / combination of features extracted from two or more deep learning models (a set of deep learning models) fine-tuned for the same task. By combining these features, a comprehensive and robust representation can be obtained, offering improved performance in the subsequent stages of the method.

[0069] This part of the method is repeated across the set of pre-trained deep learning models for a plurality of tasks.

[0070] The fused features derived from this part can be utilized in various ways. They can be stored for future stages of the framework, allowing for the integration of additional information or refinement of the classification process. Alternatively,fused features can be used to train machine learning classifiers specifically tailored for single-task classification.

[0071] This part presents several advantages and reasons to incorporate this part into any image classification task. By integrating this approach into the workflow, the following benefits can be realized:

[0072] Flexibility: Part 2 provides flexibility in incorporating additional deep learning (DL) models into the system. This flexibility allows for the expansion and enhancement of the model set, enabling the integration of new DL models as they become available or as the data set grows. By adding more DL models, the overall performance and accuracy of the method can be improved.

[0073] No training from scratch: Part 2 eliminates the need to train the models from scratch when incorporating additional data by training additional DL models on new data that can combine with old features for the same task. Training DL models from scratch can be computationally expensive and time-consuming. This part significantly reduces the required training time and computational resources, enabling a more efficient and scalable system.

[0074] Improved performance: Part 2 allows the combining the strengths of multiple deep learning models, resulting in improved performance. By integrating the features learned by different models, the fused representation can capture a more comprehensive and discriminative representation of the input data, enhancing accuracy and effectiveness.

[0075] Robustness: Part 2 enhances the robustness of the model by reducing the impact of individual model weaknesses or biases. Different models may excel in various aspects or capture other aspects of the data, and by fusing their features,the model becomes more resilient to errors or limitations inherent in a single model.

[0076] Enhanced Generalization: Part 2 enables the model to better generalize to unseen or diverse data. By incorporating features from multiple models, the fused representation can capture a broader range of patterns and variations in the data, improving the model’s ability to handle different instances and variations of the task.

[0077] Low resources: If the data set for a single task is large, Part 2 of the framework provides an opportunity to divide it into subgroups. Each subgroup can then train the same or different DL models multiple times. This approach effectively utilizes available resources by maximizing the use of the data set. A diverse set of features can be extracted by training multiple DL models on different subsets of the dataset. These features, obtained from each model, can be fused without losing valuable information. The feature fusion process combines the extracted features from all models, resulting in a comprehensive representation that covers the entire dataset for a single task.

[0078] Low Risk of Bias: Part 2 ensures a low risk of bias in the framework for a single task. When incorporating additional data, the feature fusion process ensures that new data features are combined with existing data without losing valuable information. This balanced representation of features helps mitigate bias that may arise when training from scratch with new additional data. This low risk of bias contributes to the overall fairness and robustness of the framework.

[0079] High Security: Part 2 of the framework provides high security and defense against adversarial attacks for any task. This is achieved through the flexibility that the framework offers to train a new model using images that incorporatespecific attack techniques. The features extracted from these attacked images can then be combined with clean image features to create a robust representation that includes knowledge of the attack. By incorporating adversarial images during training, the classifier becomes familiar with attack techniques and can learn to recognize and defend against them effectively. This approach enhances the model’s ability to detect and classify adversarial instances, strengthening its security and robustness in real-world scenarios.

[0080] A further stage of the method (part 3) includes the fusion or combination of features obtained from Part 2 for the plurality of tasks to perform multi-task image classification. This is illustrated in Figure 6.

[0081] As an example, the first task of the plurality of tasks may classify shoulder X-ray images as fractured or normal, while the second task of the plurality of tasks could focus on the forearm, hand, etc.

[0082] In another example, the various tasks may relate to detecting violence or physical assault in multiple scenarios or settings, such as the detection of violence in various settings such as the street, school, or mall.

[0083] A complete representation is obtained by combining the features extracted from plurality of tasks. This fused feature representation can be utilized to train machine learning classifiers to perform the desired classification tasks. The fused features can be easily updated to incorporate any modifications or the inclusion of supplementary tasks. This allows the framework to continuously evolve and improve its classification performance, ensuring its applicability and effectiveness in diverse scenarios.

[0084] This part presents several advantages, including:

[0085] Federated Learning: Part 3 of the framework introduces the possibility of employing Federated Learning. This learning approach enables institutions or hospitals to train DL models on their respective data without sharing it externally. This capability is particularly valuable in fields such as healthcare, security, and finance, where privacy and regulatory concerns impose restrictions on data sharing. In this part, the extracted features from individual tasks are the only components that need to be shared and combined with other tasks. These features can be encrypted, ensuring their confidentiality and making them difficult to decipher without proper authorization. This approach empowers institutions to participate in collaborative efforts for model development, leveraging collective expertise and data diversity without exposing sensitive information.

[0086] Flexibility: Part 3 of the framework provides a high degree of flexibility for incorporating new tasks that serve the same purpose. This flexibility allows for the expansion and adaptation of the model to handle various scenarios, such as detecting fractures in different body parts or performing object detection in diverse environments, including schools, streets, or football stadiums. Using the fused features of Part 2, the framework can seamlessly integrate new tasks into the existing model. This adaptability enables the system to accommodate various classification requirements without sacrificing performance or accuracy. Whether it involves adding new body parts for fracture detection or incorporating different scenarios for object detection, the framework can effectively leverage the shared knowledge and representation learned from the previously trained tasks.

[0087] Enhanced Generalisation: Part 3 of the framework offers a high level of generalization for any task. This flexibility allows the model to generalize well in various classification requirements. The high level of generalization provided byPart 3 is particularly valuable in real-world applications, where variations in data distribution, environmental conditions, or specific task requirements can be significant. Incorporating new scenarios or tasks ensures that the model can adapt and learn from diverse contexts, improving its capacity to handle various classification challenges.

[0088] No Training from Scratch: Part 3 of the framework eliminates the need to train the models from scratch when incorporating additional tasks. Instead, it achieves this by introducing additional deep learning (DL) models for tasks that can be combined with existing features for the same purpose. This approach offers several benefits, including reducing the risk of bias and improving efficiency.

[0089] Catastrophic forgetting: Incorporating new information into a pre-existing DL model without compromising the learned knowledge is a significant challenge. For example, consider a scenario where a model is trained to classify 1000 types of cars, and later a new type of car is introduced. If the model is fine-tuned solely with the new class, its performance on the original classes may deteriorate. One straightforward solution is to train an entirely new model from scratch using the old and new data. However, this approach is time-consuming and computationally demanding, and it can lead to an unstable state for the learned representations of the initial data. It can be repeated with every new dataset. To address this issue, a proposed framework offers a more efficient and effective solution. It allows for the incorporation of new information without the need to retrain the entire model from scratch.

[0090] In a further stage (part 4), the method may include combining the decisions of the top ML classifiers using majority voting to determine the final classificationresult, as depicted in Figures 3 and 7, at reference 300. The best classifiers are selected on the basis of their performance during the training and validation phases. By evaluating performance metrics (or evaluation metrics) such as accuracy, precision, recall, and the F1 score, the framework identifies the classifiers that consistently achieve the best results in the validation data set. These top-performing classifiers are then used in the majority-voting process.

[0091] Combining the decisions of top ML classifiers using majority voting can improve the accuracy and reliability of the classification process by integrating the output of multiple classifiers through a decision-fusion approach. This strategic fusion of decisions may be useful in mitigating the limitations of individual classifiers and enhancing the overall performance of the system, contributing to its efficacy in handling diverse and complex datasets.

[0092] This majority voting presents the opportunity for multiple advantages, including:1. Improved Accuracy and Performance: The system produces more accurate and robust predictions by combining the outputs of multiple classifiers. This is particularly beneficial in scenarios where individual classifiers may be prone to errors.2. Robustness: The ensemble approach improves model robustness by minimizing the impact of errors and biases in individual classifiers, especially when dealing with noisy or uncertain data.3. Mitigation of Overfitting: Decision fusion helps mitigate overfitting by combining multiple models. As a result, the ensemble is less likely to overfit the peculiarities of the training set and performs better with new data.4. Adaptability: The framework is adaptable, allowing easy addition or removal of classifiers. This flexibility enables the model to evolve, incorporate new information, or adapt to changes in data distribution.

[0093] The following examples and embodiments relate to the training of an image classifier with respect to identifying and / or classifying an image using the above teachings.

[0094] The method includes providing two or more deep learning models pre-trained on an initial image dataset at box 110. The Inventors have found that a minimum of two deep learning models are required. However, a greater number of deep learning models can be utilised.

[0095] The illustrated embodiment includes n deep learning models pre-trained on an initial image dataset.

[0096] In some embodiments, the image dataset used for pre-training may be the ImageNet dataset that is commonly used for pre-training deep learning models. The ImageNet dataset includes a diverse collection of natural colour images belonging to 1000 classes.

[0097] In some embodiments, due to the inherent differences between the ImageNet and the target dataset, direct transfer learning from ImageNet may be required to effectively capture relevant features or image content for the specific task or tasks of interest. As an example, detecting bone anomalies on MRI scans.

[0098] In such scenarios, the method includes a further step of updating the pretrained deep learning models on a secondary image dataset of a same image type as images of a target dataset at box 120.

[0099] This step includes updating the learning features of the pre-trained models by fine-tuning them on images similar to the target dataset and associated with the onetask of the plurality of tasks. For example, the Inventors found it is possible to align learned features of the pre-trained deep learning models with the characteristics specific to the task by further training (i.e., fine-tuning) the deep learning models on images related to or similar to the task or type of image in the target dataset.

[0100] In embodiments of the invention, an image type of the medical imaging images of the secondary image dataset is the same as an image type of the medical imaging images of the target dataset. The image type may be one or more of: Computed Tomography, X-rays, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET) and single-photon emission computed tomography (SPECT). For example, the image type of the medical imaging images of the secondary image dataset and the medical imaging images of the target dataset is x-ray images.

[0101] During the in-domain transfer learning process, the initial layers of the pretrained models further trained on the secondary image dataset are frozen. Freezing these layers means their weights and parameters remain unchanged, allowing them to retain the general knowledge gained from the pre-training. In one embodiment, this allows the remaining layers of the deep learning models to be further trained using the images associated with the task by focusing on adapting the models to the specific task. By adopting this approach, the Inventors addressed two significant challenges. Firstly, the scarcity of training data specific to a task is mitigated by leveraging the information captured by the pre-trained models on the ImageNet dataset. This helps to enhance the performance of the models in a data-limited scenario. Secondly, in-domain transfer learning facilitated better generalization and reduced the risk of overfitting. By fine-tuning the models on the images associated with the task, the deep learning models are able to learn task-specific features whileretaining the beneficial representations acquired during their pre-training on ImageNet. This type of transfer learning also helps with the next feature extraction step.

[0102] Next, the method includes fine-tuning the set of pre-trained deep learning models on n target datasets of images at box 130. Each target dataset of images is associated with one task of a plurality of tasks and the set of pre-trained deep learning models is iteratively fine-tuned on one of the target datasets of images from 1 to n. The set of pre-trained deep learning models may be re-initialised between each iteration to minimise or prevent catastrophic forgetting and / or overfitting.

[0103] Put another way, the above step involves the each of the pre-trained deep learning models of the set of pre-trained deep learning models collectively being finetuned on target dataset of images / of the n target datasets of images (where / = 1 ,... ,n, and n is the total number of target datasets of images) and then features are extracted (to be explained below) from those pre-trained deep learning models finetuned on the ithtarget dataset of images before the pre-trained deep learning models are re-initialised to then be fine-tuned on the i+1thtarget dataset of images.

[0104] In an embodiment, the performance of pre-trained deep learning models may be further enhanced by incorporating attention mechanisms and fine-tuning the models with new datasets. This approach may enable the network to focus on the most relevant parts of the input data, thus improving its accuracy and effectiveness in handling complex tasks.

[0105] To achieve this, a Squeeze-and-Excitation (SE) block may be utilized. The SE block recalibrates feature responses by modelling interdependencies between channels. This involves:a. Global Average Pooling: Reducing each channel to a single value by averaging over all spatial dimensions; b. Fully Connected Layers with ReLLI and Sigmoid Activations: Passing the pooled values through fully connected layers with ReLLI activation introducing non-linearity and sigmoid activation scaling the output between 0 and 1 ; and c. Multiplication Operation: Using the output of the sigmoid function to rescale the original feature maps, emphasizing important features and suppressing less relevant ones.

[0106] Additionally, self-attention layers may be employed after every three convolutional layers to effectively capture spatial dependencies. This allows the model to focus on different parts of the input, making it more adept at understanding complex spatial relationships.

[0107] The SE blocks are added after the final convolutional layers of the model to recalibrate the feature maps and emphasize important features. Self-attention layers are integrated into the unfrozen sections of the model, which are fine-tuned. The classification layers are updated to match the number of classes in the new dataset, while the initial layers of the pre-trained model are frozen to retain the learned representations.

[0108] By integrating self-attention and SE blocks into the deep learning model, the model can emphasize important features and enhance its ability to handle complex tasks. This approach leverages the learned representations from the pre-trained network and the enhanced feature extraction capabilities provided by the attention mechanism, resulting in improved accuracy for the target dataset.

[0109] The method then includes extracting features associated with each of the plurality of tasks from the set of pre-trained deep learning models iteratively finetuned on the n target datasets of images at box 140.

[0110] The method utilises the outputs from the set of models to extract features from an ithtarget dataset of images (where / = 1,... ,n) of the n target datasets of image to generate fused feature subsets 151a... n, wherein each fused feature subset 151a... n only includes extracted features in relation to the ithtarget dataset. Subsequently, the extracted features from each of the deep learning models are combined, or fused, at box 150 (this is the feature fusion stage). By combining these various features, the method generates a comprehensive fused feature set of the extracted features encompassing a wide range of relevant characteristics.

[0111] In some embodiments, the combining or fusing of the extracted features is includes first combining the features extracted from the set of pre-trained deep learning models for each task of the plurality of tasks 200 to generate a plurality of fused feature sub-sets 151a... n each relating to one task of the plurality of tasks 200, wherein each fused feature sub-set 151a...n only includes features extracted from the set of pre-trained deep learning models in relation to one of the tasks of the plurality of tasks 200 and subsequently combining the fused feature subsets 151a... n to generate the fused feature set 152 for the plurality of tasks 200.

[0112] The Inventors observed that each of the deep learning models provided unique insights and extracted distinct features from the images. Combining these individual feature sets allows for the complementary information in each model's outputs to be utilised. This fusion process allows for a holistic and nuanced understanding of the contents of an image, thereby enhancing the accuracy and robustness of the classification. By integrating multiple features, the methodleverages the strengths of each deep learning model and overcomes potential limitations associated with relying on a single feature set. The fusion of these features provides of the capture of a broader range of information, including subtle and prominent characteristics of supraspinatus tears. This comprehensive representation provides for more accurate and reliable predictions, facilitating the early and precise identification of image features or content.

[0113] In the next step, the method includes training one or more machine learning classifiers on the fused feature set of the extracted features at box 160. In some embodiments, the machine learning classifiers include K-Nearest Neighbour (KNN), Support Vector Machines (SVM) and Naive Bayes. However, other classifiers may be used.

[0114] These one or more machine learning classifiers are trained on the fused features extracted from the images input to the deep learning models.

[0115] These machine-learning classifiers are useful for developing are automated identification of features by leveraging the patterns and relationships in the data. Through extensive training on the fused features, these classifiers learn to discern the subtle differences and indicative patterns associated with each class. The classifiers develop the ability to discriminate between differences in images to be able to more accurately identify and classify the images.

[0116] By analysing the patterns and relationships within the fused features, the classifiers acquire proficiency in accurately identifying features (e.g., objects or patterns) in the images. The machine classifiers can generalize from the learned features to previously unseen data, allowing them to make reliable predictions regarding the presence or absence of image content. The combination of the fused features and the machine learning classifiers contributes to the overall effectivenessof the method described. The classifiers learn to make informed decisions based on the complex relationships between the features and the target classes. This enables the method to achieve high accuracy in detecting relevant image content based on the tasks.

[0117] In use, there is a method for classifying images. This method may be part of method 10 described above, or form its own separate method 20 shown in Figure 2.

[0118] The method includes receiving images shown at box 170. These images relate to a specific scenario or problem (e.g., medical images or crime detection).

[0119] In some embodiments, depending on the application, the method includes processing the images to generate one or more views at box 175. For example, the method may include processing the MRI images of the shoulder to generate one or more views of the shoulder.

[0120] In some embodiments, the processing to generate one or more views includes automatic processing of the images to generate the one or more views.

[0121] In some embodiments, the images may be 2D images. In some other embodiments, the images may be 3D images, wherein the 3D images are processed to generate one or more 2D views (or images).

[0122] The method then includes applying the images to a pre-trained machine learning classifier to thereby produce identifications and / or classifications of the images at box 180.

[0123] As describe above, the pre-trained machine learning classifier is trained on a fused feature set of features extracted from a set of deep learning models that are pre-trained on an initial image dataset and fine-tuned on images associated with a task.

[0124] More specifically, the set of deep learning models are fine-tuned on n target datasets of images, wherein each target dataset of images is associated with one task of the plurality of tasks, wherein the set of pre-trained deep learning models is iteratively fine-tuned on one of the target datasets of images from 1 to n and the set of pre-trained deep learning models is re-initialised between each iteration.

[0125] The classification, categorisation, identification of image content may include generating an output, or output data, indicating the presence of absence of content in the image.

[0126] The classification, categorisation, identification of image content may include generating an output, or output data, that grades or categorises the image content according to a classification system applied by the machine learning classifier.

[0127] In some embodiments, the method includes fine-tuning the set of pretrained deep learning models on an n+1 target dataset of images associated with an additional task at box 185. The additional task is a new task further to the previously discussed plurality of tasks.

[0128] The method also includes extracting features associated with the additional task from the set of pre-trained deep learning models fine-tuned on the n+1 target dataset of images at box 190 and combining the extracted features from the set of deep learning models fine-tuned on the n+1 target dataset of images with the fused feature set of the extracted features to generate a new fused feature set of extracted features at box 195.

[0129] Next, the method includes training one or more machine learning classifiers on the new fused feature set of the extracted features at box 200.Case Study #1 - Medical Case: Abnormalities Detection in Bone

[0130] Fractures occur frequently in various regions of the body with a broader range of movement compared to other joints, attributed to multiple factors. The objective of this article is to aid medical professionals by implementing a robust deep learning (DL) system to classify images obtained from X-ray devices as negative or positive for fractures. MURA (Musculoskeletal Radiographs) is a large data set of bone X- rays [5] (see Figure 8).

[0131] The MURA dataset contains seven skeletal bones: elbow, finger, forearm, hand, humerus, shoulder, and wrist. Each part has been divided into two subclasses: positive (abnormal) and negative (normal). The total number of images is 40,561. The data set was divided into training and testing sets. As explained in Table 1, five different body bones were considered.Table 1: Number of images of the MURA dataset.Case Study #2 - Surveillance Case: Multi-Scenario Violence Detection

[0132] This study presents a case where multi-scenario violence detection is applied across diverse settings such as streets, prisons, schools, and hockey environments.

[0133] The RLVS dataset [6] comprises 2000 videos with an equal distribution of violent and nonviolent activities. The violent videos within the data set depict physicalconfrontations that occur in various settings such as streets, prisons, and schools, thus exhibiting a wide range of environments. One notable aspect of the data set is its diversity, which encompasses individuals of different genders, races, and ages. The videos are of high resolution, ranging from 480p to 720p, and each video lasts between 3 and 7 seconds.

[0134] The Hockey data set [7] consists of 1000 video clips, each with a size of 360 288 pixels. These video clips are categorised into 500 instances of violent events and 500 cases of non-violent events. The videos were collected from real hockey games played by the National Hockey League (NHL), ensuring the inclusion of authentic violent incidents. The duration of each video clip typically ranges from one to two seconds, with a frame rate of 25 frames per second.

[0135] The movie data set [7] consists of 200 videos divided into 100 violent and 100 nonviolence videos. The violent videos were collected from movie scenes, while the nonviolence videos were collected from other actions. Unlike the hockey dataset, the movie dataset has different backgrounds.

[0136] During the data cleaning process, it was observed that specific frames within violent videos did not accurately represent violent actions and instead resembled frames from normal videos. To enhance the quality and relevance of the training data, these frames, along with any blank or noisy frames, were manually identified and removed. This step ensured that the remaining data for training the models contained more accurate and representative frames related explicitly to violent actions.Case Study #3 - Lateral Changes in Levee Soil Compaction using GroundPenetrating Radar

[0137] Ground Penetrating Radar (GPR) is a well-established technique for monitoring civil structures. By utilizing radar waves, GPR enables imaging of interior sections of walls and subsurface areas. It facilitates mapping support structures, such as reinforcement bars or buried pipes, without destructive measures or excavation. While GPR proves to be an invaluable tool, its output requires the expertise of highly trained specialists for accurate interpretation. Objects detected by GPR are often obscured by noise, and the final image is influenced by both the physical characteristics of the objects themselves and the background.

[0138] Consequently, the interpretation of GPR data can be challenging. The output requires specialised knowledge, and obtaining meaningful insights typically involves a time- consuming process of extensive manual processing and visual inspection. Deep learning can help address this issue. This task produces data from time to time, which means training DL on old and new data with every new set of existing data. This issue can cause catastrophic forgetting issues. The proposed framework helps to alleviate this issue.Experimental Assessment

[0139] This section is devoted to the experimental evaluation of the proposed fusion based on three parts.Evaluation Metrics

[0140] All models with different training scenarios were evaluated based on accuracy, specificity, recall, precision, and F1 score metrics. These evaluation metrics were calculated based on the TN, TP, FN and FP values. TN and TP implied precisely categorized negative and positive instances, while FN and FPdenoted misclassified positive and negative cases, respectively. Every evaluation metric equation is presented as follows:Evaluation of Case Study#1Evaluation of Parti: Transfer learning

[0141] Figure 9 demonstrates a comparison between two models trained for the detection of shoulder abnormalities. The first column displays the original images, with a red circle representing the region of interest marked by an expert. The second column presents a model trained after transfer learning (TL) from ImageNet. In contrast, the third column presents a model trained after TL from the same domain as the target dataset, as proposed in Part 1 of the framework.

[0142] In the first row of the figure, both models correctly predict the image based on their confidence values. However, when examining the heatmap, it becomes evident that the first model is biased and inaccurate, as it fails to detect the region of interest indicated by the red circle. On the other hand, the second model accurately identifies the region of interest with a high confidence value.

[0143] In the second row, it can be observed that the first model misses the correct classification, while the second model correctly classifies the sample. Thisexample illustrates the significance of Part 1 of the framework. It emphasizes that even a model with seemingly correct confidence values cannot always be fully trusted as it may suffer from biases or inaccuracies that prevent it from accurately identifying the desired regions or making correct classifications.

[0144] Table 2 displays the results of the individual models (Xception, lnceptionRes-NetV2) on five distinct X-ray image tasks.Table 2: Results of DL models in the MURA data set test set on Xception and lnceptionResNetV2.

[0145] Figures 10a-e show samples from each task and visually represents how the models focus accurately on the region of interest. The models demonstrate their ability to precisely identify and concentrate on the critical areas indicated by the heatmap.Evaluation of Part 2: Single Task Fusion

[0146] In this section, the features extracted by two DL models (Xception and Inception- ResNetV2) are used to train multiple ML classifiers for each task. XGBoost classifier has been selected as the primary classifier for several reasons. XGBoost provides superior performance in handling structured data and scalability in processing large datasets. Its robustness against overfitting, the ability to handle diverse

[0147] data types, and the ability to provide interpretable results were crucial for multiclass and multitask classification tasks on varied datasets such as medical images, surveillance videos, and geological radar data.

[0148] However, it is to be noted that Support Vector Machines (SVM) also demonstrated similar results to XGBoost in the experiments described herein. SVM is another widely used and powerful classifier known for its ability to handle high-dimensional feature spaces and complex decision boundaries. The decision to use XGBoost as the primary classifier was based on its simplicity, interpretability, and ease of implementation. However, the comparable performance of SVM highlights its potential as an alternative classifier for given tasks. Figures 11a-c show the confusion metrics of the five tasks with XGBoost.

[0149] Table 3 lists the results of DL models on the test set of five tasks with Part2Table 3: Results of DL models on the test set of five tasks with Part 2.

[0150] Figures 12a-c show cases where both models correctly classify the images, but each model focuses on a different region. This observation highlights the potential for improved feature representation by fusing the outputs of these models. This divergence in attention suggests that each model captures distinct and complementary aspects of the data. By combining the outputs of these models, the unique perspectives of each model can be leveraged / used to create a more comprehensive and robust feature representation. The fusion of these diverse features enhances the overall representation of the data and can lead to better classification performance.

[0151] Figures 13a-c illustrate another significant advantage of feature fusion within the proposed framework. In this case, one model failed to correctly classify the image, while the second model accurately classified it. This highlights how feature fusion can enhance the overall robustness of the solution.Evaluation of Part 3: Multi-Task Image Classification

[0152] In part 3 of the methodology, two different approaches were used to validate the effectiveness of the proposed solution.

[0153] Firstly, experiments were conducted involving multiple tasks for two classes. This allowed us to assess the performance and efficacy of the proposed solution in a scenario where multiple tasks needed to be classified into two distinct classes. By applying the solution to this setting, the ability of the method to handle complex classification tasks can be evaluated and its overall effectiveness determined.

[0154] In addition to multiple tasks for two classes, the experiments were extended to include multiple tasks and multiple classes. This further validated the robustness and scalability of the proposed solution across a broader range of classification scenarios. By considering multiple tasks and multiple classes simultaneously, it was possible to assess the solution’s capacity to handle diverse datasets and effectively categorize them into their respective classes.Multi-Bone Scenario abnormalities

[0155] Table 4 and Figures 14a-b illustrate the results of part 3 for the combination of two tasks, including hand and shoulder tasks.Table 4: Results of Part 3: Combination of two tasks.

[0156] Table 5 and Figures 15a-c present the results of part 3, where three different tasks, namely hand, shoulder, and forearm tasks, were combined. The addition of the forearm task to the previous combination of hand and shoulder tasks yielded intriguing findings. The results for the hand and shoulder tasks remained consistent and even showed improvements, particularly in the hand task. This indicates that inclusion of the forearm task did not adversely affect the performance of the classifier on the original tasks. Instead, the model demonstrated its ability to effectively incorporate the new task without requiring extensive retraining on all tasks except the new one.Table 5: Results of Part 3: Combination of three tasks.

[0157] Table 6 and Figures16a-e present the results of part 3, which involved the combination of five different tasks: hand, shoulder, forearm, elbow, and humerus tasks. Building upon the previous three tasks presented in Table 5, the framework was expanded by incorporating two additional tasks, elbow and humerus, without the need for training from scratch. The results demonstrate the exceptional generalization capability of a single classifier across all five tasks, yielding highly impressive performance. The classifier exhibits robustness and adaptability, surpassing the performance of all previous methods applied to the same dataset.Table 6: Results of Part 3: Combination of five tasks.Evaluation of Part 4: Decision Fusion

[0158] This section presents the results obtained from five well-known machine learning classifiers: XGBoost, SVM, KNN, Random Forest, and Naive Bayes. These classifiers were trained with combined features from five tasks and then tested with each task’s test set. In all the experimental evaluations, these classifiers showed high accuracy. To make the final prediction, the majority vote technique was used for the outputs of five classifiers. This means that the output was based on the category that received the highest number of votes among the classifiers. This approach guarantees a strong and well-informed final decision by taking into account the combined input of multiple classifiers. The evolution metrics are shown in Table 7.

[0159] The accuracy scores for the tasks ” Elbow,” ” Shoulder,” ” Forearm,”” Hand,” and ” Humerus” were 94.6%, 99.6%, 95.7%, 100%, and 97.5%, respectively. The specificity scores for ” Elbow,” ” Shoulder,” and ” Forearm” were 97.6%, 99.6%, and 97.8%, respectively. The recall values for the same tasks were 92.4%, 99.6%, and 93.7%. Precision scores were 98.1%, 99.6%, and 98.0% for ” Elbow,” ” Shoulder,” and ” Forearm,” respectively.

[0160] It is important to note that combining decisions from multiple classifiers can significantly improve overall results. This approach, known as decision fusion, enhances accuracy, specificity, recall, precision, and F1 scores, demonstrating the robustness of the classification process. Furthermore, the ability to include more classifiers in the fusion process suggests potential for further refinement and improvement in the system’s performance.Table 7: Results of Part 4: Decision FusionTable 8: Comparison with state-of-the-art methods with regard to accuracy

[0161] To the Inventors’ knowledge, currently, there are no articles available that report a method similar to the disclosed framework to generalise a single classifier or multiple classifiers for tasks of bone abnormalities.

[0162] Previous methods independently worked on individual tasks, providing a DL model for each task without adapting to new classes or data. Table 8 presents a comprehensive comparison of the disclosed framework with state-of-the-art approaches in terms of accuracy in five different bone classification tasks: Elbow, Shoulder, Forearm, Hand, and Humerus.

[0163] Starting with the year 2021 , He et al. [9] achieved an accuracy of 90.0% for the elbow, 85.0% for the shoulder, 86.0% for the forearm, 87.0% for the hand and 93.0% for Humerus.

[0164] Kandel et al.

[0010] reported results in the same year, with accuracies of 84.8%, 75.2%, 81.0%, 76.4%, and 83.3% for the respective bone categories.

[0165] Another study by llysal et al.

[0011] focused on shoulder classification, achieving an accuracy of 84.7% in 2021.

[0166] Moving to 2022, Yao et al.

[0012] demonstrated improved performance with accuracy scores of 87.8% for the elbow, 81. 5% for the shoulder, 83. 2% for the forearm, 81. 3% for the hand, and 88. 7% for Humerus.

[0167] In 2023, two additional studies were published. Alammar et al.

[0013] reported results for the Humerus classification by obtaining 87.8%. Reddy et al.

[0014] presented their findings, achieving accuracies of 92.4%, 89.2%, 89.7%, 91.5%, and 85.6% across the five types of bone.

[0168] The disclosed framework has demonstrated better performance than the state-of-the-art methods, achieving accuracy scores of 94.6% for the elbow, 99.6% for the shoulder, 95.7% for the forearm, 100% for the hand, and 97.5% for Humerus.

[0169] The overall results of the disclosed framework are presented in Table 8, which highlights its effectiveness and superior performance in classifying bone tissue compared to previous methods.Evaluation of Case Study#2Evaluation of Part 1: Transfer learning

[0170] In this section, transfer learning from the ImageNet dataset was used, taking advantage of the shared features between ImageNet and the target dataset. To evaluate the effectiveness of part 1 , one of the datasets that were selected for case study 2 was selected. By selecting a representative dataset, the intention was to assess the performance and adaptability of the transfer learning technique in that specific domain.

[0171] Figure 17 shows the confusion matrix of the Movie dataset with three DL models, including lnceptionResNetV2, lnceptionV3, and Xception. Table 9 depicts the results of the three models on the test set of the Movie dataset.Table 9: Results of Movie dataset with three DL models with Parti .

[0172] To assess the performance of the DL models, Figures 18, 19 and 20 are provided, which showcase the heat maps generated for the three datasets. These heatmaps serve as visual representations that validate the model’s ability to capture the regions of interest within the images. The heatmaps demonstrate the models’ capability to focus on the relevant regions and extract meaningful features for accurate classification.Evaluation of Part 2: Single Task Fusion

[0173] Next, evaluation of the dataset used in part 1 was undertaken by incorporating part 2 of the proposed solution. In this part, the extracted features were fused with three DL models from the dataset. These models were carefully selected to capture and extract relevant information relevant to a specific task. Using these DL models, it was possible to extract rich and discriminative features from the dataset, allowing a comprehensive representation of the underlying data.

[0174] Figure 21 provides valuable insight into the importance of the feature fusion process. It highlights the distinct perspectives of each DL model in the analysis of the samples. Even if one model fails to capture the target region or exhibits a suboptimal performance, the other model can compensate by focusingon the desired region. This fusion-based approach effectively reduces the impact of incorrect predictions and potential biases, enhancing the overall accuracy and robustness of the method.

[0175] The fused features were utilised to train and test three different ML classifiers, including XGBoost, SVM, and KNN, as shown in Figure 22. The results show that the three classifiers performed well on the set of tests of the data set, as shown in Table 10.Table 10: Results of the movie dataset with feature fusion and three ML classifiers-Part 2.Evaluation of Part 3: Multi-Task Image Classification

[0176] This section of the study showcased the performance of Part 3, which involved the combination of three distinct datasets to train a single classifier and then test the same classifier with the three datasets. By combining the datasets, the Inventors envisioned creating a more comprehensive and robust training set that encompasses a wider range of scenarios and variations. To achieve this, the three datasets underwent the preprocessing steps outlined in Part 1 and thefeature extraction process described in Part 2. Following these steps, the Inventors ensured that the data for each dataset was appropriately prepared and transformed to extract meaningful features. The combined data set was used to train a single classifier that could generalize across all three data sets. Table 11 lists the results of the Movie dataset with part 3 using three ML classifiers. The results have not been affected after two other datasets have been added. The same trained classifiers were used to test on two other data sets (RLVS and Hockey) as listed in Table 12 and Table 13, respectively.Table 11 : Results of the movie dataset with Part 3.Table 12: Results of the RLVS dataset with Part3.Table 13: Results of the hockey dataset with Part3.

[0177] The results of both the RLVS and Hockey datasets demonstrate the consistent and reliable performance of the classifiers. The accuracy and effectiveness achieved in the Movie data set were successfully transferred to these new datasets, reaffirming the robustness and generalization capabilities of the method. This highlights the potential of the framework to be applied to various datasets and scenarios in the field of video anomaly detection.Evaluation of Part 4: Decision Fusion

[0178] In Table 14, the results of Part 4, which focusses on the decision fusion of three datasets (Movie, RLVS, and Hockey), are presented. The evaluation metrics, expressed as percentages, highlight the application of the majority voting technique involving three machine learning classifiers. For the movie data set, the Decision Fusion process achieved perfect accuracy, recall, precision and F1 score, all registering at 100%. In the RLVS dataset, the model demonstrated high performance with 99.0% accuracy, 99.3% recall, 98.7% precision, and a 99.0%F1 score. Similarly, for the Hockey dataset, the Decision Fusion exhibited strong performance, yielding 95.8% accuracy, 96.3% recall, 95.7% precision, and a 96.0% F1 score. These outcomes underscore the effectiveness of the Decision Fusion approach across diverse datasets, showcasing its ability to achieve robust and accurate results.Table 14: Results of Part 4: Decision FusionComparison with state-of-the-art methods

[0179] Table 15 comprehensively compares the proposed framework with state- of-the-art methods across different datasets (RLVS, Hockey, and Movie).

[0180] In the RLVS dataset, the disclosed framework achieved an accuracy of 99.0%, outperforming the previously reported highest accuracy of 97.6%.

[0181] For the Hockey dataset, the disclosed framework demonstrated superior performance with an accuracy of 95.8%, compared to other methods ranging from 86.2% to 92.8%.

[0182] In the Movie dataset, the disclosed framework achieved a perfect accuracy of 100%, showing its effectiveness compared to other methods with accuraciesranging from 88.7% to 100%.

[0183] These results emphasise the superior performance and versatility of the disclosed framework across various datasets compared to existing state-of-the-art approaches.Table 15: Comparison with state-of-the-art methods with regard to accuracy.Evaluation of Case Study#3

[0184] With reference to Figures 23 to 26 and the above, his case study examines the concept of incorporating additional datasets at a later stage without requiring retraining of both the existing and new datasets from scratch. Furthermore, it addresses the challenge of catastrophic forgetting by introducing a new set of data. Same-domain transfer learning was used by using images similar to target images. Two DL models have been trained and tested, including Xception and InceptionResNet.Evaluation of Part 1 and 2 training on First part of dataset

[0185] Table 16 presents the results obtained from applying the fusion technique of two DL models with three ML classifiers in the first part of the data set. The fusion technique combines the features extracted from the DL models and utilizes them for training the ML classifiers. The performance of the classifiers in this part of the dataset is evaluated, and the corresponding results are reported in the table.Table 16: Results of Parts 1 and 2 trained in the first part of the data set.Evaluation of Part 1 and 2 training on additional data with same test set

[0186] Table 17 displays the results of the application of the fusion technique of the other two DL models with the three ML classifiers in the second part of the data set. Both parts of the data set are evaluated with the same set of tests to ensure fair comparison and evaluation.Table 17: Results of Parts 1 and 2 trained on additional data from the same task.Evaluation of Part 3 on the whole dataset

[0187] In this section, the features extracted from parts 1 and 2 of the data set are fused to train the ML classifier, as shown in Figure 21 and outlined in Table 18. The fusion of features from both parts enables the ML classifier to take advantage of the combined information and patterns present in the data.

[0188] The results obtained from this fusion approach demonstrate a notable improvement compared to using the individual parts separately. This improvement validates the efficacy of the proposed framework, enabling enhancedperformance without the need to train the classifier from scratch. Additionally, the framework successfully addresses the issue of catastrophic forgetting, which refers to the phenomenon in which the introduction of new data negatively impacts the classifier’s ability to recall previously learned information. The proposed framework offers several advantages. Firstly, it eliminates the necessity of training separate models for each dataset part, saving time and computational resources. Second, it facilitates the integration of additional data in a more efficient manner, enabling the classifier to benefit from a larger and more diverse dataset. Lastly, the improved results obtained through the fusion of features highlight the potential of the proposed framework to deliver better performance in various applications.Table 18: Results of Part 3 of the whole dataset.

[0189] Figure 22 illustrates the successful capture of the region of interest by the used models, as indicated by the red shape. The region of interest represents the specific areas or features of the images that are relevant for classification. By accurately identifying and capturing this region, the models demonstrate theirability to focus on the critical elements necessary for effective image classification.

[0190] Figure 23 provides evidence of the significance of the proposed framework. Each individual model showcased in the figure demonstrates a strong focus on a specific portion of the region of interest. This focus of attention highlights the models’ ability to capture and analyze distinct aspects of the input data.

[0191] However, the true strength of the proposed framework lies in the fusion of these individual models. By combining their respective strengths and areas of focus, the fused model ensures comprehensive coverage of the entire region of interest. The fusion of models addresses the limitations of relying solely on individual models and enhances the overall performance and effectiveness of the classification process.

[0192] Figure 24 presents further evidence that highlights the importance of the proposed framework. The figure illustrates a scenario where two models are employed to classify an input, and their predictions are compared. This comparison reveals an important aspect of the framework, namely its ability to mitigate the risk of incorrect predictions by leveraging multiple models.

[0193] If the first model does not correctly classify the input, the second model provides the correct classification. This observation is crucial as it demonstrates that by not relying solely on a single model, the proposed framework reduces the likelihood of wrong predictions and enhances the overall accuracy of the classification process.Evaluation of Part 4: Decision Fusion

[0194] The results of part 4 of the disclosure are presented in Table 19, which focusses on Decision Fusion. The Decision Fusion results of the disclosedframework are compared with the only available reference from previous work, specifically, Alzubaidi et al. [8] in 2024. The comparison includes two scenarios: the test set performance and an independent test.

[0195] The disclosed framework consistently demonstrates outstanding results in the evaluation metrics: accuracy, recall, precision, and F1 score.

[0196] In the test set, Decision Fusion achieves perfect accuracy (100%), recall (100%), precision (100%), and F1 Score (100%). Even in the more challenging scenario of the independent test, the performance remains robust with an accuracy of 98.6%, recall of 99.4%, precision of 98.3%, and F1 Score of 98.9%.

[0197] Compared to reference work [8], the reported metrics are slightly lower. That method achieves an accuracy of 98.2%, recall of 97.1%, precision of 98.0%, and F1 Score of 97.6% in the test set. However, in the independent test, the metrics are further reduced with an accuracy of 95.7%, recall of 98.1%, precision of 94.4%, and F1 Score of 96.2%. The results show that the disclosed framework outperformed the state-of-the-art method referenced in both the test set and an independent evaluation. Achieving perfect scores in key metrics on the test set demonstrates the disclosed framework’s capability, and its high performance on the independent test highlights its robustness and generalisability.Table 19: Results of Part 4: Decision Fusion Results and Comparison with the state-of-the-art method

[0198] In compliance with the statute, the invention has been described in language more or less specific to structural or methodical features. The term “comprises” and its variations, such as “comprising” and “comprised of” is used throughout in an inclusive sense and not to the exclusion of any additional features.

[0199] It is to be understood that the invention is not limited to specific features shown or described since the means herein described comprises preferred forms of putting the invention into effect.

[0200] The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended claims appropriately interpreted by those skilled in the art.

[0201] References[1] Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)[2] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251-1258 (2017)[3] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mo- bilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520 (2018)[4] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105-6114 (2019). PMLR[5] Rajpurkar, P., Irvin, J., Bagul, A., Ding, D., Duan, T., Mehta, H., Yang, B., Zhu, K., Laird, D., Ball, R.L., et al.: Mura: Large dataset for abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv:1712.06957 (2017)[6] Soliman, M.M., Kamal, M.H., Nashed, M.A.E.-M., Mostafa, Y.M., Chawky, B.S., Khattab, D.: Violence recognition from videos using deep learning techniques. In: 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 80-85 (2019). IEEE[7] Bermejo Nievas, E., Deniz Suarez, O., Bueno Garc'ia, G., Sukthankar, R.: Violence detection in video using computer vision techniques. In: Computer Analysis of Images and Patterns: 14th International Conference, CAIP 2011,Seville, Spain, August 29-31 , 2011, Proceedings, Part II 14, pp. 332-339(2011). Springer[8] Alzubaidi, L., Chlaib, H.K., Fadhel, M.A., Chen, Y., Bai, J., Albahri, A.S., Gu, Y.:Reliable deep learning framework for the ground penetrating radar data to locate the horizontal variation in levee soil compaction. Engineering Applications of Artificial Intelligence 129, 107627 (2024).[9] He, M., Wang, X., Zhao, Y.: A calibrated deep learning ensemble for abnormality detection in musculoskeletal radiographs. Scientific Reports 11(1), 9097 (2021)

[0010] Kandel, I., Castelli, M., Popovic, A.: Comparing stacking ensemble techniques to improve musculoskeletal fracture image classification. Journal of Imaging 7(6), 100 (2021)

[0011] llysal, F., Hardalac, F., Peker, O., Tolunay, T., Tokgoz, N.: Classification of shoulder x-ray images with deep learning ensemble models. Applied Sciences 11(6), 2723 (2021)

[0012] Yao, J., Guo, Z., Yu, W.: Enhanced deep residual network for bone classification and abnormality detection. Medical Physics 49(11), 6914-6929 (2022)

[0013] Alammar, Z., Alzubaidi, L., Zhang, J., Li, Y., Lafta, W., Gu, Y.: Deep transfer learning with enhanced feature fusion for detection of abnormalities in x-ray images. Cancers 15(15), 4007 (2023)

[0014] Reddy, K.N.K., Cutsuridis, V.: Deep convolutional neural networks with transfer learning for bone fracture recognition using small exemplar imagedatasets. In: 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 1-5 (2023). IEEE

[0015] Alzubaidi, L., Salhi, A., A. Fadhel, M., Bai, J., Hollman, F., Italia, K., Pareyon, R., Albahri, A., Ouyang, C., Santamaria, J., et al . Trustworthy deep learning framework for the detection of abnormalities in x-ray shoulder images. Pios one 19(3), 0299545 (2024)

[0016] Traore, A., Akhloufi, M.A.: Violence detection in videos using deep recurrent and convolutional neural networks. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 154-159 (2020). IEEE

[0017] Bi, Y., Li, D., Luo, Y.: Combining keyframes and image classification for violent behavior recognition. Applied Sciences 12(16), 8014 (2022)

[0018] Chexia, Z., Tan, Z., Wu, D., Ning, J., Zhang, B.: A generalized model for crowd violence detection focusing on human contour and dynamic features. In: 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pp. 327-335 (2022). IEEE

[0019] Jebur, S.A., Hussein, K.A., Hoomod, H.K., Alzubaidi, L.: Novel deep feature fusion framework for multi-scenario violence detection. Computers 12(9), 175 (2023)

[0020] Elmasry, R.M., Abd El Ghany, M.A., Salem, M.A.-M., Fahmy, O.M.: Multiwave-net: An optimized spatiotemporal network for abnormal action recognition using wavelet-based channel augmentation. Al 5(1), 259-289 (2024)

[0021] Tu, Y.-S., Shen, Y.-S., Chan, Y.Y., Wang, L., Chen, J.: Violent video recognition by using sequential image collage. Sensors 24(6), 1844 (2024)

[0022] Duraes, D., Santos, F., Marcondes, F.S., Lange, S., Machado, J.: Comparison of transfer learning behaviour in violence detection with different public datasets. In: Progress in Artificial Intelligence: 20th EPIA Conference on Artificial Intelligence, EPIA 2021 , Virtual Event, September 7-9, 2021, Proceedings 20, pp. 290-298 (2021). Springer

[0023] Garcia-Cobo, G., SanMiguel, J.C.: Human skeletons and change detection for efficient violence detection in surveillance videos. Computer Vision and Image Understanding 233, 103739 (2023)

[0024] Sharma, M., Baghel, R.: Video surveillance for violence detection using deep learning. In: Advances in Data Science and Management:Proceedings of ICDSM 2019, pp. 411-420 (2020). Springer

[0025] Hu, X., Fan, Z., Jiang, L., Xu, J., Li, G., Chen, W., Zeng, X., Yang, G.,Zhang, D.: Top-alcm: A novel video analysis method for violence detection in crowded scenes. Information Sciences 606, 313-327 (2022)

Claims

CLAIMS1. A method for training an image classifier for multi-task image classification, the method comprising: providing a set of deep learning models pre-trained on an initial image dataset; fine-tuning the set of pre-trained deep learning models on n target datasets of images, wherein each target dataset of images is associated with one task of a plurality of tasks and wherein the set of pre-trained deep learning models is iteratively fine-tuned on one of the target datasets of images from 1 to n and the set of pre-trained deep learning models is re-initialised between each iteration, extracting features associated with each of the plurality of tasks from the set of pre-trained deep learning models iteratively fine-tuned on the n target datasets of images; combining the extracted features from each of the fine-tuned set of deep learning models to generate a fused feature set for the plurality of tasks; and training one or more machine learning classifiers on the fused feature set of the extracted features.

2. The method according to claim 1, wherein after providing two or more deep learning models pre-trained on an initial image dataset, the method further comprises updating the pre-trained deep learning models on a secondary image dataset of images of a same image type as images of a target dataset.

3. The method according to claim 1 or claim 2, wherein combining the extracted features from each of the fine-tuned set of deep learning models to generate a fused feature set for the plurality of tasks includes first combining the features extracted from the set of pre-trained deep learning models for each task of the plurality of tasksto generate a plurality of fused feature sub-sets each relating to one task of the plurality of tasks, wherein each fused feature sub-set only includes features extracted from the set of pre-trained deep learning models in relation to one of the tasks of the plurality of tasks and subsequently combining the fused feature subsets to generate the fused feature for the plurality of tasks.

4. The method according to any one of claims 1 to 3, wherein the method includes identifying one or more ideal machine learning classifiers from the trained machine learning classifiers based on one or more performance metrics.

5. The method according to any one of claims 1 to 4, wherein fine-tuning the set of pre-trained deep learning models includes freezing initial layers of the pre-trained models and any remaining layers are trained on the images associated with the task to adapt the two or more pre-trained deep learning models to the task.

6. The method according to claim 5, wherein freezing the initial layers includes maintaining weights and parameters of the set of pre-trained deep learning models.

7. The method according to any one of claims 1 to 6, wherein the one or more machine learning classifiers include K-Nearest Neighbour (KNN), Support Vector Machines (SVM), Random Forest, XGBoost and Naive Bayes.

8. The method according to any one of claims 1 to 7, wherein the set of pretrained deep learning models include two or more convolutional neural networks (CNNs).

9. The method according to claim 8, wherein the set of pre-trained deep learning models include at least two of the Xception model, the lnceptionV3 model and the MobileNetV2 model.

10. The method according to any one of claims 1 to 9, wherein the set of pretrained deep learning models are pre-trained on the ImageNet dataset.

11. The method according to claim 10, wherein the method includes pre-training the set of deep learning models on an initial image dataset.

12. The method according to any one of claims 1 to 11, wherein the method further includes: fine-tuning the set of pre-trained deep learning models on an n+1 target dataset of images associated with an additional task; extracting features associated with the additional task from the set of pretrained deep learning models iteratively fine-tuned on the n+1 target dataset of images; combining the extracted features from the set of deep learning models finetuned on the n+1 target dataset of images with the fused feature set of the extracted features to generate a new fused feature set of extracted features; and training one or more machine learning classifiers on the new fused feature set of the extracted features.

13. A method of classifying images comprising: receiving images; and applying the images to one or more pre-trained machine learning classifiers to thereby produce identifications and / or classifications based on a plurality of tasks, wherein the one or more pre-trained machine learning classifiers are: trained on a fused feature set of features extracted from a set of deep learning models that are pre-trained on an initial image dataset; wherein the set of deep learning models are fine-tuned on n target datasets of images, wherein each target dataset of images is associated with one task of the plurality of tasks, wherein the set of pre-trained deep learning models is iterativelyfine-tuned on one of the target datasets of images from 1 to n and the set of pretrained deep learning models is re-initialised between each iteration.

14. The method according to claim 13, wherein the method includes processing the images to generate one or more views.

15. The method according to claim 13 or claim 14, wherein the method includes automatic processing of the images to generate the one or more views.

16. The method according to any one of claims 13 to 15, wherein the method includes identifying one or more ideal machine learning classifiers from the one or more pre-trained machine learning classifiers based on one or more performance metrics.

17. The method according to any one of claims 13 to 16, wherein the method includes combining decisions of the one or more pre-trained machine learning classifiers, or the one or more ideal machine learning classifiers, using majority voting to produce the identifications and / or classifications.

Images