System and method for generating synthetic images

Abstract

Embodiments relate to a system (1) and method for generating synthetic images. The system (1) comprising an image acquisition module (10) configured to receive video or sequence of images containing object to be used to train a deep learning model; a processing module (20) configured to generate a generative model to produce a plurality of synthetic images, generate a plurality of synthetic image clusters, perform similarity analysis on a real image and each synthetic image cluster, and filter the synthetic images based on the similarity analysis, wherein the real image referred to the video or sequence of images containing object to be used to train the deep learning model received from the image acquisition module (10); and an output module (30) configured to send filtered synthetic images to train the deep learning model to perform object classification, detection, or other image-based task,

Description

[0001] System And Method for Generating Synthetic Images

[0002] Field of Invention

[0003] The present invention relates to a system and method for generating synthetic images. More particularly, the present invention relates to system and method for automatically generating and filtering synthetic images.

[0004] Background of Invention

[0005] In this day and age, deep learning is widely implemented, whereby an example of such implementation is the use of deep learning in visual inspection to detect defects in manufacturing industry. Deep learning model is trained to detect and classify an object by using array of images. Synthetic images are used to overcome the issue of limited image data available for deep learning training. Therefore, many systems and methods have been invented to automatically generate synthetic images to be used to train deep learning.

[0006] An example of such system and method is disclosed in United States Patent No. US11537813B1 which relates to a system for generating and assessing suitable synthetic image data to be used as input to train machine learning system. The system generates latent vectors from latent vector spaces and uses the latent vectors as input to a first machine learning system to generate a candidate synthetic image data. The candidate image data is assessed to determine it’s suitability for inclusion into a set of synthetic image data that may be used for subsequent use in training a second machine learning system to recognize an identity of a hand presented by a user. For example, the candidate synthetic image data is compared to previously generated synthetic image data to avoid duplicative synthetic identities.

[0007] Another example of a system and method for automatically generating synthetic images is disclosed in International Patent Publication No WO 2023 / 121846 A1 which relates to a method for pre-processing training data, augmenting training data, and using synthetic training data to effectively train a machine learning model. The system obtains a training set of images for training a machine learning algorithm to detect and classify regions or objects within the images. Thereon, the system augments this training set with adversarial examples generated by adversarial algorithms that modify certain image variables to create synthetic images with various adversarial features. Finally, the system trains the machine learning algorithm using the augmented images to develop a model capable of detecting and classifying regions or objects in new images.

[0008] Although synthetic images can alleviate the lack of image data available to train deep learning models, using automatically generated synthetic images to train classification or detection deep learning models can have either outcome; a better accuracy or a worsen accuracy. This is because a generative model used to automatically generate synthetic images might have overfitted to limited data input and / or generated bad synthetic images which amplify error. Therefore, using suboptimal generated synthetic images for training classification or detection model will lead to decreased performance of the deep learning. Thus, there is a need for a system and method for automatically generating synthetic images which address the aforementioned problem.

[0009] Summary of Invention

[0010] Embodiments relate to a system (1) and method for generating synthetic images. The system (1) comprising an image acquisition module (10) configured to receive video or sequence of images containing object to be used to train a deep learning model; a processing module (20) connected to the image acquisition module (10) wherein the processing module is configured to generate a generative model to produce a plurality of synthetic images, generate a plurality of synthetic image clusters, perform similarity analysis on a real image and each synthetic image cluster, and filter the synthetic images based on similarity analysis, wherein the real image referred to an image containing object to be used to trained the deep learning model received from the image acquisition module (10); and an output module (30) connected to the processing module (20), wherein the output module (30) is configured to send filtered synthetic images to train the deep learning model to perform object classification, detection, or other image-based task. The processing module (20) generates the plurality of synthetic image clusters according to distance between feature embedding vectors of each synthetic image and cluster centre of each synthetic image cluster; and performs similarity analysis on the real image and each synthetic image cluster by computing semantic and visual similarity between the real image and the synthetic image clusters.

[0011] Preferably, the processing module (20) is further configured to automatically finetune subsequent generative model training by using a weighted combination of a plurality of real images and a plurality of synthetic images sampled from each synthetic image cluster. In a preferred embodiment, the processing module (20) further comprising a generator (21) configured to generate the generative model to produce the plurality of synthetic images and generate a plurality of synthetic image clusters by classifying each synthetic image according to distance between the feature embedding vectors of each synthetic image and cluster centre of each synthetic image cluster; and a discriminator (22) configured to perform the similarity analysis on the real image and each synthetic image cluster using semantic features and visual distance, filter the synthetic images based on the semantic-visual similarity, and automatically finetune subsequent generative model by using the weighted combination of a plurality of real images and a plurality of synthetic images sampled from each synthetic image cluster.

[0012] Moreover, the image acquisition module (10) is further configured to assign a label to the real image.

[0013] In another aspect of the invention, a method for generating synthetic images includes the steps of acquiring a real image by an image acquisition module (10), wherein the real image refers to an image containing object to be used to train a deep learning model; generating a generative model by a generator (21); generating a plurality of synthetic images by the generative model; clustering the plurality of synthetic images into a plurality of synthetic image clusters according to distance between feature embedding vectors of each synthetic image and cluster centre of each synthetic image cluster by the generator (21); performing a semantic and visual similarity analysis on the real image and each synthetic image cluster by a discriminator (22); comparing an overall similarity score of each synthetic image cluster with a preset similarity score threshold by the discriminator (22), wherein the preset similarity score threshold is determined by a range of variance on the synthetic image that is acceptable; determining whether percentage of synthetic image clusters with the overall similarity score which is less than the preset similarity score threshold is equal to or more than a preset finetune threshold by the discriminator (22), wherein the preset finetune threshold is determined according to a desired performance of the generative model; filtering the plurality of synthetic images by removing synthetic images in the synthetic image cluster with the overall similarity score equals to or greater than the preset similarity score threshold by the discriminator (22) if the percentage of synthetic image clusters with the overall similarity score which is less than the preset similarity score threshold is equal to or more than the preset finetune threshold; and performing finetuning on the generative model by the discriminator (22) if the percentage of synthetic image clusters with the overall similarity score which is less than the preset similarity score threshold is less than the preset finetune threshold. Preferably, generating the generative model by the generator (21) includes the sub-steps of formatting the real image in a current dataset according to input requirement of a chosen generative training algorithm; determining whether there are synthetic image clusters and their weightage value stored in a database (40); initialising a generative architecture from machine learning or deep learning approach if there are no synthetic image clusters and their weightage value stored in the database (40); and training the generative model with the current dataset. On the other hand, if there are synthetic image clusters and their weightage value stored in the database (40), the sub-steps include retrieving an input dataset from the discriminator (22), wherein the input dataset comprises a weighted combination of real images and samples of synthetic images from each synthetic image cluster; formatting the samples of synthetic images from each synthetic image cluster and real images retrieved from the input dataset according to the input requirements of the chosen generative training algorithm; adding the samples of synthetic images from each synthetic image cluster and real images retrieved from the input dataset to the current dataset; initialising the generative architecture from machine learning or deep learning approach; and training the generative model with the current dataset.

[0014] Preferably, clustering synthetic images into a plurality of synthetic image clusters according to distance between feature embedding vectors of each synthetic image and the cluster centre of each synthetic image cluster by the generator (21) further comprising the sub-steps of extracting semantic feature vector from each synthetic image by removing an output layer from the synthetic image using a pretrained neural network model; retrieving feature embedding of each synthetic image by reducing dimensionality of feature vectors of the synthetic image; fitting a clustering algorithm to feature space to obtain cluster centres for a number of clusters; obtaining distance vector of each synthetic image by using distance measure to compute distance between feature embeddings of each synthetic image and each cluster centre; and assigning each synthetic image to a cluster according to the distance vector.

[0015] Typically, performing a semantic and visual similarity analysis on the real image and each synthetic image cluster by the discriminator (22) further comprising the sub-steps of computing semantic similarity between the real image and each synthetic image of each synthetic image cluster; computing visual similarity between the real image with each synthetic image of each synthetic image cluster; and obtaining the overall similarity score between the real image to each synthetic image cluster.

[0016] In addition, computing semantic similarity between the real image and each synthetic image of each synthetic image cluster further comprising the sub-steps of extracting semantic feature vector from the real image by removing an output layer from the real image using a pretrained neural network model; retrieving feature embedding of the real image by reducing dimensionality of feature vectors of the real image; loading the cluster centre of each synthetic image cluster; initialising clustering algorithm; and obtaining distance vector between the real image and the cluster centre of each synthetic image cluster.

[0017] Additionally, computing visual similarity between the real image with each synthetic image of each synthetic image cluster further comprising the sub-steps of retrieving a number of samples of synthetic images from each synthetic image cluster based on the distance between the synthetic image to its corresponding cluster centre; extracting query keypoints and query descriptors from each synthetic image sampled from each synthetic image cluster; extracting sample keypoints and sample descriptors from the real image; determining top two matches between the query descriptors of each synthetic image sampled from each synthetic image cluster and each sample descriptor of the real image by using distance measures, wherein the top two matches for each sample descriptor of the real image are two query descriptors among all synthetic images sampled from the synthetic image cluster with the shortest distance value; filtering for unique matches between the real image and each synthetic image from each synthetic image cluster by comparing the top two matches between the query descriptors of the synthetic image and each sample descriptor of the real image; computing uniqueness percentage for each synthetic image sampled from each synthetic image cluster; computing coverage percentage for each synthetic image sampled from each synthetic image cluster; computing mean distance of the unique matches for each synthetic image sampled from each synthetic image cluster; tuning similarity distance value for each synthetic image sampled from each synthetic image cluster; computing mean distance for each synthetic image cluster; and penalising the mean distance for each synthetic image cluster by ratio of samples with zero unique matches.

[0018] Preferably, the number of synthetic images retrieved from each synthetic image cluster depends on the capability of the discriminator (22), and wherein N number of samples closest to its corresponding cluster centre are retrieved for each synthetic image cluster.

[0019] Furthermore, filtering for unique matches between the real image and each synthetic image from each synthetic image cluster by comparing the top two matches between the query descriptors of the synthetic image and each sample descriptor of the real image includes the sub-steps of computing ratio between the distance value of best match, ki with the distance value of second-best match, k2, wherein ki refers to the query descriptor with the shortest distance value to the sample descriptor, whereas k2 refers to the query descriptor with the second shortest distance value to the sample ;and comparing the ratio between distance value of ki with the distance value of k2 with a predetermined threshold, wherein the predetermined threshold is set according to a required parameter for query descriptors of the synthetic image to be considered as a correct match with the sample descriptor of the real image, and wherein a match is considered as a unique match if the ratio between the distance value of ki with the distance value of k2 equals to or less than the predetermined threshold.

[0020] Moreover, computing uniqueness percentage for each synthetic image sampled from each synthetic image cluster is by dividing number of unique matches of the synthetic image with number of total matches found between each query descriptor of the synthetic image and the sample descriptors of the real image.

[0021] In addition, computing coverage percentage for each synthetic image sampled from each synthetic image cluster is based on the following equation: wherein Nunique represents number of unique matches of the synthetic image, NqUery_kPrepresents number of query keypoints of the synthetic image, and NSampie_kPrepresents number of sample keypoints of the real image.

[0022] Additionally, computing mean distance of the unique matches for each synthetic image sampled from each synthetic image cluster is by averaging total distance value of all unique matches of the synthetic image across total number of unique matches of the synthetic image.

[0023] Moreover, tuning similarity distance value for each synthetic image sampled from each synthetic image cluster is according to the following equation: wherein S represents similarity distance value, %Unique uniqueness percentage of the synthetic image, %cover represent coverage percentage of the synthetic image, and dmean represent mean distance of the unique matches of the synthetic image. Furthermore, computing mean distance for each synthetic image cluster is by averaging the similarity distance value of all of the synthetic images sampled from the synthetic image cluster across the number of samples of synthetic images for the synthetic image cluster.

[0024] In addition, the mean distance for each synthetic image cluster by ratio of samples with zero unique matches is according to the following equation: wherein Smean represents mean distance of the synthetic image cluster and %SamPies_zero_unique represents percentage of synthetic image sample with zero unique matches within the synthetic image cluster.

[0025] Additionally, obtaining the overall similarity score between the real image to each synthetic image cluster further comprising the sub-steps of normalising semantic similarity score; normalising visual similarity score; and computing weighted average of semantic and visual similarity score.

[0026] Preferably, the variance on the synthetic images in the synthetic image cluster is considered to be acceptable if the overall similarity score of the synthetic image cluster is less than the preset similarity score threshold. Thus, a lower preset similarity score threshold means that less variance is acceptable, whereas a higher preset similar score threshold means that more variance is acceptable

[0027] Preferably, performing finetuning on the generative model by the discriminator (22) if the percentage of synthetic image clusters with the overall similarity score that is less than the preset similarity score threshold is less than the preset finetune threshold further comprising the sub-steps of computing weightage values for each synthetic image cluster from the overall similarity score of each synthetic image cluster; normalising the computed weightage value for each synthetic image cluster; computing the number of synthetic images to sample from each synthetic image cluster based on the weightage values for each synthetic image cluster; adding the weighted combination of the real image and samples of synthetic images from each synthetic image cluster to an input dataset; and repeating from the step of generating a generative model by the generator (21), wherein a new generative model is generated using the input dataset. Brief Description of the Drawings

[0028] The drawings constitute a part of this specification and include an exemplary or preferred embodiment of the invention, which may be embodied in various forms. It should be understood, however, the disclosed preferred embodiment is merely exemplary of the invention. Therefore, the figures disclosed herein are not to be interpreted as limiting, but merely as the basis for the claims and for teaching one skilled in the art of the invention.

[0029] Figure 1 shows a block diagram of a system (1) for generating synthetic images according to an embodiment of the present invention.

[0030] Figure 2 shows a flowchart of a method for generating synthetic images according to an embodiment of the present invention.

[0031] Figure 3 (a) shows an example of a real image of a metal oxide silicon field effect transistor, a MOSFET.

[0032] Figure 3 (b) shows an example of a synthetic image of MOSFET generated from the real image of Figure 3 (a)

[0033] Figure 4 shows a flowchart of sub-steps for generating the generative model by a generator of the method of Figure 2.

[0034] Figure 5 shows a flowchart of the sub-steps for clustering synthetic images into synthetic image clusters by the generator of the method of Figure 2.

[0035] Figures 6 (a) and (b) show flowcharts of sub-steps for performing similarity analysis on a real image and each synthetic image cluster by a discriminator of the method of Figure 2.

[0036] Detailed Description of Preferred Embodiments

[0037] Detailed description of preferred embodiments of the present invention is disclosed herein. It should be understood, however, that the embodiments are merely exemplary of the present invention, which may be embodied in various forms. Therefore, the details disclosed herein are not to be interpreted as limiting, but merely as the basis for the claim and for teaching one skilled in the art of the invention. The numerical data or ranges used in the specification are not to be construed as limiting. An exemplary embodiment of the present invention can be seen in Figure 1 which illustrates a block diagram of a system (1) for generating synthetic images. The system (1) automatically generates a plurality of synthetic images and filters the generated synthetic images by comparing the generated synthetic images to a real input image using semantic and visual similarity. The filtered synthetic images will be used to train a deep learning model to perform object classification, detection, or other image-based tasks. The system (1) comprises an image acquisition module (10), a processing module (20), an output module (30), and a database (40). The image acquisition module (10), the processing module (20) and the output module (30) are all connected to the database (40).

[0038] The image acquisition module (10) is configured to receive video or sequence of images containing object to be used to train a deep learning model, wherein an image containing object to be used to train the deep learning model received by the image acquisition module (10) will be referred to herein onward as a real image. The image acquisition module (10) can be any device capable of receiving signals from vision sensors and represent it in digital form of 2- dimensionals, 2D pixels. For example, the image acquisition module (10) is a camera. The image acquisition module (10) is also configured to assign a label to the real image, wherein the label refers to the category or class assigned to the image data. The image acquisition module (10) stored the data of the real image together with the label in the database (40) as a current dataset. Optionally, the image acquisition module (10) may also obtain the real image from other sources, such as the internet.

[0039] The image acquisition module (10) is connected to the processing module (20) to send the current dataset to the processing module (20). In general, the processing module (20) is configured to generate a generative model to produce a plurality of synthetic images from the real image and to generate a plurality of synthetic image clusters. The processing module (20) is also configured to perform similarity analysis on the real image and each synthetic image cluster. The processing module (20) further filters the synthetic images based on semantic- visual similarity.

[0040] The processing module (20) comprises a generator (21) and a discriminator (22) which are interconnected to each other.

[0041] In the preferred embodiment of the present invention, the generator (21) is configured to generate the generative model, wherein the generative model produces a plurality of synthetic images from the real image. Examples of the generative model include but are not limited to Generative Adversarial Networks, GANs, text-to image diffusion models such as Stable Diffusion, a latent diffusion model commonly known in the art, and any other generative model capable of producing synthetic images. The generator (21) is also configured to generate a plurality of synthetic image clusters by classifying each synthetic image according to distance between feature embedding vectors of each synthetic image and a cluster centre of each synthetic image cluster. The generative model, synthetic images, and synthetic image clusters are forwarded to the database (40) to be stored for future use.

[0042] The discriminator (22) is configured to perform the similarity analysis on the real image and each synthetic image cluster using semantic features and visual distance, and filter the synthetic images based on the semantic-visual similarity until number of images is considered sufficient for the subsequent deep learning task such as object classification, detection, or other image-based task. The number of images required for the deep learning task is depending on various factors, including complexity of the task, the diversity of the dataset, the chosen model, and the desired performance level.

[0043] The discriminator (22) is further configured to automatically finetune subsequent generative model using a weighted combination of real images and samples of synthetic images from each synthetic image cluster.

[0044] The processing module (20) is further connected to the output module (30) to send the filtered synthetic images to the output module (30). The output module (30) uses the filtered synthetic images to train the deep learning model to perform object classification, detection, or other image-based tasks. The output module (30) is also configured to display result of the filtered synthetic images on a display screen.

[0045] Figure 2 illustrates a flowchart of a method for generating synthetic images according to an embodiment of the present invention.

[0046] Initially, an image acquisition module (10) such as a camera or scanner acquires a real image as sample image data as in step 100. The image acquisition module (10) may either capture the real image directly or obtain the real image from other sources, such as an image stored in a fixed or portable electronic storage device or the internet. Once the real image is obtained, the image acquisition module (10) assigns a label to the real image. Thereon, the image acquisition module (10) sends the real image together with the label to the database (40) to be stored as the current dataset. Simultaneously, the image acquisition module (10) forwards the current dataset to the processing module (20) for further action. As a non-limiting example, the image acquisition module (10) captures an image of an electronic component such as a metal oxide silicon field effect transistor, MOSFET with a bent leg as shown in Figure 3 (a). The bent leg of the MOSFET is a form of defect, therefore, the image acquisition module (10) assigns the label of “fail” to the image of MOSFET captured.

[0047] Thereon, the image acquisition module (10) sends the image of the MOSFET captured as the real image together with the assigned label to the database (40) and processing module (20) as the current dataset.

[0048] Thereon, the generator (21) generates a generative model as in step 200. The generator (21) obtains the real image for a chosen generative algorithm training from the image acquisition module (10) and generates the generative model based on the machine learning or deep learning approach. The sub-steps of generating the generative model will be further explained with reference to Figure 4.

[0049] In step 300, the generative model generates a plurality of synthetic images. The generator (21) then sends the synthetic images to the database (40) for storing.

[0050] In step 400, the generator (21) clusters the synthetic images into a plurality of synthetic image clusters, by classifying each synthetic image according to distance between feature embedding vectors of each synthetic image and the cluster centre of each synthetic image cluster. The sub-steps for clustering the synthetic images into a plurality of synthetic image clusters will be further explained with reference to Figure 5.

[0051] In step 500, the discriminator (22) performs local similarity analysis on the real image and each synthetic image cluster. The discriminator (22) computes semantic similarity and visual similarity between the real image and the synthetic images of each synthetic image cluster. The semantic and visual similarity scores of each synthetic image cluster are combined to obtain the overall similarity score between the real image to each synthetic image cluster, herein also referred to as the overall similarity score of each synthetic image cluster. The substeps of performing local similarity analysis on the real image and each synthetic image cluster by the discriminator (22) will be further explained with reference to Figures 6 (a) and (b).

[0052] Thereon, the discriminator (22) compares the overall similarity score of each synthetic image cluster with a preset similarity score threshold, wherein the preset similarity score threshold is determined by an acceptable range of variance on the synthetic image. The variance on the synthetic images in the synthetic image cluster is considered to be acceptable if the overall similarity score of the synthetic image cluster is less than or passes the preset similarity score threshold. A lower preset similarity score threshold means that less variance is acceptable, while a higher preset similarity score threshold means that more variance is acceptable.

[0053] Referring to the previous example, the real image is the image of a MOSFET with a bent leg as shown in Figure 3 (a), wherein the MOSFET is a three-legged MOSFET. However, the synthetic images in the synthetic image cluster are images of two-legged MOSFET as shown in Figure 3 (b) and the overall similarity score computed for the synthetic image cluster is 0.639. The discriminator (22) shall consider the two-legged MOSFET in the synthetic images to be similar to the three-legged MOSFET with a bent leg in the real image if the overall similarity score computed for the synthetic image cluster is less than the preset similarity score threshold. In order for the discriminator (22) to recognise that the two-legged MOSFET differs from the three-legged MOSFET with a bent leg, the preset similarity score threshold is set at 0.5. Thus, since the overall similarity score of the synthetic image cluster is higher than the preset similarity score threshold, i.e. 0.639>0.5, the overall similarity score of the synthetic image cluster does not pass the preset similarity score threshold and the variance on the synthetic images in the synthetic image cluster is considered not acceptable. On the other hand, in order for the discriminator (22) to recognise the two-legged MOSFET is similar to the three-legged MOSFET with a bent leg, the preset similarity score threshold is set to 0.7. Consequently, the overall similarity score of the synthetic image cluster is lower than the preset similarity score threshold i.e. 0.639<0.7, therefore, the overall similarity score of the synthetic image cluster passes the preset similarity score threshold and the variance on the synthetic images in the synthetic image cluster is considered acceptable.

[0054] The discriminator (22) then determines number and percentage of the synthetic image clusters with the overall similarity score less than the preset similarity score threshold. The discriminator (22) then determines whether the percentage of synthetic image clusters with the overall similarity score less than the preset similarity score threshold is equal to or more than a preset finetune threshold as in decision 650. The preset finetune threshold is determined according to a desired performance of the generative model. For example, the system (1) may require at least 50% of the synthetic image clusters having the overall similarity score less than the preset similarity score threshold as a benchmark to indicate that the generative model is performing as desired. Therefore, if the overall similarity score of more than 50% of the synthetic image clusters is equal to or more than, or in other words does not pass, the preset similarity score threshold, then it indicates that the generative model is underperforming, and vice versa.

[0055] If the percentage of synthetic image clusters with the overall similarity score less than the preset similarity score threshold is less than the preset finetune threshold, the discriminator (22) performs finetuning on the generative model as in step 700. The discriminator (22) normalises the overall similarity score of each synthetic image cluster to compute weightage values for each synthetic image cluster. Subsequently, the discriminator (22) normalises the computed weightage value for each synthetic image cluster. Thereon, based on the normalised weightage values for each synthetic image cluster, the discriminator (22) computes the number of synthetic images to sample from each synthetic image cluster.

[0056] The discriminator (22) adds a weighted combination of a plurality of real images and samples of synthetic images from each synthetic image cluster to an input dataset, wherein the input dataset is used to orient a next generative model training to output synthetic images that are more similar to the new real image. The discriminator (22) retrieves a number of synthetic images from the synthetic image clusters stored in the database (40) based on the weightage value of each synthetic image cluster. For example, there are three synthetic image clusters, namely, Clusterl , Cluster2, and Clusters and their weightage values are twenty, thirty, and fifty respectively. Additionally, the generative model requires a total of one hundred images as a training dataset. Therefore, twenty synthetic images are retrieved from Clusterl , thirty synthetic images are retrieved from Cluster2, fifty synthetic images are retrieved from Clusters. The synthetic images from each synthetic image cluster are retrieved at random. Thereon, the discriminator (22) retrieves a number of real images stored in the database (40) according to a preset ratio of real images to synthetic images. For example, the preset ratio of real images to synthetic images is set to be 50:50. Based on the previous example, if one hundred synthetic images are retrieved from the synthetic image clusters, the discriminator (22) then retrieves one hundred real images from the database (40) accordingly. The sampled synthetic images and real images are added to the input dataset and is sent to the generator (21) for further action. Additionally, the input dataset is sent to the database (40) for storing.

[0057] Once the input dataset is completed, the method repeats from the step 200, which is generating a generative model. However, the new generative model is generated using the input dataset.

[0058] On the other hand, if the percentage of synthetic image clusters with the overall similarity score less than the preset similarity score threshold is equal to or more than the preset finetune threshold as in decision 650, the discriminator (22) filters the synthetic images as in step 800. The discriminator (22) removes the synthetic images in the synthetic image clusters with the overall similarity score equals to or greater than the preset similarity score threshold from the database (40). Thereon, in decision 950, the discriminator (22) determines whether a total amount of synthetic images meet a number of images required for the subsequent deep learning task such as object classification, detection, or other image-based task. The number of images required for deep learning task depends on various factors, including complexity of the task, the diversity of the dataset, the chosen model, and the desired performance level. If the total amount of synthetic images is less than the number of images required for the subsequent deep learning task, the method is repeated from step 300, to generate more synthetic images.

[0059] Otherwise, if the total amount of synthetic images meets or exceeds the required number of images for the subsequent deep learning task, the processing module (20) sends a filtered synthetic image dataset to the output module (30) to be used to train models for classification or detection or other image-based tasks as in step 1000. The filtered synthetic image dataset comprises a combination of filtered synthetic images and the real image as well as annotation for a vision task model training, wherein the filtered synthetic images are the synthetic images in all of the synthetic image clusters with the overall similarity score less than the preset similarity score threshold. Prior to sending the filtered synthetic image dataset to the output module (30), the processing module (20) formats the synthetic images in the filtered synthetic image dataset according to input requirements of the deep learning models for classification or detection or other image-based task. Preferably, the synthetic images are formatted by removing noises from the synthetic images and resizing the synthetic images to match the input requirement of the deep learning models.

[0060] Referring now to Figure 4, which illustrates a flowchart of sub-steps for generating the generative model by the generator (21) of the step 200 of the method of Figure 2. Initially, the generator (21) receives the real image in the current dataset from the image acquisition module (10) and formats the real image in the current dataset according to input requirement of a chosen generative training algorithm as in step 201. Preferably, the real image is formatted by removing noises from the real image and resizing the real image to match the input requirement of the chosen generative training algorithm.

[0061] Thereon, the generator (21) determines whether there are synthetic image clusters and their weightage value stored in the database (40) as in decision 202.

[0062] If there are no synthetic image clusters and their weightage value stored in the database (40), the generator (21) directly initialises a generative architecture from machine learning or deep learning approach as in step 205. The generator (21) initialises the generative architecture by defining a model architecture, initialising model parameters, and configuring a training process. In step 206, the generator (21) trains the generative model with the current dataset consist of the real image. The generator (21) forwards the generative model to the database (40) for storing.

[0063] On the other hand, if there are synthetic image clusters and their weightage value stored in the database (40) as in decision 202, which indicates that a previous generative model is being finetuned, the generator (21) retrieves the input dataset from the discriminator (22) as in step 203, wherein the input dataset comprises the weighted combination of real images and sampled synthetic images.

[0064] In step 204, the generator (21) formats the sampled synthetic images and real images retrieved from the input dataset according to the input requirement of the chosen generative training algorithm and adds the sampled synthetic images and real images retrieved from the input dataset to the current dataset.

[0065] Then, the generator (21) initialises the generative architecture from machine learning or deep learning approach as in step 205 and trains the generative model with the current dataset as in step 206, wherein the current dataset now comprises the real image retrieved from the image acquisition module (10) as well as the sampled synthetic images and real images retrieved from the input dataset.

[0066] Figure 5 illustrate a flowchart of the sub-steps for clustering the synthetic images into the plurality of synthetic image clusters by the generator (21) of the step 400 of the method of Figure 2. Initially, in step 401 , the generator (21) extracts semantic feature vector from each synthetic image. Semantic feature vector is a compact representation of semantic information contained in the synthetic image. The semantic feature vector is retrieved by removing an output layer or a classification layer of the synthetic image and passing the synthetic image through a pretrained neural network model. Outputs from a layer before the output layer or classification layer are retrieved as the semantic feature vector.

[0067] Thereon, as in step 402, the generator (21) retrieves feature embedding vector of the synthetic image from feature vectors of each synthetic image. Feature vectors of the synthetic image are first extracted using the pretrained neural network model before dimensionality of the extracted feature vectors is reduced, whereby the reduced feature vectors are the feature embedding vector. The reduction of the dimensionality of the extracted feature vectors is essential to speed up the process of clustering while preserving its essential semantic information. The slow process of clustering typically happens because the extracted feature vectors have large number of dimensions. In step 403, the generator (21) fits a clustering algorithm to feature space to obtain cluster centres for a number of clusters. The clustering algorithm is applied to the feature vectors. Thereon, the clustering algorithm identifies groups of similar data points or known as clusters in the feature space. The clustering algorithm outputs the cluster centres and cluster labels, wherein the cluster centres are mean feature vectors of each cluster, and the cluster labels assign each data point to a cluster.

[0068] Thereafter, in step 404, the generator (21) obtains distance vector of each synthetic image by using distance measures to compute distance between feature embeddings of the synthetic image and each cluster centre, wherein the feature embeddings represent data points in the feature space. Examples of the distance measures include but are not limited to Euclidean

[0069] Distance, Cosine Similarity, Mahalanobis Distance, etc.

[0070] In step 405, the generator (21) assigns each synthetic image to the cluster according to the distance vector. In the preferred embodiment, the generator (21) assigns the synthetic image to the cluster with minimum distance between its feature embeddings and the cluster centre.

[0071] Finally, in step 406, the generator (21) saves the cluster centres, distance vector of all synthetic images, and synthetic image cluster labels in the database (40).

[0072] Figures 6 (a) and (b) illustrates flowcharts of sub-steps of performing similarity analysis on the real image and each synthetic image cluster by the discriminator (22) of the step 500 of the method of Figure 2. Generally, the discriminator (22) computes semantic similarity score and visual similarity score between the real image and the synthetic image clusters and combines the semantic and visual similarity scores to obtain the overall similarity score between the real image to each synthetic image cluster.

[0073] To compute the semantic similarity score between the real image and the synthetic image clusters, the discriminator (22) prepares the real image for comparison to the synthetic images by retrieving the real image from the database (40) as in step 501.

[0074] In step 511 , the discriminator (22) extracts the semantic feature vector from the real image by using the pretrained neural network model. The semantic feature vector is retrieved by removing the output or classification layer from the real image and sending it through the pretrained neural network model. The semantic feature vector is retrieved from the layer before the classification layer or output layer. Thereon, in step 512, the discriminator (22) retrieves feature embedding vector of the real image by reducing the dimensionality of the extracted feature vectors of the real image by using the pretrained neural network model.

[0075] In step 513, the discriminator (22) loads the cluster centres of the synthetic image clusters from the database (40). The discriminator (22) then initialises clustering algorithm as in step 514. The clustering algorithm is initialised by specifying the number of clusters and assigning the loaded cluster centres from the database (40) as clustering algorithm centroids.

[0076] Once the clustering algorithm is initialised, the discriminator (22) obtains distance vector between the real image and the cluster centre of each synthetic image cluster as in step 515.

[0077] The discriminator (22) utilises distance measures to compute distance between feature embeddings of the real image and the cluster centre of each synthetic image cluster. The distance vector between the real image and the cluster centre of each synthetic image cluster is assigned as the semantic similarity score for the corresponding synthetic image cluster.

[0078] Thereon, in order to compute visual similarity between the real image and the synthetic images in the synthetic image clusters, the discriminator (22) retrieves a number of samples of synthetic images from each synthetic image cluster based on the distance between the synthetic image to its corresponding cluster centre as in step 521. The number of samples of synthetic images retrieved depends on the capability of the discriminator (22). For each synthetic image cluster, the discriminator (22) retrieves N number of samples closest to its corresponding cluster centre.

[0079] In the step 522 the discriminator (22) extracts query keypoints and query descriptors from each synthetic image sampled from each synthetic image cluster, while in step 523, the discriminator (22) extract sample keypoints and sample descriptors from the real image. The query keypoints, query descriptors, sample keypoints, and sample descriptors are extracted using a local feature descriptor algorithm.

[0080] Then, in step 524, the discriminator (22) uses distance measures to determine top two matches between the query descriptors of each synthetic image sampled from each synthetic image cluster and each sample descriptor of the real image. For each match, the discriminator (22) computes distance value between the query descriptors of each synthetic image sampled from each synthetic image cluster and each sample descriptor of the real image. The top two matches are determined by the shortest distance value between the sample descriptor of the real image and query descriptors among all synthetic images sampled from the synthetic image cluster, wherein the query descriptor with the shortest distance value to the sample descriptor is the best match, ki while the query descriptor with the second shortest distance value to the sample descriptor is the second-best match, k2.

[0081] In step 525, the discriminator (22) filters to find unique matches between the real image and each synthetic image from each synthetic image cluster by comparing the top two matches, ki and k2. Matches is considered unique when ki is closer to the sample descriptor than k2 by a predetermined threshold. The predetermined threshold is set according to a required parameter for the query descriptors of the synthetic image to be considered as a match with the sample descriptor of the real image. The required parameter is an arbitrary parameter set by the user according to required strictness of the system (1) in terms of how similar the query descriptors have to be to be considered correct match. In order to determine whether the match is a unique match, the discriminator (22) compares ratio between the distance value of ki with the distance value of k2 with the predetermined threshold. The match is considered as a unique match if the ratio between the distance value of ki with the distance value of k2 equals to or less than the predetermined threshold. In other words, the greater the difference between the distance value of ki and the distance value of k2, the more unique the match is. For example, the distance value of ki is 0.5 while the distance value of k2 is 0.8. If the predetermined threshold is set as 0.5, the match is not considered as a unique match since the ratio between the distance value of ki with the distance value of k2 is 0.625 and is greater than the predetermined threshold i.e. 0.625>0.5. On the other hand, if the predetermined threshold is set as 0.7, the match is considered as a unique match since the ratio between the distance value of ki with the distance value of k2 is less than the predetermined threshold i.e. 0.625<0.7.

[0082] In step 526, the discriminator (22) computes uniqueness percentage and coverage percentage for each synthetic image sampled from each synthetic image cluster. The uniqueness percentage of the synthetic image is computed by dividing the number of unique matches of the synthetic image with the number of total matches found between each query descriptor of the synthetic image and the sample descriptors of the real image. Meanwhile, the coverage percentage of the synthetic image is computed by using the following equation: wherein Nunique represents number of unique matches of the synthetic image, Nquery_kPrepresents number of query keypoints of the synthetic image, and NSampie_kPrepresents number of sample keypoints of the real image. Thereafter, the discriminator (22) computes mean distance of the unique matches for each synthetic image sampled from each synthetic image cluster as in step 527. The total distance value of all unique matches of the synthetic image are averaged across the total number of unique matches of the synthetic image.

[0083] In step 528, the discriminator (22) tunes similarity distance value for each synthetic image sampled from each synthetic image cluster by using uniqueness percentage and coverage percentage. The similarity distance value is tuned according to the following equation:

[0084] 5 = (1 - %unlque) (l - % cover) (c meon , wherein S represents similarity distance value, %Unique represents uniqueness percentage of the synthetic image, %Cover represents coverage percentage of the synthetic image, and dmean represents mean distance of the unique matches of the synthetic image.

[0085] In step 529, the discriminator (22) computes mean distance for each synthetic image cluster by averaging the similarity distance value of all the synthetic images sampled from the synthetic image clusters across the number of samples of synthetic images for the synthetic image clusters.

[0086] In addition, in step 530, the discriminator (22) penalises the mean distance for each synthetic image cluster by ratio of samples with zero unique matches. The mean distance ratio is penalised according to the following equation: wherein Smean represents mean distance of the synthetic image cluster and %SamPies_zero_unique represents percentage of synthetic image sample with zero unique matches within the synthetic image cluster. The penalised mean distance between the real image and the synthetic image clusters is assigned as the visual similarity score.

[0087] Once the semantic similarity score and the visual similarity score for all synthetic image clusters are computed, the discriminator (22) combines the semantic similarity score and the visual similarity score to obtain the overall similarity score between the real image to each synthetic image cluster. The discriminator (22) normalises the semantic similarity score and visual similarity score of each synthetic image cluster as in step 541. The semantic similarity score and the visual similarity score are normalised to values ranging from 0 to 1. Finally, in step 542, the discriminator (22) computes weighted average of the semantic and visual similarity score for each synthetic image cluster by computing the average of normalised values of the semantic similarity score and the visual similarity score. If the visual similarity score is invalid, the discriminator (22) only uses semantic similarity score to compute the weighted average of semantic and visual similarity score, wherein the visual similarity score of the synthetic image cluster is invalid when all synthetic image sampled from the synthetic image cluster have zero unique matches. The weighted average of semantic and visual similarity score for each synthetic image cluster is assigned as the overall similarity score for the corresponding synthetic image cluster and is forwarded to the database (40) for storing. The invention being thus described, it will be readily appreciated by those skilled in the art that modifications may be made to the invention without departing from the embodiments as disclosed therein. Such modifications are to be considered as included in the following claims unless the claims by their language expressly state otherwise.

Claims

Claims1 . A system (1) for generating synthetic images comprising; a. an image acquisition module (10) configured to receive video or sequence of images containing object to be used to train a deep learning model; b. a processing module (20) connected to the image acquisition module (10), wherein the processing module (20) is configured to generate a generative model to produce a plurality of synthetic images and filter the synthetic images; and c. an output module (30) connected to the processing module (20), wherein the output module (30) is configured to send filtered synthetic images to train the deep learning model to perform object classification, detection, or other image-based task, characterised in that the processing module (20) is further configured to: a. generate a plurality of synthetic image clusters according to distance between feature embedding vectors of each synthetic image and cluster centre of each synthetic image cluster; and b. perform similarity analysis on a real image and each synthetic image cluster by computing semantic and visual similarity between the real image and the synthetic image clusters, wherein the real image referred to an image containing object to be used to train the deep learning model received from the image acquisition module (10).

2. The system (1) as claimed in Claim 1 , wherein the processing module (20) is further configured to finetune subsequent generative model training by using a weighted combination of a plurality of real images and a plurality of synthetic images sampled from each synthetic image cluster.

3. The system (1) as claimed in Claim 1 , wherein the processing module (20) further comprising: a. a generator (21) configured to generate the generative model to produce the plurality of synthetic images and generate a plurality of synthetic image clusters by classifying each synthetic image according to distance between the feature embedding vectors of each synthetic image and cluster centre of each synthetic image cluster; andb. a discriminator (22) configured to perform the similarity analysis on the real image and each synthetic image cluster using semantic features and visual distance, filter the synthetic images based on the semantic-visual similarity, and automatically finetune subsequent generative model by using a weighted combination of a plurality of real images and a plurality of synthetic images sampled from each synthetic image cluster.

4. The system (1) as claimed in Claim 1 , wherein the image acquisition module (10) is further configured to assign a label to the real image.

5. A method for generating synthetic images is characterised by the steps of: a. acquiring a real image by an image acquisition module (10), wherein the real image refers to an image containing object to be used to train a deep learning model; b. generating a generative model by a generator (21); c. generating a plurality of synthetic images by the generative model; d. clustering the plurality of synthetic images into a plurality of synthetic image clusters according to distance between feature embedding vectors of each synthetic image and cluster centre of each synthetic image cluster by the generator (21); e. performing a semantic and visual similarity analysis on the real image and each synthetic image cluster by a discriminator (22); f. comparing an overall similarity score of each synthetic image cluster with a preset similarity score threshold by the discriminator (22), wherein the preset similarity score threshold is determined by an acceptable range of variance on the synthetic image; g. determining whether percentage of synthetic image clusters with the overall similarity score less than the preset similarity score threshold is equal to or more than a preset finetune threshold by the discriminator (22), wherein the preset finetune threshold is determined according to a desired performance of the generative model; h. filtering the plurality of synthetic images by removing synthetic images in the synthetic image cluster with the overall similarity score equals to or greater than the preset similarity score threshold by the discriminator (22) if the percentage of synthetic image clusters with the overall similarity score that is less than the preset similarity score threshold is equal to or more than the preset finetune threshold; andi. performing finetuning on the generative model by the discriminator (22) if the percentage of synthetic image clusters with the overall similarity score that is less than the preset similarity score threshold is less than the preset finetune threshold.

6. The method as claimed in Claim 5, wherein generating the generative model by the generator (21) includes the sub-steps of: a. formatting the real image in a current dataset according to input requirements of a chosen generative training algorithm; b. determining whether there are synthetic image clusters and their weightage value stored in a database (40); c. initialising a generative architecture from machine learning or deep learning approach if there are no synthetic image clusters and their weightage value stored in the database (40); and d. training the generative model with the current dataset.

7. The method as claimed in claim 6, wherein if there are synthetic image clusters and their weightage value stored in the database (40), the sub-steps include: a. retrieving an input dataset from the discriminator (22), wherein the input dataset comprises a weighted combination of real images and samples of synthetic images from each synthetic image cluster; b. formatting the samples of synthetic images from each synthetic image cluster and real images retrieved from the input dataset according to the input requirements of the chosen generative training algorithm; c. adding the samples of synthetic images from each synthetic image cluster and real images retrieved from the input dataset to the current dataset; d. initialising the generative architecture from machine learning or deep learning approach; and e. training the generative model with the current dataset.

8. The method as claimed in Claim 5, wherein clustering the plurality of the synthetic images into a plurality of synthetic image clusters according to distance between feature embedding vectors of each synthetic image and the cluster centre of each synthetic image cluster by the generator (21) further comprising the sub-steps of: a. extracting semantic feature vector from each synthetic image by removing an output layer from the synthetic image using a pretrained neural network model;b. retrieving feature embedding of each synthetic image by reducing dimensionality of feature vectors of the synthetic image; c. fitting a clustering algorithm to feature space to obtain cluster centres for a number of clusters; d. obtaining distance vector of each synthetic image by using distance measure to compute distance between feature embeddings of each synthetic image and each cluster centre; and e. assigning each synthetic image to a cluster according to the distance vector.

9. The method as claimed in Claim 5, wherein performing the semantic and visual similarity analysis on the real image and each synthetic image cluster by the discriminator (22) further comprising the sub-steps of: a. computing semantic similarity between the real image and each synthetic image of each synthetic image clusters; b. computing visual similarity between the real image with each synthetic image of each synthetic image cluster; and c. obtaining the overall similarity score between the real image to each synthetic image cluster.

10. The method as claimed in Claim 9, wherein computing semantic similarity between the real image and each synthetic image of each synthetic image cluster further comprising the sub-steps of: a. extracting semantic feature vector from the real image by removing an output layer from the real image using a pretrained neural network model; b. retrieving feature embedding of the real image by reducing dimensionality of feature vectors of the real image; c. loading the cluster centre of each synthetic image cluster; d. initialising clustering algorithm; and e. obtaining distance vector between the real image and the cluster centre of each synthetic image cluster.

11. The method as claimed in Claim 9, wherein computing visual similarity between the real image with each synthetic image of each synthetic image cluster further comprising the sub-steps of:a. retrieving a number of samples of synthetic images from each synthetic image cluster based on the distance between the synthetic image to its corresponding cluster centre; b. extracting query keypoints and query descriptors from each synthetic image sampled from each synthetic image cluster; c. extracting sample keypoints and sample descriptors from the real image; d. determining top two matches between the query descriptors of each synthetic image sampled from each synthetic image cluster and each sample descriptor of the real image by using distance measures, wherein the top two matches for each sample descriptor of the real image are two query descriptors among all synthetic images sampled from the synthetic image cluster with the shortest distance value; e. filtering for unique matches between the real image and each synthetic image from each synthetic image cluster by comparing the top two matches between the query descriptors of the synthetic image and each sample descriptor of the real image; f. computing uniqueness percentage for each synthetic image sampled from each synthetic image cluster; g. computing coverage percentage for each synthetic image sampled from each synthetic image cluster; h. computing mean distance of the unique matches for each synthetic image sampled from each synthetic image cluster; i. tuning similarity distance value for each synthetic image sampled from each synthetic image cluster; j. computing mean distance for each synthetic image cluster; and k. penalising the mean distance for each synthetic image cluster by ratio of samples with zero unique matches.

12. The method as claimed in Claim 11 , wherein the number of synthetic images retrieved from each synthetic image cluster depends on the capability of the discriminator (22), and wherein N number of samples closest to its corresponding cluster centre are retrieved for each synthetic image cluster.

13. The method as claimed in Claim 11 , wherein filtering for unique matches between the real image and each synthetic image from each synthetic image cluster by comparing the top two matches between the query descriptors of the synthetic image and each sample descriptor of the real image includes the sub-steps of:a. computing ratio between the distance value of best match, ki with the distance value of second-best match, k2, wherein ki refers to the query descriptor with the shortest distance value to the sample descriptor, whereas k2 refers to the query descriptor with the second shortest distance value to the sample; and b. comparing the ratio between distance value of ki with the distance value of k2 with a predetermined threshold, wherein the predetermined threshold is set according to a required parameter for query descriptors of the synthetic image to be considered as a correct match with the sample descriptor of the real image, and wherein a match is considered as a unique match if the ratio between the distance value of k1 with the distance value of k2 equals to or less than the predetermined threshold.

14. The method as claimed in Claim 11 , wherein computing uniqueness percentage for each synthetic image sampled from each synthetic image cluster is by dividing number of unique matches of the synthetic image with number of total matches found between each query descriptor of the synthetic image and the sample descriptors of the real image.

15. The method as claimed in Claim 11 , wherein computing coverage percentage for each synthetic image sampled from each synthetic image cluster is based on the following equation:wherein Nunique represents number of unique matches of the synthetic image, Nquery_kPrepresents number of query keypoints of the synthetic image, and N sample_kp represents number of sample keypoints of the real image.

16. The method as claimed in Claim 11 , wherein computing mean distance of the unique matches for each synthetic image sampled from each synthetic image cluster is by averaging total distance value of all unique matches of the synthetic image across total number of unique matches of the synthetic image.

117. The method as claimed in Claim 11 , wherein tuning similarity distance value for each synthetic image sampled from each synthetic image cluster is according to the following equation:S = (1 - %unlque) (l - % cover) (tf mecm. wherein S represents similarity distance value, %Unique represents uniqueness percentage of the synthetic image, %Cover represent coverage percentage of the synthetic image, and dmean represent mean distance of the unique matches of the synthetic image.

18. The method as claimed in Claim 11 , wherein computing mean distance for each synthetic image cluster is by averaging the similarity distance value of all of the synthetic images sampled from the synthetic image cluster across the number of samples of synthetic images for the synthetic image cluster.

19. The method as claimed in Claim 11 , wherein penalising the mean distance for each synthetic image cluster by ratio of samples with zero unique matches is according to the following equation:_ ‘-'mean _ p1enalised mean distance = -1 _ Q / 1 / 0s amp les_zero_unique wherein Smean represents mean distance of the synthetic image cluster and %samples_zero_unique represents percentage of synthetic image sample with zero unique matches within the synthetic image cluster.

20. The method as claimed in Claim 9, wherein obtaining the overall similarity score between the real image to each synthetic image cluster further comprising the sub-steps of; a. normalising semantic similarity score; b. normalising visual similarity score; and c. computing weighted average of semantic and visual similarity score.

21. The method as claimed in Claim 5, wherein the variance on the synthetic images in the synthetic image cluster is considered to be acceptable if the overall similarity score of the synthetic image cluster is less than the preset similarity score threshold.

22. The method as claimed in Claim 21 , wherein a lower preset similarity score threshold means that less variance is acceptable, whereas a higher preset similar score threshold means that more variance is acceptable.

23. The method as claimed in Claim 5, wherein performing finetuning on the generative model by the discriminator (22) if the percentage of synthetic image clusters with the overall similarity score that is less than the preset similarity score threshold is less than the preset finetune threshold further comprising the sub-steps of: a. computing weightage values for each synthetic image cluster from the overall similarity score of each synthetic image cluster; b. normalising the computed weightage value for each synthetic image cluster; c. computing the number of synthetic images to sample from each synthetic image cluster based on the weightage values for each synthetic image cluster; d. adding a weighted combination of the real image and samples of synthetic images from each synthetic image cluster to an input dataset; and e. repeating from the step of generating a generative model by the generator (21), wherein a new generative model is generated using the input dataset.

Images