Multimedia identification model incremental learning method and system

Abstract

The invention provides a multimedia identification model incremental learning method and a multimedia identification model incremental learning system, which are characterized in that a two-stage grouping method is used for merging semi-automatically collected multimedia samples and original data sets, groups and subgroups are uniformly sampled, and dynamic boundaries are designed to optimize a multimedia identification model.

Description

Technical Field

[0001] The present invention relates to the training of multimedia recognition models, and more particularly to an incremental learning method and system for multimedia recognition models. Background Art

[0002] The multimedia recognition model trained by artificial intelligence technology can be used to recognize multimedia data such as sound, video or picture. For example, it can be used to recognize the breed of animals such as cats or dogs, the identity of passers-by, or the brand and model of vehicles appearing in the aforementioned multimedia data.

[0003] Although multimedia recognition models perform well when trained on manually annotated databases, real-world data is long-tailed, and semi-automatic data collection will encounter the problem of uneven data distribution. Therefore, during the training process, clusters with fewer samples often do not perform well.

[0004] In order to alleviate the problem of uneven data distribution, a common method is to use over-sampling or under-sampling methods to rebalance the groups, but this will lead to over-fitting problems and still cannot solve the problem of poor training for groups with a small number of samples. Summary of the invention

[0005] To solve the above problems in the prior art, the present invention aims to combine uniform sampling and recognition training to optimize the distribution of long-tail data in the feature space, so as to improve the problem of poor training of groups with fewer samples and avoid over-fitting at the same time.

[0006] To solve the above problems, the present invention provides an incremental learning method for a multimedia recognition model, comprising: clustering according to multiple features of multiple input multimedia and multiple features of multimedia in an original data set to generate cluster samples; calculating subgroups of each group for multiple groups in the cluster samples to uniformly sample each subgroup to generate the uniformly sampled samples; and performing incremental learning of the multimedia recognition model based on the uniformly sampled samples, wherein the loss function used in the incremental learning includes dynamic boundary values ​​between the multiple groups, and the dynamic boundary values ​​are determined based on the number of samples of each group and the number of samples of each subgroup.

[0007] The present invention also provides a multimedia recognition model incremental learning system, comprising: a clustering device, used to cluster according to multiple features of multiple input multimedia and multiple features of multimedia in an original data set to generate cluster samples; a uniform sampling device, used to calculate subgroups of each group for multiple groups in the cluster samples to uniformly sample each subgroup to generate the uniformly sampled samples; and a multimedia recognition training device, used to perform incremental learning of the multimedia recognition model based on the uniformly sampled samples, wherein the loss function used in the incremental learning includes dynamic boundary values ​​between the multiple groups, and the dynamic boundary values ​​are determined based on the number of samples of each group and the number of samples of each subgroup.

[0008] Based on the above, the present invention uses semi-supervised multimedia clustering and uniform sampling to perform incremental learning on the multimedia recognition model. Among them, a pre-trained feature extraction model is used to extract the features of the input multimedia, and then two-stage clustering is used to merge the repeated groups, and virtual labels are set for the cluster samples. Different sampling rates and uniform sampling sample parameters are set according to the number of samples in the cluster samples to calculate the training loss, providing a better incremental learning multimedia recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 4 is a circuit block diagram of a multimedia recognition model incremental learning system according to an embodiment of the present invention.

[0010] Figure 2 FIG. 4 is a circuit block diagram of a clustering device according to an embodiment of the present invention.

[0011] Figure 3 FIG. 4 is a circuit block diagram of a uniform sampling device according to an embodiment of the present invention.

[0012] Figure 4 FIG. 4 is a circuit block diagram of a multimedia recognition training device according to an embodiment of the present invention.

[0013] Figures 5 to 9 4 is a flowchart of a multimedia recognition model incremental learning method according to an embodiment of the present invention.

[0014] Fig.10 Schematic diagram of the training of a multimedia recognition model with fixed boundaries.

[0015] Fig.11 FIG. 4 is a schematic diagram of training a multimedia recognition model for dynamic boundaries according to an embodiment of the present invention.

[0016] Fig.12 4 is a circuit block diagram of a multimedia recognition model incremental learning system according to an embodiment of the present invention.

[0017] Main component symbols

[0018] 10 Multimedia Recognition Model Incremental Learning System

[0019] 100,101,102 Multimedia input device

[0020] 110,111,112 Feature extraction device

[0021] 120 Clustering Device

[0022] 130 Uniform sampling device

[0023] 140 Multimedia Identification Training Device

[0024] 1210 High Similarity Sample Exclusion Device

[0025] 1220 First stage clustering device

[0026] 1230 Clustering result merging device

[0027] 1240 Second stage clustering device

[0028] 1250 Virtual label generating device

[0029] 1310 First stage uniform sampling device

[0030] 1320 Subgroup Computing Device

[0031] 1330 Second stage uniform sampling device

[0032] 1340 Uniform sampling and parameter output device 1410 Original Model Reading Device

[0033] 1420 Forward Computing Device

[0034] 1430 Loss Calculation Device

[0035] 1440 Anti-transmission computing device

[0036] 1450 Model weight updating device

[0037] Group 21,22

[0038] 210,220 Group Heart

[0039] 211~212,221~223 subgroups

[0040] 219,229 samples

[0041] 30 Multimedia Recognition Model Incremental Learning System 301 CPU

[0042] 302 Random Access Memory

[0043] 303,304 Graphics Processor

[0044] 305 Bus

[0045] 306 Sound Interface

[0046] 307 Speakers

[0047] 308 Network Interface

[0048] 309 Transceiver

[0049] 310 Network Camera

[0050] 311 Input and Output Interface

[0051] 312 External Camera

[0052] 313 Storage Device

[0053] S20 Incremental Learning Method for Multimedia Recognition Model

[0054] S500~S540,S600~S640,S700~S730,S800~S830,S900~S940 Steps

[0055] IM Input Multimedia

[0056] Multimedia features of DF original dataset

[0057] IF Input multimedia features

[0058] CS clustering samples

[0059] PL Virtual Label

[0060] BS Uniformly sampled samples

[0061] BP sample parameters

[0062] Multimedia Identification Model Based on IP Incremental Learning

[0063] SF excludes samples with high similarity

[0064] FC first stage clustering results

[0065] Clustering results of MC merging

[0066] Parameters of uniform sampling in the first stage of CD

[0067] CE subgroup results

[0068] Parameters of uniform sampling in the second stage of CF

[0069] RM Original multimedia recognition model

[0070] FR forward results

[0071] TL training loss

[0072] GR gradient results

[0073] m s Fixed Boundary

[0074] m d Dynamic boundaries. DETAILED DESCRIPTION

[0075] Please refer to Figure 1 , which is a circuit block diagram of a multimedia recognition model incremental learning system 10 according to an embodiment of the present invention. In this embodiment, the multimedia recognition model incremental learning system 10 includes a plurality of multimedia input devices (such as the illustrated multimedia input devices 100-102), a plurality of feature extraction devices (such as the illustrated feature extraction devices 110-112), a clustering device 120, a uniform sampling device 130, and a multimedia recognition training device 140, wherein each feature extraction device 110-112 is electrically coupled to a corresponding multimedia input device 100-102, the clustering device 120 is electrically coupled to each feature extraction device 110-112, the uniform sampling device 130 is electrically coupled to the clustering device 120, and the multimedia recognition training device 140 is electrically coupled to the uniform sampling device 130. In this embodiment, Figure 1 Each device shown is a physical circuit or electronic device with data processing capabilities.

[0076] In this embodiment, each multimedia input device 100-102 is used to receive input multimedia IM generated or provided by one or more multimedia sources through wired or wireless means. Each input multimedia IM can be an electronic signal or electronic data such as sound, film or picture. The multimedia source can be a camera, a microphone or a similar electronic device for generating input multimedia IM. The multimedia source can also be an online video downloader, a data storage device, a database or a similar electronic device for providing input multimedia IM. The input multimedia of the same group is not limited to appearing in a certain multimedia source, and a certain group of input multimedia can appear repeatedly in different multimedia sources. The multimedia sources can be the same or different databases, such as: ImageNet, MS COCO, Google Open Image Dataset or Mozilla Common Voice Dataset and other databases. The multimedia sources can be the same or different hardware, such as a mobile phone with a camera function, a camera or an IP Camera. The multimedia sources can be located in the same or different places, such as a building's access control camera, a camera on the top of an elevator, or a camera in an automated teller machine (ATM).

[0077] Next, the feature extraction devices 110-112 receive the above-mentioned input multimedia IM from the multimedia input devices 100-102, and extract the feature IF of each input multimedia IM. Next, the clustering device 120 converts the feature IF of the input multimedia and the multimedia features DF of the original data set into cluster samples CS and virtual labels PL. Next, the uniform sampling device 130 converts the cluster samples CS and virtual labels PL into uniformly sampled samples BS and sample parameters BP. Next, the multimedia recognition training device 140 converts the uniformly sampled samples BS and sample parameters BP into an incrementally learned multimedia recognition model IP.

[0078] The aforementioned feature extraction devices 110-112 extract the features IF of the input multimedia IM, wherein the feature extraction model is a model pre-trained using the original data set, and the feature extraction method can use the Support Vector Machine (SVM), Gaussian Mixture Model (GMM), Hidden Markov Model (HMM) in machine learning, or the Convolutional Neural Network (CNN), Autoencoder, Vision Transformer (ViT), Graph Convolutional Network (GCN) in deep learning, or their equivalent methods, but the present invention is not limited thereto.

[0079] The original data set may include multiple multimedia and their features DF, or only include features DF extracted from multiple multimedia without including the multimedia itself. The original data set may be collected and accumulated by the administrator or user of the multimedia recognition model incremental learning system 10, or provided by others.

[0080] In one embodiment, it is assumed that there are I multimedia input devices and I corresponding feature extraction devices, where I is a positive integer. The first multimedia input device 100 has M0 input multimedia IMs, the second multimedia input device 101 has M1 input multimedia IMs, and so on. The I multimedia input device 102 has M I-1 In this embodiment, only one feature can be extracted for each input multimedia IM. Therefore, the first feature extraction device 110 obtains the feature IF of M0 input multimedia, the second feature extraction device 111 obtains the feature IF of M1 input multimedia, and so on. The first feature extraction device 112 obtains the feature IF of M1 input multimedia. I-1 The feature IF of the input multimedia, wherein the feature extraction devices 110-112 are the same devices, and use the model pre-trained with the original data set to perform feature extraction.

[0081] In another embodiment, at least one feature may be extracted from each input multimedia IM. For example, in an application for recognizing human faces, a feature may be extracted for each human face in each input multimedia IM (e.g., a photo or a video). Therefore, the number of features extracted from each input multimedia IM is equal to the number of human faces in the input multimedia IM.

[0082] Please refer to Figure 2, which is a circuit block diagram of a clustering device 120 according to an embodiment of the present invention. The clustering device 120 includes a high-similarity sample exclusion device 1210, a first-stage clustering device 1220, a clustering result merging device 1230, a second-stage clustering device 1240, and a virtual label generating device 1250 electrically coupled in series. In this embodiment, Figure 2 Each device shown is a physical circuit or electronic device with data processing capabilities.

[0083] The high similarity sample exclusion device 1210 receives the aforementioned input multimedia feature IF, and filters the high similarity samples in the input multimedia feature IF to obtain the sample SF with high similarity excluded. The first-stage clustering device 1220 then performs the first-stage clustering process on the sample SF with high similarity excluded to generate the first-stage clustering result FC. Then, the clustering result merging device 1230 merges the multimedia features DF of the original data set and the first-stage clustering result FC into a merged clustering result MC. Then, the second-stage clustering device 1240 performs the second-stage clustering process on the merged clustering result MC to exclude duplicate groups and obtain the cluster sample CS. Then, the virtual label generating device 1250 generates a virtual label (Psudo Label) PL according to the cluster sample CS.

[0084] The aforementioned samples are the feature IFs of the input multimedia, and samples with high similarity usually come from the same multimedia source. Therefore, when filtering the high similarity samples in the feature IFs of the input multimedia, the aforementioned high similarity sample exclusion device 1210 can calculate the distance between every two feature IFs for the feature IFs of the input multimedia from the same multimedia source. If the distance is less than a threshold, indicating that the two feature IFs have high similarity, one of the feature IFs is excluded and the other feature IF is retained. After the feature IFs of the input multimedia from each multimedia source are excluded as described above, a sample SF with high similarity excluded can be obtained. When calculating the distance of the feature IFs of the input multimedia, distance calculation methods such as Euclidean distance, cosine similarity, or discrepancy alignment metric (DAM) or their equivalent methods can be used, and the present invention is not limited thereto.

[0085] The first stage clustering device 1220 then performs clustering processing on the samples SF that exclude high similarity. Assuming that the number of features IF of all input multimedia is M0+M1+...+M I-1, after excluding samples with high similarity, they can be divided into N clusters. The clustering method can use "Density-Based Spatial Clustering of Applications with Noise" (DBSCAN) or a graph convolutional network (GCN), such as GCN-V and GCN-E, or their equivalent methods, but the present invention is not limited thereto.

[0086] The aforementioned virtual label generating device 1250 generates a virtual label PL according to the cluster sample CS, wherein the cluster sample CS includes each group after the aforementioned second stage clustering processing, and the samples in each group. Each sample in the cluster sample CS has a corresponding virtual label PL, and the virtual label PL is equivalent to the identification code of the group to which the sample belongs. Therefore, samples in the same group have the same virtual label PL, while samples in different groups have different virtual labels PL.

[0087] Please refer to Figure 3 , which is a circuit block diagram of a uniform sampling device 130 according to an embodiment of the present invention. The uniform sampling device 130 includes a first-stage uniform sampling device 1310, a subgroup calculation device 1320, a second-stage uniform sampling device 1330, and a uniform sampling sample and parameter output device 1340, wherein the second-stage uniform sampling device 1330 is electrically coupled to the subgroup calculation device 1320, and the uniform sampling sample and parameter output device 1340 is electrically coupled to the first-stage uniform sampling device 1310 and the second-stage uniform sampling device 1330. In this embodiment, Figure 3 Each device shown is a physical circuit or electronic device with data processing capabilities.

[0088] The uniform sampling device 130 receives the aforementioned cluster samples CS and virtual labels PL. The virtual labels PL are used to indicate the groups to which each sample belongs in the uniform sampling below. The first-stage uniform sampling device 1310 uniformly samples each group in the cluster samples CS to obtain the first-stage uniform sampling parameters CD. The subgroup calculation device 1320 clusters the samples in each group of the cluster samples CS to obtain the subgroup results CE. The second-stage uniform sampling device 1330 uniformly samples each subgroup in the subgroup results CE to obtain the second-stage uniform sampling parameters CF. The uniformly sampled sample and parameter output device 1340 uses the first-stage uniform sampling parameters CD and the second-stage uniform sampling parameters CF to convert the cluster samples CS into uniformly sampled samples BS and sample parameters BP.

[0089] The aforementioned first-stage uniform sampling device 1310 performs uniform sampling on each group in the cluster sample CS, which means that the probability of each group being sampled is the same. Therefore, no matter how many samples there are in each group, the probability of being sampled is the same. The uniform sampling method can use over-sampling or under-sampling, etc., or their equivalent methods, and the present invention is not limited to this. The parameters CD of the first-stage uniform sampling include the results of the first-stage uniform sampling of each group (for example, which groups are sampled), and the number of samples of each group. The aforementioned number of samples of each group refers to the number of samples of each group in the cluster sample CS. The first-stage uniform sampling will not change the aforementioned number of samples of each group.

[0090] When the aforementioned subgroup calculation device 1320 clusters the samples in each group of the clustered samples CS, the clustering method may use "Density-Dependent Spatial Clustering with Noise Applications" (DBSCAN) or a graph convolutional network (GCN), such as GCN-V and GCN-E, or their equivalent methods, but the present invention is not limited thereto.

[0091] The aforementioned second-stage uniform sampling device 1330 performs uniform sampling on each subgroup in the subgroup result CE, which means that the probability of each subgroup being sampled is the same. Therefore, no matter how many samples there are in each subgroup, the number of samples extracted from each subgroup is similar or even the same. The uniform sampling method can use oversampling or undersampling, or their equivalent methods, and the present invention is not limited to this. The parameters CF of the second-stage uniform sampling include the results of the second-stage uniform sampling of each subgroup (for example, which samples are extracted from each subgroup), and the number of samples of each subgroup. The aforementioned number of samples of each subgroup refers to the number of samples of each subgroup in the subgroup result CE. The second-stage uniform sampling is to extract a part of the samples from the sampled subgroups without changing the aforementioned number of samples of each subgroup.

[0092] The uniformly sampled samples and parameter output device 1340 generates uniformly sampled samples BS and sample parameters BP, wherein the uniformly sampled samples BS include samples extracted from each subgroup in the second stage of uniform sampling, and the sample parameters BP include the number of samples of each group (the number of samples of each group in the clustered sample CS) and the number of samples of each subgroup (the number of samples of each subgroup in the subgroup result CE).

[0093] Please refer to Figure 4 , which is a circuit block diagram of a multimedia recognition training device 140 according to an embodiment of the present invention. The multimedia recognition training device 140 includes an original model reading device 1410, a forward propagation calculation device 1420, a loss calculation device 1430, a reverse propagation calculation device 1440, and a model weight updating device 1450 electrically coupled in series. In this embodiment, Figure 4Each device shown is a physical circuit or electronic device with data processing capabilities.

[0094] As described above, the multimedia recognition training device 140 converts the uniformly sampled samples BS and sample parameters BP into an incrementally learned multimedia recognition model IP. The original model reading device 1410 is used to read and provide the original multimedia recognition model RM. The original multimedia recognition model RM is the multimedia recognition model previously trained using the original data set. The forward propagation calculation device 1420 uses the original multimedia recognition model RM to calculate the forward propagation (ForwardPropagation) result FR of the uniformly sampled sample BS. The loss calculation device 1430 uses the sample parameters BP to calculate the training loss (Training Loss) TL of the forward propagation result FR. The backward propagation calculation device 1440 performs backward propagation (BackwardPropagation) on the training loss TL to obtain the gradient (Gradient) result GR. The model weight updating device 1450 uses the gradient result GR to update the weight of the original multimedia recognition model RM to obtain the incrementally learned multimedia recognition model IP. The weight update of the original multimedia recognition model RM is the incremental learning of the original multimedia recognition model RM. The original multimedia recognition model RM after the incremental learning is the incrementally learned multimedia recognition model IP, and the model weight update device 1450 stores the incrementally learned multimedia recognition model IP for the next incremental learning to be read by the original model reading device 1410 as the original multimedia recognition model RM for the next incremental learning. In addition, the aforementioned forward pass, training loss, back propagation and gradient are all known terms in the field of neural networks, so there is no need to elaborate on their calculation details.

[0095] Figure 5 is a flowchart of a multimedia recognition model incremental learning method S20 according to an embodiment of the present invention. Figure 5 Describe this process.

[0096] First, in step S500 , the multimedia input device 100 is used to receive input multimedia IM from various multimedia sources.

[0097] In step S510, the feature extraction device 110 extracts the feature IF of each input multimedia.

[0098] In step S520 , the clustering device 120 performs clustering on the multimedia features in the original data set and the features IF of the input multimedia to convert them into cluster samples CS and virtual labels PL in the frequency domain.

[0099] In step S530 , the uniformly sampled samples BS and sample parameters BP are calculated by the uniform sampling device 130 .

[0100] Finally, in step S540, the multimedia recognition training device 140 performs incremental learning, that is, updates the weights of the original multimedia recognition model RM to obtain an incrementally learned multimedia recognition model IP.

[0101] The following will refer to Figure 6 The process of clustering the features of each input multimedia and the features of the multimedia in the original data set in step S520 is further described.

[0102] First, in step S600, the high similarity sample exclusion device 1210 is used to exclude high similarity samples in the feature IF of the input multimedia to avoid repeated or almost identical samples from interfering with clustering, thereby obtaining excluded high similarity samples SF.

[0103] In step S610, the first stage clustering device 1220 clusters the samples SF excluding the samples with high similarity to generate the first stage clustering result FC.

[0104] In step S620, the multimedia features DF of the original data set and the first stage clustering results FC are merged by the clustering result merging device 1230 to obtain a merged clustering result MC. Assuming that the first stage clustering results FC have N clusters and the multimedia features DF of the original data set have P clusters, N+P clusters will be obtained after merging.

[0105] In step S630 , the second stage clustering device 1240 deletes duplicate groups in the merged clustering result MC to generate a cluster sample CS.

[0106] Finally, in step S640, the virtual label generating device 1250 converts the clustered samples CS into virtual labels PL. For example, if a cluster has x samples, the virtual labels PL of the x samples are set to L x In addition, if a group has y samples, the virtual label PL of these y samples is set to L y .

[0107] The advantage of performing clustering in two stages is to avoid too many and complex multimedia features and to facilitate merging the collected multimedia features with the original dataset multiple times.

[0108] The following will refer to Figure 7 The second stage clustering process of step S630 performed by the second stage clustering device 1240 is further described.

[0109] First, in step S700, for each group in the merged clustering result MC, the centroid distance D1 between each group is calculated, wherein the centroid distance can be calculated using distance calculation methods such as Euclidean distance, cosine similarity, or discrepancy alignment metric (DAM).

[0110] In step S710, two or more groups whose center distance D1 is less than a threshold value T1 are merged into one group. Specifically, if there are N+P groups in total, the group including the most samples will be the main group, and the groups whose distance D1 from the main group is less than T1 among the other N+P-1 groups will be merged into the main group, and the above procedure will be repeated for the remaining un-merged groups. After the above fusion is completed, the clustering result MC has Q1 groups left, where Q1 is less than or equal to N+P. Among these Q1 groups, some groups actually belong to the same group, but the center distance D1 is not close enough, so they are not merged into one group.

[0111] In step S720, for each two groups whose center distance D1 is less than another threshold T2, the distance D between the samples of the two groups is calculated. k , where T2 is greater than T1. Assuming that the two groups have m and n samples respectively, there are a total of m×n distances D k . Distance D k The method can use the above-mentioned group center distance method.

[0112] In the multi-dimensional feature space, the cluster centers of different clusters may not be very close to each other, but the samples in the cluster may be very close. Therefore, in step S730, the distance between samples is converted to the distance between clusters, that is, the distance between samples D is used. k Calculate the distance D2 between the clusters, and then merge two or more clusters whose distance D2 is less than another threshold T3 into one cluster. The cluster fusion procedure in step S730 is similar to the cluster fusion procedure in step S710. At this time, the clustering result MC has Q2 clusters left, where Q2 is less than or equal to Q1. Use D k When calculating D2, you can use Average-Linkage Agglomerative, Ward Linkage, or other hierarchical clustering distance methods.

[0113] The following reference Figure 8 The process of calculating the uniformly sampled sample BS and sample parameter BP in step S530 is further described. This process can be performed by the uniform sampling device 130. In one embodiment, Figure 8 Each step of the process can be represented by Figure 3 The corresponding devices shown are executed.

[0114] First, in step S800, a first-stage uniform sampling method is used to make each group have the same probability of being sampled, so as to avoid the group with a large number of samples being sampled frequently and the group with a small number of samples being sampled seldom, which causes the trained multimedia recognition model to degenerate into being optimized only for the group with a large number of samples, wherein the uniform sampling method can be an over-sampling method or an under-sampling method.

[0115] In step S810, the samples of the sampled group are clustered to calculate the sub-clusters (Sub-Cluster) of the group. For example, some segments of sound or some parts of images may have a high degree of similarity. Although these samples belong to the same group, the group can be divided into two or more sub-groups. For example, K-Means or the aforementioned DBSCAN method can be used to calculate the sub-groups. Since the input multimedia IM is data collected by semi-supervised clustering, the number and diversity of its samples may not be sufficient. For example, a certain group only often appears from a certain multimedia input device and rarely appears from other multimedia input devices, or the location where it appears is mostly in a corner of the screen and rarely appears in other places. This will cause the data for multimedia recognition training to be too single, and only the features of frequently appearing samples are learned, and the accuracy of rare samples is poor. When the samples in the group can be divided into multiple sub-groups, it means that the number and diversity of samples are insufficient.

[0116] In step S820, a second stage of uniform sampling is performed on the subgroups, wherein the uniform sampling method may be an over-sampling or under-sampling method to avoid the uneven number of samples in the subgroups, large subgroups being sampled frequently, and manipulating the subsequent multimedia recognition training results.

[0117] Finally, in step S830 , the sample parameter BP is calculated for use in calculating the training loss in the following multimedia recognition training.

[0118] Please also refer to Figure 3 and Figure 8 , step 800 is executed by the first stage uniform sampling device 1310, step 810 is executed by the subgroup calculation device 1320, step 820 is executed by the second stage uniform sampling device 1330, and step 830 is executed by the uniform sampling sample and parameter output device 1340, wherein step 800, step 810 and step 820 can be performed simultaneously.

[0119] The following will refer to Fig. 9The process of incremental learning of the multimedia recognition model in step S540 performed by the multimedia recognition training device 140 is further described. In one embodiment, Fig. 9 Each step of the process can be represented by Figure 4 The corresponding devices shown are executed.

[0120] First, in step S900 , the weights of the original multimedia recognition model RM are loaded, wherein the weights of the original multimedia recognition model RM are obtained by previously training using the original data set.

[0121] In step S910, the original multimedia recognition model RM is used to calculate the forward transmission result FR of the uniformly sampled sample BS.

[0122] In step S920, the training loss TL of the forward transmission result FR of the uniformly sampled sample BS is calculated using the sample parameter BP.

[0123] In step S930, the training loss TL is back-transferred to obtain the gradient result GR.

[0124] In step S940 , the weights of the incrementally learned multimedia recognition model IP are optimized using the gradient result GR. The incrementally learned multimedia recognition model IP using semi-supervised multimedia clustering is better than the original multimedia recognition model RM trained only using the original dataset.

[0125] In one embodiment, when using the sample parameter BP to calculate the training loss TL of the uniformly sampled sample BS, the training loss TL is designed to adjust the loss of each uniformly sampled sample according to the sample parameter BP. In detail, during multimedia recognition training, the geodesic distance between the features of each group will be optimized on a normalized hypersphere. The features of each sample in each group need to be close to each other, while the features of each sample in different groups need to be far away from each other. In practice, a margin penalty will be given. The present invention uses the sample parameter BP to adjust the margin penalty of each sample in order to allocate feature spaces of different sizes to each group according to the number of samples, wherein a group with insufficient diversity is allocated a smaller feature space, and a group with sufficient diversity is allocated a larger feature space. The multimedia recognition model obtained by such training will be better than the traditional training method of allocating a fixed feature space to each group. For example, the multimedia input devices 100-102 will only receive certain input multimedia IM at a fixed time or a fixed location, the received images or sounds are always similar, and the light changes or angle variability are low, causing the input multimedia features IF to be too concentrated in the feature space and insufficient in diversity, resulting in the feature distribution degenerating to a point close to one point. If a fixed margin is set during training and the same feature space is allocated to each group, the group with insufficient diversity will be more dispersed in the feature space than the group with sufficient diversity, resulting in a poorly trained model. Therefore, the present invention parameterizes the number of samples of the group to dynamically adjust the margin, and then allocates feature spaces of different sizes to each group, so that the trained multimedia recognition model has better performance for the group with insufficient diversity.

[0126] Please refer to Fig.10 , which is an example of using conventional techniques to perform the aforementioned calculation of sample parameters BP for multimedia recognition training to calculate training loss (step S830) and to optimize the weights of the incrementally learned multimedia recognition model IP (step S940).

[0127] In this example, Fig.10 Two groups 21 and 22 are shown. Group 21 includes two subgroups 211 and 212, each of which includes a plurality of samples, for example, subgroup 211 includes five samples 219. Group 22 includes three subgroups 221, 222, and 223, each of which includes a plurality of samples, for example, subgroup 221 includes eleven samples 229. Group centroids 210 and 220 are the centroids of group 21 and group 22, respectively. A static margin m is defined between group 21 and group 22. s .

[0128] When the multimedia recognition training optimization model is trained, the process will minimize the distance between samples in the same group, for example, samples 219 in group 21 are close to each other, and samples 229 in group 22 are also close to each other, and maximize the distance between the center of the group in different groups, for example, the center of the group 210 of group 21 and the center of the group 220 of group 22 are far away from each other, but the disadvantage of such training is that the center of the group cannot perfectly represent all samples, so that the training of groups with fewer samples is poor. In this case, even if the center of the group has been trained far enough, the distance between samples of different groups with fewer samples may not be far enough, for example, the distance between the samples of subgroup 212 and subgroup 223 is still a little close.

[0129] If Fig.10 The example of the present invention is changed to the technical solution of the present invention, then Fig.11 shown. Fig.11 and Fig.10 One of the differences is that there is a dynamic margin m defined between group 21 and group 22. d , rather than a fixed boundary m s .

[0130] The present invention performs uniform sampling on the subgroups, so that the center of the group after uniform sampling can better represent the sample center of the group, and the subgroups with fewer samples in different groups can also move away from each other, for example, the distance between the samples of subgroup 212 and subgroup 223 becomes smaller than Fig.10 In addition, the present invention further dynamically adjusts the boundary penalty of the loss function (LossFunction) used to calculate the training loss, and allocates a smaller feature space to a group with a smaller number of samples, such as group 21, and allocates a larger feature space to a group with a larger number of samples, such as group 22, so that samples in the same group can be closer in the feature space, and samples in different groups can be farther away from each other.

[0131] Two loss functions are used for multimedia recognition training, namely the CosFace function of Margin-Based Softmax Cross-Entropy and the CosFace function of Margin-Based Softmax Cross-Entropy. and ArcFace function For example, where y i is the target cluster, y j is the non-target cluster, s is the radius of the hypersphere, m s is a fixed boundary (or fixed boundary value). The present invention uses the sample parameter BP to convert the traditional fixed boundary m s Modify to dynamic boundary m d (or dynamic boundary value). For example, md =m s +dαβ, where d is a constant used to control the floating range of the dynamic boundary, and α is the parameter of uniform sampling in the first stage. Among them, N is the number of samples in the group, N max is a constant, for example, N=7 for group 21, β is the parameter for uniform sampling in the second stage, Among them, M is the number of samples in the subgroup, M max is a constant, for example, M=2 for the subgroup 212 , and the sample parameter BP in this example includes N and M.

[0132] Please refer to Fig.12 , which is a circuit block diagram of a multimedia recognition model incremental learning system 30 according to an embodiment of the present invention. The components of the multimedia recognition model incremental learning system 30 include a central processing unit 301, a random access memory 302, graphics processors 303 and 304, a bus 305, a sound interface 306, a speaker 307, a network interface 308, a transceiver 309, a network camera 310, an input and output (I / O) interface 311, an external camera 312, and a storage device 313.

[0133] The CPU 301, the RAM 302, the GPUs 303 and 304 are all electrically coupled to the bus 305. The CPU 301 is used to control other components of the multimedia recognition model incremental learning system 30. In addition, the CPU 301 cooperates with the GPUs 303 and 304 to execute the aforementioned multimedia recognition model incremental learning method S20. The RAM 302 is used to store data required for the execution of the multimedia recognition model incremental learning method S20 and data generated when the multimedia recognition model incremental learning method S20 is executed. The bus 305 is used to transmit signals between the components of the multimedia recognition model incremental learning system 30.

[0134] The sound interface 306, the network interface 308 and the input / output interface 311 are all electrically coupled to the bus 305. The speaker 307 is electrically coupled to the sound interface 306. The transceiver 309 is electrically coupled to the network interface 308, and the network camera 310 is coupled to the transceiver 309 via a wired or wireless network. The external camera 312 and the storage device 313 are all electrically coupled to the input / output interface 311.

[0135] The speaker 307 is used to play sound, and the sound interface 306 is used to forward and convert signals related to the speaker 307 between the bus 305 and the speaker 307. The network camera 310, the external camera 312 and the storage device 313 are all multimedia sources that generate or provide input multimedia IM. The storage device 313 can be a non-volatile memory or a non-volatile data storage device such as a hard disk, which is used to store the input multimedia IM, the CPU 301 and the graphics processors 303 and 304 to load to execute the multimedia recognition model incremental learning method S20, the original multimedia recognition model RM, and the incrementally learned multimedia recognition model IP. The transceiver 309 is used to forward and convert signals related to the network camera 310 between the network camera 310 and the network interface 308. The network interface 308 is used to transmit network packets to each other through the transceiver 309 and the network camera 310 according to the network communication protocol. The input / output interface 311 is used to forward and convert signals related to the external camera 312 between the bus 305 and the external camera 312 , and is used to forward and convert signals related to the storage device 313 between the bus 305 and the storage device 313 .

[0136] In another embodiment, the Figures 1 to 4 Each device shown is implemented as a software module, which can be stored in the storage device 313 and loaded by the CPU 301 and the GPUs 303 and 304 to execute the multimedia recognition model incremental learning method S20. In another embodiment, Figures 1 to 4 The various means shown may be implemented as firmware or hardware modules.

[0137] Fig.12 The multimedia recognition model incremental learning system 30 shown includes a central processing unit 301 and two graphics processing units 303 and 304, but the present invention is not limited thereto. In another embodiment, the number of the central processing unit and the graphics processing unit of the multimedia recognition model incremental learning system 30 can be adjusted according to demand.

[0138] In one embodiment, the present invention provides a computer-readable storage medium, such as a memory, a floppy disk, a hard disk, or an optical disk. The computer-readable storage medium stores instructions, and the instructions can be read by a processor (such as a central processing unit and / or a graphics processing unit) of an electronic computing device such as a computer to execute the above-mentioned multimedia recognition model incremental learning method S20. In another embodiment, the computer-readable storage medium is a non-transitory computer readable storage medium.

[0139] According to the above content, the present invention provides a multimedia recognition model incremental learning system and a corresponding method, which uses semi-supervised multimedia clustering and uniform sampling to perform incremental learning on the multimedia recognition model. The present invention adopts two-stage clustering to avoid data that is too complex and not conducive to clustering calculations, and facilitates the merging of the collected input multimedia features with the original data set multiple times. In addition, the present invention uses the number of samples of groups and subgroups to set dynamic boundary penalties during uniform sampling, so that feature spaces of different sizes can be dynamically allocated to each group to avoid groups with insufficient diversity from being too dispersed in the feature space. The multimedia recognition model IP trained in this way will be better than the model in which a fixed feature space is allocated to each group, and will have better performance for groups with insufficient diversity.

Claims

1. A multimedia recognition model incremental learning method, comprising: Clustering is performed according to multiple features of multiple input multimedia and multiple features of multimedia in the original data set to generate cluster samples; Calculating subgroups of each of the multiple groups in the cluster sample to uniformly sample each of the subgroups to generate uniformly sampled samples; and Incremental learning of a multimedia recognition model is performed based on the uniformly sampled samples, wherein a loss function used in the incremental learning includes a dynamic boundary value between the multiple groups, and the dynamic boundary value is determined based on the number of samples of each of the groups and the number of samples of each of the subgroups.

2. The incremental learning method of multimedia recognition model according to claim 1, wherein: The step of clustering according to the multiple features includes: Excluding some features whose distance is less than a corresponding threshold from among the features of the plurality of input multimedia, and clustering the features of the plurality of input multimedia that are not excluded to generate a plurality of input groups; and The multiple input groups and the multiple original groups formed by the multiple features of the original data set are fused, wherein the multiple groups in the cluster sample are the fusion results of the multiple input groups and the multiple original groups.

3. The incremental learning method of multimedia recognition model according to claim 2, wherein: The step of fusing the plurality of input groups and the plurality of original groups comprises: At least two groups whose distances among the plurality of input groups and the plurality of original groups are smaller than corresponding thresholds are merged, wherein the distance is the distance between the group centers of the at least two groups or is calculated based on the sample distances of the at least two groups.

4. The incremental learning method of multimedia recognition model according to claim 1, wherein: The subgroups of each group are obtained by clustering the multiple features in each group.

5. The incremental learning method of multimedia recognition model according to claim 1, wherein: The multiple groups in the cluster sample are groups extracted by uniformly sampling all the groups in the cluster sample with the same probability.

6. The incremental learning method of multimedia recognition model according to claim 1, wherein: The uniformly sampled samples include features extracted by uniformly sampling the multiple features in each of the subgroups with the same probability.

7. The incremental learning method of multimedia recognition model according to claim 1, wherein: The incremental learning of the multimedia recognition model includes: Calculate the forward result of the uniformly sampled sample using the multimedia recognition model; Use the loss function to calculate the training loss of the forward pass result; Back propagating the training loss to obtain the gradient result; and The gradient result is used to optimize the weights of the multimedia recognition model.

8. The incremental learning method of multimedia recognition model according to claim 1, wherein: The dynamic boundary value includes a product of a first value and a second value, the first value decreases as the number of samples of the group corresponding to the loss function increases, and the second value decreases as the number of samples of the subgroup corresponding to the loss function increases.

9. A multimedia recognition model incremental learning system, comprising: A clustering device, used for clustering according to multiple features of multiple input multimedia and multiple features of multimedia in the original data set to generate cluster samples; A uniform sampling device, used for calculating subgroups of each of the multiple groups in the cluster sample, so as to uniformly sample each of the subgroups to generate uniformly sampled samples; and A multimedia recognition training device is used for performing incremental learning of a multimedia recognition model based on the uniformly sampled samples, wherein the loss function used in the incremental learning includes a dynamic boundary value between the multiple groups, and the dynamic boundary value is determined based on the number of samples of each group and the number of samples of each subgroup.

10. The multimedia recognition model incremental learning system according to claim 9, wherein: The clustering device further comprises: Excluding some features whose distance is less than a corresponding threshold from among the features of the plurality of input multimedia, and clustering the features of the plurality of input multimedia that are not excluded to generate a plurality of input groups; and The multiple input groups and the multiple original groups formed by the multiple features of the original data set are fused, wherein the multiple groups in the cluster sample are the fusion results of the multiple input groups and the multiple original groups.

11. The multimedia recognition model incremental learning system according to claim 9, wherein: The multimedia identification training device comprises: Calculate a forward result of the uniformly sampled sample using the multimedia recognition model; Use the loss function to calculate the training loss of the forward pass result; Back propagating the training loss to obtain the gradient result; and The gradient result is used to optimize the weights of the multimedia recognition model.

Images