An indoor scene visual recognition method and system fusing multiple balance strategies

Abstract

The application discloses an indoor scene visual recognition method and system fusing multiple balance strategies, and comprises the following steps: using a preheated model to calculate the class center of each class in a long-tail training set; constructing multiple training subsets with different feature distributions through different resampling strategies; combining a self-defined loss function, training the preheated model using the training subsets until the loss function converges, and making the model tend to learn balanced features among the training subsets; and processing the feature imbalance problem within the classes in the training set. Meanwhile, a regular term is applied to the classifier to adjust the weight difference of the head and tail classes, and a trained model is obtained after the loss function converges to a certain extent, so that the imbalance of the weights of each class on the classifier caused by the imbalance of the class samples in the training set is reduced. The application simultaneously solves the problems caused by the imbalance of the class samples and the imbalance of the non-class attributes of the intra-class samples in the model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of long-tail visual recognition and balanced representation learning, and particularly relates to an indoor scene visual recognition method and system fusing multiple representation balancing strategies. BACKGROUND

[0002] Long-tail visual recognition is one of the most challenging and key technologies in the field of computer vision, because any natural acquisition and acquisition dataset has more or less the problem of long-tail distribution imbalance, which is often overlooked, thus affecting the model training in some inexplicable way. In previous studies, people pay attention to the imbalance between classes, which has been artificially solved in most computer vision tasks. For example, most of the currently public datasets will be artificially balanced after natural collection, so unless it is a special long-tail research field, the currently public datasets are mostly class-balanced. But this does not mean that class rebalancing is meaningless, because this research will reduce the necessity of artificial class balancing operation after natural collection of the dataset, and reduce the time and labor cost of this link. In addition to the class balancing problem, there is another problem that has not been concerned and solved before, that is, the intra-class long-tail imbalance problem, such as some common phenomena: why are the performances of samples in the same class inconsistent and show long-tail distribution; why are some samples in the tail class predicted as having similar attributes to the head class in the visual recognition task.

[0003] In the existing long-tail visual recognition task, especially in indoor scenes (such as classrooms, canteens, shopping malls, etc.), due to the long-tail distribution of sample distribution between classes of objects in indoor space, the model obtained by directly training the acquired training set cannot obtain a distribution similar to the class-balanced test set in the same class. High class richness of samples means that potential confounding factors can be avoided, and on the contrary, it means that it is more susceptible to confounding factors. The present method focuses on using long-tail distribution data to train more robust and balanced features for visual recognition tasks in small spaces such as indoors.

[0004] The long-tail visual recognition task aims to improve the performance of a given long-tail training set under a class-balanced evaluation method. The most obvious confounding factor under the long-tail distribution dataset is the "class", so the "class" is first deconfounded. How to effectively use the unbalanced data to reduce the cost of data collection and train a more balanced model is a problem we are concerned about. The rebalancing method for the imbalance between classes can be divided into four categories in general: the first category is the resampling strategy on the training dataset, such as downsampling of the head classes and upsampling of the tail classes. This method has the problem of insufficient utilization of the dataset. Downsampling of the head classes makes part of the data not fully utilized, and upsampling of the tail classes has the problem of deviation of the new sample distribution from the original distribution. The second category is reweighting, that is, the processing of the loss function in the training stage. Because of the flexibility and simplicity of the loss function calculation, this method is applied to many tasks that require complex modeling. The third category is transfer learning. Based on the imbalance of the long-tail data distribution, the learned knowledge is transferred to the feature learning of the tail classes through some way, such as using the distribution information of the head classes to enhance the samples of the tail classes. This way often has a complex model. The fourth category is model integration, which aims to improve the performance of the head classes and the tail classes in the long-tail training set through multiple sub-models.

[0005] In addition, existing researches are mostly aimed at the inter-class imbalance problem, and the given method is to train an inter-class imbalance classifier, which tends to improve the confidence of the tail classes and suppress the confidence of the head classes, thereby "correcting" the problem that the model originally tends to predict the tail class samples as the head class category, so as to achieve inter-class balance of the head and tail classes. However, even if the number of samples between classes is balanced, the imbalance within the same class may also cause imbalance within the class due to the uneven distribution of features, affecting the recognition effect of the model. SUMMARY

[0006] The purpose of the present application is to overcome the balanced representation learning problem in the long-tail training set obtained from indoor scenes, especially for the previously ignored intra-class imbalance problem, and to provide an indoor scene visual recognition method and system that integrates multiple representation balancing strategies.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] In the first scheme, a kind of indoor scene visual recognition method integrating multiple representation balancing strategies is provided, and the method comprises:

[0009] S1, sampling a long-tail training set;

[0010] S2, preheating the model and defining the loss function;

[0011] S3, using the preheated model to calculate the class center of each class in the long-tail training set;

[0012] S4, constructing multiple training subsets with different feature distributions through different resampling strategies;

[0013] S5, using the training subsets to train the preheated model until the loss function converges, combining with the self-defined loss function, so that the model tends to learn the balanced features among the training subsets;

[0014] S6, applying a regularization term to the classifier of the model trained in step S5 to adjust the weight difference of the head and tail classes, and obtaining the trained model when the loss function converges to a certain extent.

[0015] S7, using the trained model for visual recognition.

[0016] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the multiple training subsets with different feature distributions are constructed through different resampling strategies, including:

[0017] A plurality of new small subsets are obtained by using different resampling methods for each class in the long-tail training set, and then all small subsets using the same resampling method are combined together to obtain multiple large training subsets.

[0018] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the different resampling methods include:

[0019] One is to sample the samples in each class with the same weight, and the other is to sample the samples in each class with weights according to the 80-20 law.

[0020] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the samples in each class are sampled with weights according to the 80-20 law, including:

[0021] A portion of the samples in the current class are upsampled until the proportion of the samples with the lowest 20% prediction confidence in the class reaches 80% of the original set sample number; meanwhile, another portion of the samples in the current class are downsampled to 20% of the original set sample number.

[0022] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the upsampled method is MixUp data enhancement.

[0023] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the preheated model is trained using the training subset until the loss function converges, comprising:

[0024] The step S4 of periodically reconstructing the training subset is repeated, and the model is trained using the reconstructed training subset.

[0025] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the periodicity refers to repeating every 20 epochs.

[0026] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the class center of each class is updated before each repetition.

[0027] As a preferred option, an indoor scene visual recognition method integrating multiple representation balancing strategies, the step S6 comprises:

[0028] Randomly initialize the parameters of the model classifier and periodically adjust the classifier using the training subset obtained by resampling.

[0029] In the second scheme, an indoor scene visual recognition system integrating multiple representation balancing strategies is provided, the system comprises:

[0030] A data acquisition module for sampling a long-tail training set;

[0031] A model preheating module for preheating the model and customizing the loss function;

[0032] A class center calculation module for calculating the class center of each class in the long-tail training set using the preheated model;

[0033] A training subset construction module for constructing multiple training subsets with different feature distributions through different resampling strategies;

[0034] An intra-class balancing training module for training the preheated model using the training subset until the loss function converges, combining the customized loss function, so that the model tends to learn the balanced features among the training subsets;

[0035] An inter-class balancing training module for adjusting the weight difference of the head and tail classes by imposing a regular term on the classifier of the model obtained by the intra-class balancing training module, and obtaining a trained model when the loss function converges to a certain extent.

[0036] A recognition module for visual recognition using the trained model.

[0037] It needs to be further explained that the technical features corresponding to the above options can be combined or replaced with each other to form new technical solutions without conflict.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] (1) The present application constructs multiple training subsets with different feature distributions through different resampling strategies, and then uses the training subsets to train the preheated model until the loss function converges, using the self-defined center loss function, so that the model tends to learn the balanced features between the training subsets. The model tends to learn the unbiased representation within the class, solving the previously ignored intra-class bias problem. At the same time, a regularization term is applied to the classifier to adjust the weight difference of the head and tail classes. When the loss function converges to a certain extent, a trained model is obtained, which reduces the imbalance of the weights of each class on the classifier caused by the imbalance of the class samples in the training set. The present application simultaneously solves the problems caused by the imbalance of class samples and the imbalance of non-class attributes of intra-class samples in model training. On the basis of the previous inter-class balancing work, further consideration is given to the influence of some "non-class factors" such as background, posture, angle, etc. on the visual recognition task results. The two balancing methods complement each other, making the overall effect of the model better.

[0040] (2) The present application explicitly points out and includes the "intra-class long tail" in modeling, which explains why the performance within the same class presents a long-tail distribution, and why some samples of the tail class in the long-tail training set are predicted to have similar "non-class attributes". And it improves the problem of not improving both accuracy and precision in the original research.

[0041] (3) The "intra-class bias" balancing method given by the present application has the characteristics of non-invasiveness, which can be combined with many methods such as cRT, LWS, BalancedSoftmax, BBN, etc. The method of the present application can be directly embedded into the existing long-tail recognition model without affecting the structure of the model before, and provides the model with the ability of intra-class bias balancing, achieving complementation. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of an indoor scene visual recognition method fusing multiple representation balancing strategies is shown for the embodiments of the present application;

[0043] Figure 2 A diagram showing the inter-class sample long-tail distribution and intra-class attribute long-tail distribution is shown for the embodiments of the present application;

[0044] Figure 3 A structural causal diagram of the new modeling method for long-tail visual recognition problems is shown for the embodiments of the present application;

[0045] Figure 4 The overall framework of the embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0046] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0047] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as there is no conflict between them.

[0048] In an exemplary embodiment, a method for visual recognition of indoor scenes by fusing multiple representation balancing strategies is provided, as shown in Figure 1 The method comprises:

[0049] S1, sampling to obtain a long-tail training set;

[0050] S2, preheating a model and defining a loss function;

[0051] S3, using the preheated model to calculate the class center of each class in the long-tail training set;

[0052] S4, constructing multiple training subsets with different feature distributions through different resampling strategies;

[0053] S5, using the training subsets to train the preheated model in combination with the defined loss function until the loss function converges, so that the model tends to learn features balanced among the training subsets;

[0054] S6, applying a regularization term to the classifier of the model trained in step S5 to adjust the weight difference of the head and tail classes, and obtaining a trained model when the loss function converges to a certain extent.

[0055] S7, using the trained model for visual recognition.

[0056] Specifically, the success of the existing long-tail visual recognition balancing method mainly comes from expanding the confidence boundary of the tail classes to include more tail class samples and thus improving the accuracy of the tail classes. This is actually a trade-off between accuracy and precision, because the confusing features are not focused on, and the model is not really guided to ignore these confusing features. Accuracy is defined as Precision is defined as Wherein, #AllSamples refers to the total number of picture samples in the data set, #CorrectPredictions refers to the number of picture samples that the model predicts the correct class (prediction is correct), and the accuracy is the proportion of the model predicting the class of n pictures, of which the prediction is correct. #SamplesPredictionAsThisClass refers to the number of picture samples predicted by the model as a certain class, such as the model predicting 10 out of 100 pictures as tigers and 12 as mice, then #SamplesPredictionAsThisClass of the tiger class is equal to 10, and #SamplesPredictionAsThisClass of the mouse class is equal to 12. The precision rate is also called the precision rate, which refers to the proportion of samples predicted correctly in the samples predicted as a certain class in the data set, and then each class is calculated and the total number of classes is processed. #class is the total number of classes in the data set, such as the model predicting 10 out of 100 pictures as tigers, but only 6 of them are real tigers, so the precision rate is equal to 6 / 10.

[0057] Due to the difference in the number of samples of the head and tail classes, the weights of the model classifier of the head and tail classes are different, which causes the difference in the decision boundary of the head and tail classes, and finally causes the imbalance problem of the performance of the head and tail classes in the model. To solve this problem, the method provided by the present application is to apply a regularization term to the classifier of the model to reduce the weight difference of the head and tail classes in the classifier and alleviate the imbalance between classes. In addition, one point that is ignored in the previous popular method is the imbalance within the class. The method provided by the present application is to construct multiple training subsets with different feature distributions through different resampling strategies, and then combine a custom center loss function to train the preheated model using the training subsets until the loss function converges, so that the model tends to learn balanced features between training subsets, so that the model tends to learn class-unbiased representations, solving the previously ignored class imbalance problem. The present application simultaneously solves the problems caused by the imbalance of class samples and the imbalance of non-class attributes of class samples in model training, and further considers the influence of some "non-class factors" such as background, posture, angle, etc. on the results of visual recognition tasks on the basis of the previous inter-class balance work. The two balancing methods complement each other, so that the overall effect of the model is better.

[0058] In one example, an indoor scene visual recognition method that fuses multiple representation balancing strategies, said method is constructed by different resampling strategies to construct multiple training subsets with different feature distributions, comprising:

[0059] For each class in the long-tailed training set, different resampling methods are used to obtain multiple new small subsets, and then all small subsets using the same resampling method are combined to obtain multiple large training subsets.

[0060] Specifically, the sample X can be represented as "class information" and "a series of attributes". That is, the sample X is represented by two latent feature vectors, Zc and Za. Where Zc is a class feature with invariance, which can be understood as the template information or prototype of the class, and Za is an attribute feature that changes with the distribution of the field, such as texture, posture, background, lighting, etc. In this way, the visual recognition problem under the long-tailed data set can have a new modeling method, which explains the class bias and attribute bias. First, give the structure of the causal model of this modeling method, as shown in Figures 2-3 Zc is the class prototype, that is, there is a corresponding class prototype Zc for a given class Y. Here Zc is defined as a binary vector, which contains multiple components of Y, such as Y = person Zc = [head = 1, torso = 1, arm = 1, leg = 1, other = 0]. This is also to adapt to more fine-grained classification, such as Y = cow head Zc can be [head = 1, torso = 1, arm = 1. leg = 1, horn = 1, other = 0], without another irrelevant one-hot vector to represent. And Zc has a corresponding attribute set Za, such as "hair" has "long hair" and "short hair". The attribute Za is affected by the external noise ε of non-class factors. Specifically, the object image X is affected by the class template Zc and the corresponding attribute set Za.

[0061] First, the visual recognition task can be regarded as P(y|x), Zc is shared by all samples within the class, and the difference between Za causes the difference between different samples within the class. According to Figure 3 , the following modeling formula can be given:

[0062]

[0063] Finally, the result of the decomposition is the class template, the intra-class attribute offset, and the inter-class offset from left to right, which together cause the different performances between different samples. Here, first of all, pay attention to the difference within the class, although Zc is shared by all samples within the class, but due to the difference of Za, the visual recognition of samples of the same class will also appear difficult and easy, such as green banana in banana is the tail class, then the green banana will become a difficult sample, as shown in Figure 2 In addition to this, the intra-class attribute offset can also explain why the sample is misclassified. For example, green is very common in loofah, so there may be a pseudo-correlation between "green" and "melon", which makes the green banana have a high probability of being classified into the "loofah" class.

[0064] 1) The performance difference between samples can be explained as follows:

[0065] Because

[0066] Then p(Y = banana | Zc = color, Za = yellow) < p(Y = banana | Zc = color, Za = green)

[0067] 2) The samples in the category produce false associations with other categories (samples are misclassified into other categories) can be explained as follows: if the number of green loofah samples is too large, then will be much greater than 1, and the value of p(Y = loofah | Zc = color, Za = green) will be large. That is, if the color of a banana sample is green, it is likely to be classified as loofah, and the green banana itself further exacerbates this phenomenon because it is a tail class.

[0068] The above fully describes how intra-class bias is generated and acts on the visual recognition task. For intra-class bias, the overall architecture given by the present application is as shown in Figure 4 , in combination with Figure 4 The embodiment gives a specific visual recognition process:

[0069] First, sample a long-tail training set {{x, y}}, x is a sample, y is a class label, and the sample picture size is adjusted to 112X112;

[0070] Then, preheat the backbone f(·; θ) (feature extractor) and the classifier g(·, ω), and the loss function uses cross-entropy θ, ω ∈ argmin θ,ω L cls (f(x; θ), y; ω), where θ and ω are the learnable parameters of the backbone network and the classifier, respectively, L cls is the cross-entropy, the time is 60 epochs, and the optimizer is SGD with a batch size of 256.

[0071] The class centers {Cy} of each category are calculated by the preheated model of the previous step;

[0072] Then, after preheating, the model is trained using the training subset, and the training subset is periodically reconstructed and used to train the model until the loss function converges. The periodicity refers to repeating every 20 epochs. Specifically, two training subsets are constructed by different resampling strategies for the following training {(x e1 , y e1 ), {(x e2 , y e2 )} = SubSetConstruct({x, y}, θ, ω), and the loss function is θ, ω ∈ argminθ,ω ∑ s∈ε ∑ i∈s (L cls +α·L IFL ), while updating the class centers Cy of each class before each iteration

[0073] {Cy}→MovingAverage({Cy},{(f(x s1 ; θ), y s1 )},{(f(x s2 ; θ), y s2 )})

[0074] where

[0075] where

[0076] Finally, the balanced feature extractor f(·; θ) is obtained after the training process above. An additional 10 epochs of training is performed to handle the class imbalance, during which the parameters θ of the feature extractor are frozen, and the parameters ω of the linear classifier are randomly initialized and adjusted separately using the resampled class-balanced training set for the next 10 epochs. The final balanced feature extractor f(·; θ) and classifier g(·, ω) are obtained.

[0077] In one example, an indoor scene visual recognition method that fuses multiple representation balancing strategies, the different resampling methods include:

[0078] One is to assign the same weight to the samples in each class for sampling, and the other is to assign weights to the samples in each class according to the 80-20 law.

[0079] Specifically, because there is no theoretical proof that the multiple features learned from the training set can be disentangled, Zc and Za cannot be separated by simple and direct feature selection, and the scheme given by the present application is to guide the model to reduce the learning of Za by constructing two training subsets. First, we get the empirical conclusion through experiments: the cosine similarity of each sample to its class center is inversely proportional to the rarity of its Za, that is, the more rare the Za of this sample, the smaller the prediction logit of this sample given by the model. Through this conclusion, the prediction logit of the sample given by the model can be used as the position of the Za of the sample in the long-tail distribution of Za within the class of the sample. Then is the resampling method when constructing the training subset, because the difference between the two training subsets is the distribution of Za in the subset, and according to the aforementioned empirical conclusion, the distribution of Za can be represented by the prediction logit of the sample given by the model, so the specific training subset resampling strategy is as follows. After obtaining the preheated feature extractor and classifier, the prediction confidence of all samples in the current entire training set can be obtained, and when the sample label is k, the prediction confidence is P(Y=k|X in k). As mentioned in the foregoing, the prediction confidence of the sample can be used to represent the distribution of Za of the current sample x in the belonging class y. Because the reason for constructing multiple training subsets is to guide the model to reduce the learning of Za that will bring attribute bias, and instead use Zc as the basis for visual recognition tasks. Here the specific training subset resampling method has two kinds. One is to give the same weight to the samples in each class, so that the distribution of Za of the samples in each class in the subset obtained is the same as that in the original set. The second resampling method is according to the 80-20 rule, and the sampling weight given to each sample in the class is (1-p(Y=k|Zc,Za)) β , where β is automatically adjusted to sample the 20% of the samples with the lowest p(Y=k|Zc,Za) value in the k class to reach 80% of the class, briefly, it can be understood that the weight given to each sample in the class k by the second resampling method is just opposite to the first resampling method. Then use the two resampling methods to get two new small subsets for each class in the original training set, then combine all the small subsets using the same resampling method, finally get two large training subsets, thus completing the construction of a training subset.

[0080] Further, according to the 80-20 rule, the samples in each class are sampled by assigning weights, including:

[0081] The part of the samples in the current category is up-sampled until the proportion of the samples with the lowest prediction confidence in the category reaches 80% of the original set sample number; meanwhile, another part of the samples in the current category is down-sampled to 20% of the original set sample number. Specifically, because the part of the samples needs to be up-sampled, MixUp is used here. MixUp is a data enhancement method commonly used for up-sampling. First, find the 20% of the samples in the current class with the lowest p(Y=k|Zc,Za) value, then randomly take two samples, and then randomly select a fusion ratio μ in [0, 1], μ conforms to the beta distribution. Then add each pixel of the two images randomly selected before to fuse, outputs=μ*images1+(1-μ)*images, and the output obtains a new sample for up-sampling. The label of the newly generated sample is In this way, a pair of new {x, y} is formed. The up-sampling process in the current class is performed until the proportion of the samples with the lowest p(Y=k|Zc,Za) value in the category reaches 80% of the original set sample number, and meanwhile another part of the original set is down-sampled to 20% of the original set sample number, thus completing the construction process of the second training subset.

[0082] In another exemplary embodiment, a visual recognition system for indoor scenes is provided, which fuses multiple representation balancing strategies, and the system comprises:

[0083] A data collection module is configured to sample a long-tail training set;

[0084] A model warm-up module is configured to warm up a model and customize a loss function;

[0085] A class center calculation module is configured to calculate the class center of each class in the long-tail training set using the warmed-up model;

[0086] A training subset construction module is configured to construct multiple training subsets with different feature distributions through different resampling strategies;

[0087] An intra-class balancing training module is configured to train the warmed-up model using the training subsets in combination with the customized loss function until the loss function converges, so that the model tends to learn balanced features among the training subsets;

[0088] An inter-class balancing training module is configured to impose a regular term on the classifier of the model obtained by the intra-class balancing training module to adjust the weight difference between the head and tail classes, and obtain a trained model when the loss function converges to a certain extent.

[0089] A recognition module is configured to perform visual recognition using the trained model.

[0090] The training subset construction module constructs multiple training subsets with different feature distributions through different resampling strategies, the intra-class balance training module uses the training subsets output by the training subset construction module to train the preheated model in combination with a self-defined center loss function until the loss function converges, so that the model tends to learn balanced features among the training subsets, and the model tends to learn an unbiased representation within the class, thereby solving the intra-class bias problem that has been ignored. The inter-class balance training module applies a regularization term to the classifier to adjust the weight difference of the head and tail classes, and obtains a trained model when the loss function converges to a certain extent, thereby reducing the imbalance of the weights of each class on the classifier caused by the imbalance of the samples in the training set. At the same time, the problems caused by the imbalance of the class samples and the non-class attributes of the intra-class samples to the model training are solved, and on the basis of the previous inter-class balance work, further consideration is given to the influence of some "non-class factors" such as background, posture, and perspective on the visual recognition task results, and the two aspects of the balance method complement each other, so that the overall effect of the model is better.

[0091] In another exemplary embodiment, the present application provides a storage medium having computer instructions stored thereon, the computer instructions being executable to perform the steps of the indoor scene visual recognition method integrating multiple representation balancing strategies.

[0092] Based on such understanding, the technical solution of the present embodiment or the part that essentially contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0093] In another exemplary embodiment, the present application provides a terminal including a memory and a processor, the memory having computer instructions executable on the processor stored thereon, and the processor executing the computer instructions to perform the steps of the indoor scene visual recognition method integrating multiple representation balancing strategies.

[0094] The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present application.

[0095] Embodiments of the subject matter and the functional operations described in this specification can be implemented in various computing systems, including in computer software or firmware, in computer hardware, including the structural equivalents of such software and firmware, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, namely, one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively, or additionally, the program instructions can be encoded on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computing system can include, among other computer-readable media, computer storage media (e.g., RAM, ROM, EEPROM, flash memory, or other memory technology), computer- readable non-transitory media (e.g., hard-disk storage, or other magnetic storage devices, or optical storage devices), or any other volatile or non-volatile computer-readable media.

[0096] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) and / or by programmable data processing apparatuses that can be

[0097] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. Computer programs, also called programs, routines, software modules, software applications, components, or the like, can be written in any form of programming language, including compiled or interpreted languages, and / or computer

[0098] While this specification contains many specifics, these should not be construed as limiting the scope of any invention or of what can be claimed, but as merely providing illustrations of some of the embodiments of the inventions. Certain features that are, for clarity, described above in the context of separate embodiments can also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described above in the context of a single embodiment can also be provided separately or in any suitable subcombination. In addition, while features can be described above as being implemented using one or more components, it should be appreciated that such components can be implemented by a combination of hardware and / or software that is managed by one or more processors. Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order, nor that all illustrated operations be performed, to implement a desired result. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0099] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order, nor that all illustrated operations be performed, to implement a desired result. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0100] The above detailed description merely describes certain embodiments of the application, and is not intended to limit the scope of the application. Various modifications and alterations to this description will become apparent to those skilled in the art from this description, which is offered by way of example only, and without departing from the scope of the application.

Claims

1. A method for fusing a plurality of indoor scene visual recognition strategies characterized by a balance strategy, characterized in that, The method comprises: S1, sampling to obtain a long-tail training set; S2, preheating a model and defining a loss function; S3, using the preheated model to calculate the class center of each class in the long-tail training set; S4, constructing multiple training subsets with different feature distributions through different resampling strategies; the step of constructing multiple training subsets with different feature distributions through different resampling strategies comprises: using different resampling methods to obtain multiple new small subsets for each class in the long-tail training set, and then grouping all small subsets obtained by using the same resampling method to obtain multiple large training subsets; the different resampling methods comprise: one is to sample by assigning the same weight to the samples in each class, and the other is to sample by assigning weights to the samples in each class according to the 80-20 law; S5, using the training subsets to train the preheated model until the loss function converges, in combination with the defined loss function, so that the model tends to learn the balanced features among the training subsets; S6, applying a regularization term to the classifier of the model trained in step S5 to adjust the weight difference between the head and tail classes, and obtaining a trained model when the loss function converges; S7, using the trained model for visual recognition.

2. The method according to claim 1, wherein, Sampling by assigning weights to the samples in each class according to the 80-20 law comprises: up-sampling part of the samples in the current class until the proportion of the samples with the lowest prediction confidence in the current class reaches 80% of the proportion of the original set of samples; meanwhile, another part of the samples in the current class are down-sampled to 20% of the proportion of the original set of samples.

3. The method according to claim 2, wherein, The up-sampling method is MixUp data enhancement.

4. The method of claim 1, wherein the method is a fusion of multiple representation balancing strategies. The step of using the training subsets to train the preheated model until the loss function converges comprises: periodically repeating step S4 to reconstruct the training subsets, and training the model using the reconstructed training subsets.

5. The method according to claim 4, wherein, The periodicity refers to repeating once every 20 epochs.

6. The method of claim 4, wherein the method further comprises: The class center of each class is updated before each repetition.

7. The method of claim 1, wherein the method is a fusion of multiple representation balancing strategies. The step S6 comprises: randomly initializing the parameters of the model classifier and periodically adjusting the classifier using the training subsets obtained by resampling.

8. A fusion of various indoor scene visual recognition systems representing a balance strategy, characterized in that, The system comprises: a data acquisition module for sampling to obtain a long-tail training set; a model preheating module for preheating a model and defining a loss function; a class center calculation module for using the preheated model to calculate the class center of each class in the long-tail training set; a training subset construction module for constructing multiple training subsets with different feature distributions through different resampling strategies; the step of constructing multiple training subsets with different feature distributions through different resampling strategies comprises: using different resampling methods to obtain multiple new small subsets for each class in the long-tail training set, and then grouping all small subsets obtained by using the same resampling method to obtain multiple large training subsets; the different resampling methods comprise: one is to sample by assigning the same weight to the samples in each class, and the other is to sample by assigning weights to the samples in each class according to the 80-20 law; The in-class balance training module uses the training subsets to train the preheated model until the loss function converges, and makes the model tend to learn the features balanced among the training subsets in combination with a self-defined loss function; The inter-class balance training module adjusts the weight difference between the head and tail classes by imposing a regular term on the classifier of the model obtained by the in-class balance training module, and obtains the trained model after the loss function converges; The recognition module uses the trained model for visual recognition.

Images