Model training, image classification methods, servers and storage media

Abstract

The embodiments of the present application provide a model training, image classification method, server and storage medium. In this embodiment, the multiple expert models in the first model include at least two expert models with different network structures, so that different expert models can be used for knowledge learning, which is beneficial for identifying samples of different difficulty levels when the number of samples is limited. The fusion model is trained according to the image features extracted from the sample image after the expert model converges, so that the fusion model can dynamically learn how to reasonably use the knowledge learned by multiple expert models according to the knowledge learned by the expert model, so that expert models with different performances make different contributions to the final output classification results, and efficient learning can be performed even when the number of sample images is small.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a model training, image classification method, server and storage medium. Background Art

[0002] The development of neural networks has continuously promoted the progress of artificial intelligence, and the training of neural networks is inseparable from massive training data and pre-trained models. In some application scenarios, there is often a technical problem that insufficient training data cannot train a sufficiently effective neural network model. Therefore, a solution is yet to be proposed. Summary of the invention

[0003] Various aspects of the embodiments of the present application provide a model training, image classification method, server and storage medium for improving the performance of a neural network model when training data is limited.

[0004] An embodiment of the present application provides an image classification method, comprising: inputting an image to be processed into a first model; the first model comprises: multiple expert models and a fusion model connected to the multiple expert models; the multiple expert models comprise at least two expert models with different network structures; performing feature extraction on the sample image through the multiple expert models to obtain image features corresponding to each of the multiple expert models; performing fusion processing on the image features corresponding to each of the multiple expert models according to the feature fusion parameters of each of the multiple expert models through the fusion model; performing classification operation processing on the fused features through the fusion model to obtain a first classification result.

[0005] An embodiment of the present application also provides a scene classification method, comprising: acquiring video data obtained by shooting a target place; inputting any frame image in the video data into a first model; the first model comprises: multiple expert models and a fusion model connected to the multiple expert models; the multiple expert models comprise at least two expert models with different network structures; performing feature extraction on the image through the multiple expert models to obtain image features corresponding to each of the multiple expert models; performing fusion processing on the image features corresponding to each of the multiple expert models according to the feature fusion parameters of each of the multiple expert models through the fusion model; performing classification operation processing on the fused features through the fusion model to obtain a scene category corresponding to the target place.

[0006] The embodiment of the present application also provides a model training method, wherein a first model to be trained includes: multiple expert models, and a fusion model connected to the multiple expert models; the multiple expert models include at least two expert models with different network structures; the method includes: obtaining image features obtained by extracting features of a sample image by the multiple expert models after convergence; the sample image is marked with a true label value; through the fusion model, according to the feature fusion parameters of each of the multiple expert models, the image features corresponding to each of the multiple expert models are fused, and the fused features are classified and processed to obtain a first classification result; with the learning goal of making the gap between the first classification result and the true label value converge to a first specified range, the feature fusion parameters of each of the multiple expert models are adjusted.

[0007] The embodiment of the present application also provides an image classification method, comprising: responding to a call request from a client to a first interface, obtaining an image to be processed contained in an interface parameter; inputting the image to be processed into a first model; the first model comprising: a plurality of expert models and a fusion model connected to the plurality of expert models; the plurality of expert models comprising at least two expert models with different network structures; performing feature extraction on the sample image through the plurality of expert models to obtain image features corresponding to each of the plurality of expert models; performing fusion processing on the image features corresponding to each of the plurality of expert models through the fusion model according to the feature fusion parameters of each of the plurality of expert models, and performing classification operation processing on the fused features to obtain a first classification result; and returning the first classification result to the client.

[0008] An embodiment of the present application also provides a server, comprising: a memory, a processor, and a communication component; the memory is used to store one or more computer instructions; the processor is used to execute one or more computer instructions to execute the steps in the method provided in the embodiment of the present application.

[0009] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed, the steps in the method provided in the embodiment of the present application can be implemented.

[0010] In an embodiment of the present application, the multiple expert models in the first model include at least two expert models with different network structures, so that different expert models can be used for knowledge learning, which is beneficial for identifying samples of different difficulty levels when the number of samples is limited. The fusion model is trained based on the image features extracted from the sample image after the expert model converges, so that the fusion model can dynamically learn how to reasonably use the knowledge learned by multiple expert models based on the knowledge learned by the expert model, so that expert models with different performances make different contributions to the final output classification results, greatly improving the overall performance of the first model. At the same time, this training method can reduce the dependence on the number of sample images, and can also perform efficient learning when the number of sample images is small, thereby obtaining better model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A flowchart of a model training method provided for an exemplary embodiment of the present application;

[0013] Figure 2 A flowchart of a model training method provided for another exemplary embodiment of the present application;

[0014] Figure 3 A schematic diagram of a network structure of a hybrid expert model provided for an exemplary embodiment of the present application;

[0015] Figure 4 A flowchart of an image classification method provided for an exemplary embodiment of the present application;

[0016] Figure 5 A flowchart of an image classification method provided for another exemplary embodiment of the present application;

[0017] Figure 6 A schematic diagram of the structure of a terminal device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0019] The development of neural networks has continuously promoted the progress of artificial intelligence, and the training of neural networks is inseparable from massive training data and pre-trained models. In some application scenarios, there is often a technical problem of not being able to train a sufficiently effective neural network model due to the lack of sufficient training data.

[0020] In some model training methods, a multi-expert mixed model can be used for knowledge learning. When performing classification prediction, in one method, the prediction results output by multiple experts are averaged as the final prediction result. In another method, a routing structure is used to determine which expert model outputs the prediction results to output the final prediction result. However, the above method does not fully utilize the characteristics of the multi-expert mixed model, and does not consider the performance advantages and disadvantages of different expert models, resulting in limited accuracy of the classification results of the output image. Therefore, it is impossible to improve the performance of the trained neural network model in the absence of sufficient training data.

[0021] In view of the above technical problems, some exemplary embodiments of the present application provide a solution, which will be described in detail in the following part with reference to the accompanying drawings.

[0022] Figure 1 A flow chart of a model training method provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0023] Step 101, obtaining image features obtained by extracting features of a sample image by multiple expert models in the first model after convergence; the sample image is marked with a label true value; the multiple expert models include at least two expert models with different network structures.

[0024] Step 102: using the fusion model in the first model, according to the feature fusion parameters of each of the multiple expert models, fuse the image features corresponding to each of the multiple expert models, and perform classification operation on the fused features to obtain a first classification result.

[0025] Step 103: adjusting the feature fusion parameters of each of the plurality of expert models with the learning goal of making the gap between the first classification result and the true value of the label converge to a first specified range.

[0026] In this embodiment, the first model for image classification can be a mixed expert model, which can be implemented based on a multi-expert model (Mixture of Experts, MoE). Among them, the multi-expert model is a type of neural network. Different from general neural network training, the multi-expert model includes multiple models, each of which is an expert. In addition to multiple expert models, the multi-expert model also includes a gating module, which is used to control which expert is selected to use. In this embodiment, the gating module is implemented as a fusion model connected to multiple expert models. The mixed expert model refers to the integration of multiple expert models for a single task.

[0027] In this embodiment, the multiple expert models include at least two expert models with different structures. That is, the network structures of the multiple expert models include at least two different network structures. Of course, the network structures of the multiple expert models may also be different, and this embodiment does not limit this. In the image classification scenario, the multiple expert models may include at least two convolutional neural network models with different structures, or different expert models may be implemented based on convolutional neural network models with different structures. Among them, the multiple layers of convolution in the convolutional neural network model constitute the feature extractor in the expert model, so that the expert model can extract features from the input training data. Of course, in different embodiments, other neural network models can also be used as feature extractors, not limited to the convolutional neural network model.

[0028] The number of expert models in the first model may be 2, 3, or N, which is not limited in this embodiment. N is a positive integer greater than 2.

[0029] In some optional embodiments, the network structure of the expert model may include but is not limited to: ResNet, EfficientNet, RegNet, ResNeSt, SKNet, ECAResNet, NFNet, EfficientNetV2, ViT, T2T-ViT, DeiT, Swin, VOLO and other network structures. In practice, at least two network structures may be selected from the above network structures as the network structures of multiple expert models, and at least two network structures may be selected from other network structures (such as AlexNet, VggNet) other than the above network structures, which is not limited in this embodiment.

[0030] In this embodiment, the multiple expert models in the first model refer to expert models that have been trained and converged. Multiple expert models can learn the ability to extract image features based on sample images under the supervision of supervisory signals to obtain their own model parameters. Among them, the training process of the expert model will be explained in subsequent examples and will not be repeated here. After the multiple expert models converge, the fusion model (Fusion Module) can be trained based on the outputs of the multiple expert models, so as to dynamically learn the knowledge learned by the multiple expert models, so as to learn how to make full use of the performance advantages of each of the multiple experts.

[0031] When training the fusion model, the image features obtained by extracting features from the sample images after convergence of multiple expert models can be obtained. The sample images are labeled with ground truth. This labeling refers to the category labeling of the sample images, so that the sample images have category labels, so that the fusion model can be supervised based on the category labels.

[0032] After the feature images extracted by the multiple expert models are input into the fusion model, the fusion model can fuse the image features corresponding to the multiple expert models according to the feature fusion parameters of the multiple expert models and perform classification operations to obtain the classification results.

[0033] The feature fusion parameters of each of the multiple expert models are located in the fusion model and can be regarded as part of the model parameters of the fusion model. After obtaining the classification result output by the fusion model, the model parameters of the fusion model can be adjusted under the supervision of the true value of the label annotated on the sample image. That is, the feature fusion parameters of each of the multiple expert models are adjusted with the learning goal of making the difference between the classification result output by the fusion model and the true value of the label converge to the first specified range.

[0034] Among them, steps 101 to 103 can be executed iteratively in a loop until the difference between the classification result output by the fusion model and the true value of the label converges to a first specified range, the iterative process can be stopped, and the result model obtained by training can be output.

[0035] In this embodiment, the multiple expert models in the first model include at least two expert models with different network structures, so that different expert models can be used for knowledge learning, which is conducive to identifying samples of different difficulty levels when the number of samples is limited. The fusion model is trained based on the image features extracted from the sample image after the expert model converges, so that the fusion model can dynamically learn how to reasonably use the knowledge learned by multiple expert models based on the knowledge learned by the expert model, so that expert models with different performances make different contributions to the final output classification results, greatly improving the overall performance of the first model. At the same time, this training method can reduce the dependence on the number of sample images, and can also perform efficient learning when the number of sample images is small, thereby obtaining better model performance.

[0036] It is worth noting that in the above and below embodiments of the present application, before training the first model based on the sample image, a data enhancement operation may be further performed on the sample image. The data enhancement operation may include: a data enhancement operation based on a single sample and / or a data enhancement operation based on multiple samples.

[0037] Optionally, the data enhancement operation based on a single sample may include but is not limited to at least one of the following: Random Flipp, Random Crop, Color Jitter, Random Erasing, a combined enhancement strategy based on NAS search (AutoAugment and RandomAugment), etc. Optionally, the data enhancement based on multiple samples may include but is not limited to at least one of the MixUp algorithm and the CutMix algorithm.

[0038] Based on the above data augmentation operation, a small number of sample images can be transformed. On the one hand, the number of sample images can be enriched to provide more data support for the model process. On the other hand, the complexity of the sample images can be increased through the above data augmentation operation, thereby improving the robustness of the trained model.

[0039] In some optional embodiments, the training process of the first model can be an "end-to-end" training process. That is, after the training data is input from the input end of the first model (i.e., the input end of multiple expert models), the final prediction result is output from the output end of the first model (i.e., the fusion model). The prediction error of the model can be obtained by comparing the final prediction result with the annotated label, and back propagation is performed based on the prediction error and the parameters of each model are adjusted until the prediction error converges or reaches the expected effect, and the result model obtained by training is output. In this way, multiple expert models and fusion models can be trained synchronously.

[0040] In some other optional embodiments, the training process of the first model can be a "two-stage" training process. Among them, the first stage is the training stage of multiple expert models, and the second stage is the training stage of the fusion model. In this way, the prediction errors of multiple expert models can be converged or achieve the expected effect through the training of the first stage. After that, the fusion model can be trained based on the image features of the outputs of the trained multiple expert models through the second stage of training until the prediction error of the fusion model converges or achieves the expected effect, and the training result model is output. In this embodiment, training the fusion model based on multiple expert models that have completed training can effectively reduce the number of iterations required for training and improve the efficiency of model training.

[0041] The above-mentioned “two-stage” training process will be exemplarily described below with reference to the accompanying drawings.

[0042] Figure 2 A flowchart of a model training method provided for another exemplary embodiment of the present application.

[0043] like Figure 2 As shown, the method includes:

[0044] Step 201: Obtain a sample image, where the sample image is annotated with a true label value.

[0045] Step 202: input the sample image into a first model; the first model includes: a plurality of expert models, and a fusion model connected with the plurality of expert models; the plurality of expert models include at least two expert models with different network structures.

[0046] Step 203: extract features of the sample image using the multiple expert models to obtain image features corresponding to each of the multiple expert models.

[0047] Step 204: Perform classification operations on the image features corresponding to the multiple expert models through the classifiers corresponding to the multiple expert models to obtain classification results of the multiple expert models.

[0048] Step 205 , adjusting the parameters of each of the multiple expert models with the goal of making the gap between the classification results of each of the multiple expert models and the true value of the label converge to a second specified range.

[0049] Step 206: Obtain image features obtained by extracting features from the sample image after the multiple expert models converge.

[0050] Step 207: using the fusion model, according to the feature fusion parameters of each of the multiple expert models, fuse the image features corresponding to each of the multiple expert models, and perform classification operation processing on the fused features to obtain a first classification result.

[0051] Step 208: adjusting the feature fusion parameters of each of the plurality of expert models with the learning goal of making the difference between the first classification result and the true value of the label converge to a first specified range.

[0052] In this embodiment, steps 202 to 206 are the first stage of training multiple expert models. Steps 202 to 206 can be iteratively executed until the learning objectives of the multiple expert models converge to a specified range. Steps 207 to 209 are the second stage of training the fusion model. The second stage can start training after the learning objectives of the multiple expert models converge to a specified range. Among them, the training process of the second stage can refer to the records of the aforementioned embodiment, and the following section will exemplify the training process of the first stage.

[0053] In the first stage, after the sample image is input into the first model, feature extractors in multiple expert models may be used to extract features from the sample image to obtain image features corresponding to each of the multiple expert models.

[0054] During the training process of the neural network model, the sample images required for training can be divided into three categories according to the difficulty of recognition: "simple", "medium difficulty" and "difficult". Among them, the sample images of the simple category can be a normal display of the subject of the target category, with obvious details and no problems such as occlusion, noise and blur. A simple network structure can achieve a good recognition effect. In the sample images of the medium difficulty category, the subject occupies the foreground part of the picture, but there will be a long distance and unclear texture details. Therefore, appropriate data enhancement and regularization effects are needed to achieve the ideal recognition effect. In the image samples of the difficult category, the subject of the target category is often mixed in the background and occupies a small proportion in the image, so it is more difficult to recognize. Therefore, a model with extremely strong feature expression and regularization capabilities is needed for recognition.

[0055] In the neural network model, networks of different scales and structures have different capabilities for expressing and identifying features. Therefore, in order to meet the training requirements of sample images of different difficulty, expert models with different network structures can be set in the first model. Expert models with different structures are responsible for identifying samples of different difficulty, so as to adapt to the training scenarios with inconsistent sample image quality.

[0056] When the network structure of the expert model is different, the implementation form of the feature extractor in the expert model is also different. For example, the first model includes: expert model 1, expert model 2 and expert model 3. Among them, expert model 1 is a ResNeSt model, expert model 2 is a ResNet model, and expert model 3 is an EfficientNet. Then, the feature extractor in expert model 1 is a feature extraction network in ResNeSt, the feature extractor in expert model 2 is a feature extraction network in ResNet, and the feature extractor in expert model 3 is a feature extraction network in EfficientNet.

[0057] Figure 3 The structure of the first model is shown as follows: Figure 3 As shown, multiple expert modules can share shallow convolutions and can further perform feature extraction based on the feature maps output by the shallow convolutions.

[0058] In this embodiment, if Figure 3 Each expert model shown may also include a classifier. Figure 3 The classifier 1 shown is connected to the expert model 1, the classifier 2 is connected to the expert model 2, and the classifier 3 is connected to the expert model 3. The classifier is used to perform classification operation processing according to image features, thereby outputting the classification result of each expert model for the sample image.

[0059] After obtaining the classification results of each of the multiple expert models, the multiple expert models can be trained according to the difference between the classification results of the multiple expert models and the true value of the label marked on the sample image. During the training process, the learning objectives of the multiple expert models can be determined, and the model parameters of the multiple expert models can be optimized according to the learning objectives. Among them, the learning objectives of the multiple expert models can be: to make the difference between the classification results of each of the multiple expert models and the true value of the label on the sample image converge to the second specified range.

[0060] The optimized model parameters mainly include: parameters of feature extractors in multiple expert models. When the difference between the classification results of the multiple expert models and the true value of the label on the sample image converges to the second specified range, the first stage of the loop iteration process can be stopped, and the trained multiple expert models are output. The model parameters of the output multiple expert models can enable the multiple expert models to extract image features that can accurately classify the image from the input image.

[0061] Among multiple expert models, any expert model includes multiple feature calculation modules ( Figure 3Multiple feature calculation modules constitute the feature extractor of the expert model. Taking the expert model implemented as a convolutional neural network model as an example, each feature calculation module in the expert model may include: a convolution layer, a pooling layer, and a fully connected layer. The number of feature calculation modules contained in the expert model can be called the depth of the expert model.

[0062] In some exemplary embodiments, in order to improve the performance of the expert model, the depth of each expert model can be dynamically set in the first training stage, thereby obtaining a dynamic neural network. In the training and reasoning stages, the dynamic neural network can dynamically adjust its own structure or parameters according to the different input data, so it has better performance than the static model in terms of computational cost and recognition effect. In this embodiment, when training based on the dynamic model, certain structures in the network (such as channels, layers, sub-networks), etc. can be dynamically selected to be activated.

[0063] In this embodiment, for any expert model, when extracting features from a sample image based on the expert model, some feature calculation modules can be randomly selected from the expert model to dynamically set the depth of the expert model. Through the randomly selected some feature calculation modules, feature extraction operations can be performed on the input sample image to obtain image features output by the expert model.

[0064] like Figure 3 As shown in FIG, expert models 1, 2, ... N, respectively, include multiple feature calculation modules. For each expert model, when performing feature extraction operations, some feature calculation modules of the expert model can be randomly skipped based on the Stochastic Depth algorithm. Figure 3 As shown in the figure, a "switch" module can be set for each feature calculation module. During the training process, the switches corresponding to different feature calculation modules can be randomly turned on or off, thereby achieving the effect of dynamically changing the depth of the expert model. This method greatly enhances the regularization effect of the model and effectively prevents the expert model from overfitting.

[0065] It is worth noting that, in the present embodiment, for each feature calculation module in the expert model, the pooling layer in the feature calculation module may be a Blur pooling layer (Blur Pooling), so that the translation invariance of the expert model can be improved based on the Blur pooling layer.

[0066] In this embodiment, based on the method of dynamically changing the depth of the expert model, the amount of calculation in the training process can be saved, making the training more efficient. When the scale of the neural network model is large, the larger-scale neural network can rely on more parameters to obtain stronger expressive power. If a dynamic structure is added to a larger-scale neural network, the same excellent expressive power can be obtained at a lower cost of increasing the amount of calculation, thereby improving the performance of the expert model. In addition, compared with the fixed parameter amount of the static model, the dynamic network dynamic calculation feature enables it to have stronger adaptability on different hardware platforms and computing environment deployments.

[0067] In some exemplary embodiments, when multiple expert models are trained according to the learning objectives of the multiple expert models, the following optional implementations may be used to construct the learning objectives corresponding to the multiple expert models:

[0068] Optionally, the classification loss of each of the multiple expert models can be calculated according to the classification results of each of the multiple expert models and the true value of the label on the sample image. Among them, the classification loss can be calculated based on at least one loss function among the cross entropy loss function, absolute value loss function, log loss function, square loss function, exponential loss function, Hinge loss function and perceptual loss function, and this embodiment does not limit this. Of course, in addition to the loss functions listed above, the classification loss can also be calculated based on other optional loss functions, and this embodiment does not limit this.

[0069] In some optional embodiments, when constructing the classification loss, in order to further optimize the performance of the expert model, the true value of the label on the sample image may be smoothed.

[0070] Optionally, for the label of the i-th category, the operation of smoothing the label l using a label smoothing method can be expressed by the following formula:

[0071]

[0072] Among them, ε represents a very small value, and C1 represents a constant.

[0073] In addition to the above-mentioned smoothing method, in some exemplary embodiments, a soft label (or distilled label) can be generated based on the knowledge distillation method to guide the training process. Optionally, a teacher model (Teacher Model) can be set in the first model during the training phase. The teacher model is used to output a classification result based on the sample image, and the classification result is used as a soft label to guide the training process of the multi-expert model.

[0074] In this embodiment, optionally, a preset teacher model M may be obtained. tThe second classification result obtained by classifying the sample image I; wherein the preset teacher model can be a pre-trained expert model or other optional classification models, which is not limited in this embodiment. According to the second classification result, the true value of the label marked on the sample image is smoothed to obtain a smoothed label l train The above process can be described by the following formula:

[0075] l train =λ*l ls +(1-λ)M t (I) Formula 2

[0076] Here, λ represents a weighting coefficient, and the value of λ can be set according to actual needs, for example, λ=0.5, λ=0.6, etc., which is not limited in this embodiment.

[0077] According to the classification results of the respective expert models and the smoothed labels, the classification losses of the respective expert models are calculated.

[0078] Optionally, the loss of each expert model may include a cross entropy loss based on a soft label, and the calculation process of the cross entropy loss may be shown in the following formula:

[0079] l soft-CE =-l train *log(M t (I)) Formula 3

[0080] Optionally, in addition to the cross entropy loss based on the soft label, the classification loss of each expert model may also include a cosine loss. After superimposing the above cross entropy loss and cosine loss, the classification loss of each expert model can be obtained. The calculated classification loss of the jth expert model can be expressed by the following formula:

[0081] L j = l soft-CE +(1-cosine(l train , M t (I))) Formula 4

[0082] When the first model includes N expert models, the total classification loss of the N expert models can be described by the following formula:

[0083]

[0084] Optionally, in addition to the classification loss, the difference metric loss corresponding to the multiple expert models can be calculated according to the image features corresponding to the multiple expert models. The difference metric loss is used to measure the difference between the multiple expert models, thereby supervising the multiple expert models to learn different knowledge and making full use of the performance advantages of multiple experts.

[0085] In some optional embodiments, when calculating the difference metric loss corresponding to multiple expert models, the average features extracted by the multiple expert models can be calculated according to the image features output by each of the multiple expert models; then, the similarity between the features output by each of the multiple expert models and the average features is calculated to obtain the difference metric loss corresponding to the multiple expert models. Optionally, the similarity can be calculated using KL (Kullback-Leibler) divergence (or relative entropy loss).

[0086] The above calculation difference metric loss L d The process can be expressed by the following formula:

[0087]

[0088]

[0089] Among them, p j represents the image features output by the jth expert model, N represents the number of expert models included in the first model, and p represents the average features of multiple expert models.

[0090] According to the classification loss and difference measurement loss of each of the multiple expert models, the learning objectives of the multiple expert models can be determined. The learning objectives can be described by the following formula:

[0091] L=L MoE +L d Formula 8

[0092] According to the learning objectives of the multiple expert models, the parameters of the multiple expert models can be adjusted until the learning objectives of the multiple expert models converge to the second specified range.

[0093] After the model training operations based on the above-mentioned embodiments make the multiple expert models converge, the above-mentioned steps 207 to 209 can be executed to train the fusion model. The feature fusion parameters of the multiple expert models to be learned in the fusion model can be implemented as the feature weighting coefficients of the multiple expert models, which are used to allocate the contribution of the image features extracted by the multiple expert models to the final classification results. Among them, the fusion model fuses the image features corresponding to the multiple expert models according to the feature fusion parameters of the multiple expert models, and performs classification operations on the fused features. The process can be described by the following formula:

[0094] M fusion =Linear(contact(E1,E2,...,E N ), C2) Formula 9

[0095] In the above formula, M fusion Represents the classification results output by the fusion model, E1, E2, ..., E D It represents the image features extracted by each of the N expert models, contact() represents the feature fusion calculation, Linear() represents the linear operation used for classification, and C2 is a constant.

[0096] In addition to the model training method described in the foregoing embodiments, the present application also provides an image classification method, which will be exemplarily described below.

[0097] Figure 4 A flowchart of an image classification method provided by an exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes:

[0098] Step 401: input the image to be processed into a first model; the first model includes: a plurality of expert models and a fusion model connected with the plurality of expert models; the plurality of expert models include at least two expert models with different network structures.

[0099] Step 402: extract features of the sample image using the multiple expert models to obtain image features corresponding to each of the multiple expert models.

[0100] Step 403: Perform fusion processing on the image features corresponding to each of the multiple expert models through the fusion model according to the feature fusion parameters of each of the multiple expert models.

[0101] Step 404: Perform classification operation processing on the fused features through the fusion model to obtain a first classification result.

[0102] Among them, the training process of the first model can refer to the records of the aforementioned embodiments, and will not be repeated here. In different application scenarios, the implementation form of the image to be processed may be different. For example, in some embodiments, the image to be processed may include: an image obtained by photographing the target place. Among them, the target place may be a place to be classified, a place to be predicted for potential risks, and so on, for example, it may include but is not limited to public places such as stations, carriages, and venues. In other embodiments, the image to be processed may include: an image to be analyzed for risk content. Among them, the images to be analyzed for risk content may include but are not limited to pornographic images, violent images, and so on.

[0103] Among them, the feature fusion parameters of each of the multiple expert models can be the feature weighting coefficients of each of the multiple expert models. The feature weighting coefficients of each of the multiple expert models can be dynamically learned during the model training process. In some exemplary embodiments, before the image features corresponding to each of the multiple expert models are fused according to the feature fusion parameters of each of the multiple expert models, the feature fusion parameters of each of the multiple expert models are dynamically learned according to the image features obtained by extracting features of the sample images after convergence of the multiple expert models and the true values ​​of the labels marked on the sample images.

[0104] The fusion model fuses the image features corresponding to the multiple expert models, and performs classification operations on the fused features. The operation of obtaining the first classification result can be as shown in the above formula 9 and will not be repeated here.

[0105] In this embodiment, the multiple expert models in the first model include at least two expert models with different network structures, so that the different classification performances of different expert models can be fully utilized, reducing the dependence on the number of sample images. At the same time, the feature fusion parameters of the multiple expert models in the fusion model are dynamically learned, so that the expert models with different performances can make different contributions to the final output classification results dynamically according to the actual training results of different expert models, greatly improving the accuracy of image classification results.

[0106] Figure 5 A flowchart of an image classification method provided by another exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes:

[0107] Step 501: respond to a call request from a client to a first interface, and obtain an image to be processed contained in the interface parameters.

[0108] Step 502: input the image to be processed into a first model; the first model includes: a plurality of expert models and a fusion model connected with the plurality of expert models; the plurality of expert models include at least two expert models with different network structures.

[0109] Step 503: extract features of the sample image using the multiple expert models to obtain image features corresponding to each of the multiple expert models.

[0110] Step 504: Through the fusion model, according to the dynamically learned feature fusion parameters of the multiple expert models, the image features corresponding to the multiple expert models are fused, and the fused features are classified to obtain a first classification result.

[0111] Step 505: Return the first classification result to the client.

[0112] The execution subject of this embodiment may be a server device, such as a conventional server or a cloud server, wherein the client may be implemented as a mobile phone, computer, tablet computer or other device on the user side.

[0113] In this embodiment, the image classification method provided by the aforementioned embodiments can be encapsulated as a software tool that can be used by a third party, such as a SaaS (Software-as-a-Service) tool. Among them, the SaaS tool can be implemented as a plug-in or an application. The plug-in or application can be deployed on a server-side device, and a specified interface can be opened to third-party users such as a client. For ease of description, in this embodiment, the specified interface is described as a first interface. Furthermore, third-party users such as a client can easily access and use the mobile object detection service provided by the server-side device by calling the first interface.

[0114] For example, in some scenarios, the SaaS tool may be deployed on a cloud server, and a third-party user may call a first interface provided by the cloud server to use the SaaS tool online. When the third-party user calls the first interface, the image to be classified may be provided to the SaaS tool by configuring the interface parameters of the first interface. Optionally, the image to be classified may include any frame image in the video data obtained by continuous shooting, or any frame image in the image sequence obtained by discrete shooting, which is not limited in this embodiment.

[0115] After receiving the call request for the first interface, the SaaS tool can obtain the image to be classified provided by the client by parsing the interface parameters of the first interface. After the SaaS tool identifies the category to which the image belongs based on the first model deployed on the server, it can return the category to which the image belongs to the client through the first interface or other communication methods. The training process and reasoning process of the first model can refer to the records of the aforementioned embodiment, which will not be repeated here.

[0116] In this embodiment, the server device can provide image classification services to the client based on the SaaS tool running on it, and the client user can use the image classification service provided by the server device by calling the interface provided by the SaaS tool. Based on the interaction between the client and the server device, the client can completely hand over the image classification operation to the server device, and then, with the help of the powerful computing power and reliable image classification algorithm of the server device, low-cost and high-accuracy image classification operations can be achieved.

[0117] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The sequence numbers of the operations, such as 401, 402, etc., are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. The execution subjects of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices. For example, the execution subject of steps 201 to 204 can be device A; for another example, the execution subject of steps 201 and 202 can be device A, and the execution subject of steps 203 and 204 can be device B; and so on. In addition, these processes may include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0118] The image classification method provided in the above examples of this application can be applied to a variety of risk content and risky place identification scenarios based on image classification, such as image-based terrorism identification scenarios, image-based pornography identification scenarios, etc. In such risk identification scenarios, it is usually difficult to collect enough training samples in a short period of time to meet model training. In the case of insufficient training data, a model that meets the risk content identification needs can be efficiently trained based on the model training method provided in each embodiment of this application. The following will be exemplified in conjunction with specific application scenarios.

[0119] In some application scenarios, it is necessary to identify and warn of violent / terrorist incidents in target places (such as public places). The public places may include: inside of vehicles, shopping malls, waiting rooms of stations, etc. In the above-mentioned places, a risk scene identification system can be deployed based on image acquisition equipment, servers and alarm equipment.

[0120] Among them, the image acquisition device can be implemented as a variety of electronic devices capable of high-definition shooting, including but not limited to electronic devices based on CCD (Charge-coupled Device) image sensors or CMOS (Complementary Metal Oxide Semiconductor) image sensors for imaging, such as high-speed cameras, camcorders, rotating cameras, infrared night vision cameras, etc., which will not be repeated. Among them, the number of image acquisition devices can be one or more, which can be selected according to actual needs, and this embodiment does not limit it. For example, in the waiting room of a railway station, multiple image acquisition devices can be set so that the shooting range of the image acquisition device can cover the waiting room of the railway station.

[0121] The server can be implemented as a conventional server, a cloud server, a cloud host, a virtual center, or other server, and this embodiment does not limit this. The server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture and will not be described in detail.

[0122] The image acquisition device may send a video stream obtained by continuous shooting to a server for processing, or may send an image sequence obtained by interval shooting to a server for processing, which is not limited in this embodiment.

[0123] The communication between the image acquisition device and the server can be carried out by wired communication or wireless communication. The wireless communication method includes short-distance communication methods such as Bluetooth, ZigBee, infrared, WiFi (WIreless-Fidelity, wireless fidelity technology), long-distance wireless communication methods such as LORA, and wireless communication methods based on mobile networks. When connected through mobile network communication, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax, etc., which is not limited in this embodiment.

[0124] After receiving the video stream or discrete image sent by the image acquisition device, the server can classify each frame of the video stream or discrete image based on the first model recorded in the above embodiments. Specifically, after the server obtains the video data obtained by shooting the target place, any frame of the video data can be input into the first model; the first model includes: multiple expert models and a fusion model connected to the multiple expert models; the multiple expert models include at least two expert models with different network structures. The server can extract features of the image through the multiple expert models to obtain the image features corresponding to each of the multiple expert models, and through the fusion model, according to the feature fusion parameters of each of the multiple expert models, the image features corresponding to each of the multiple expert models are fused, and the fusion model is used to classify the fused features to obtain the scene category corresponding to the target place. When the scene category output by the first model indicates that the target place is a high-risk scene, an alarm message can be output based on the alarm device.

[0125] In addition to the image processing scenarios described in the aforementioned embodiments, the model training methods provided in the embodiments of the present application can also be applied to a variety of other different artificial intelligence scenarios. For example, natural language processing scenarios, data analysis scenarios, etc. In natural language processing scenarios, the training data can be implemented as corpus data used for training, such as multiple text corpora or multiple audio corpora. In behavioral data analysis scenarios, the training data can be implemented as multiple behavioral data used for training. In commodity data analysis scenarios, the training data can be implemented as click statistics, sales statistics, etc. for different commodities, and they are no longer listed one by one.

[0126] Figure 6 A schematic diagram of the structure of a server provided for an exemplary embodiment of the present application is shown in FIG. Figure 6 As shown, the server may include: a memory 601 , a processor 602 , a communication component 603 , and a power supply component 604 . Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 Components shown.

[0127] The memory 601 may be configured to store various other data to support operations on the server 12. Examples of such data include instructions for any application or method operating on the server 12, contact data, phone book data, messages, pictures, videos, etc. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0128] In this embodiment, the memory 601 is used to store one or more computer instructions.

[0129] The processor 602 is coupled to the memory 601 and is used to execute one or more computer instructions in the memory 601, so as to: input the image to be processed into the first model through the communication component 603; the first model includes: multiple expert models and a fusion model connected to the multiple expert models; the multiple expert models include at least two expert models with different network structures; the sample image is subjected to feature extraction through the multiple expert models to obtain image features corresponding to each of the multiple expert models; the image features corresponding to each of the multiple expert models are subjected to fusion processing through the fusion model according to the feature fusion parameters of each of the multiple expert models, and the fused features are subjected to classification operation processing to obtain a first classification result.

[0130] Further optionally, before the processor 602 performs fusing processing on the image features corresponding to each of the multiple expert models according to the feature fusion parameters of each of the multiple expert models through the fusion model, it is also used to: dynamically learn the feature fusion parameters of each of the multiple expert models according to the image features obtained by extracting features of the sample images after convergence by the multiple expert models and the true values ​​of the labels marked on the sample images through the fusion model.

[0131] In addition to the aforementioned embodiments, Figure 6 The server shown can also execute the following scene classification method, wherein the processor 602 is mainly used to: obtain video data obtained by shooting a target place; input any frame image in the video data into a first model; the first model includes: multiple expert models and a fusion model connected to the multiple expert models; the multiple expert models include at least two expert models with different network structures; extract features of the image through the multiple expert models to obtain image features corresponding to each of the multiple expert models; fuse the image features corresponding to each of the multiple expert models according to the feature fusion parameters of each of the multiple expert models through the fusion model; and perform classification operation on the fused features through the fusion model to obtain the scene category corresponding to the target place.

[0132] In addition to the aforementioned embodiments, Figure 6 The server shown can also execute the following image classification method, wherein the processor 602 is mainly used for: responding to a call request of the client to the first interface, obtaining the image to be processed contained in the interface parameters; inputting the image to be processed into the first model; the first model includes: multiple expert models and a fusion model connected to the multiple expert models; the multiple expert models include at least two expert models with different network structures; performing feature extraction on the sample image through the multiple expert models to obtain image features corresponding to each of the multiple expert models; performing fusion processing on the image features corresponding to each of the multiple expert models according to the dynamically learned feature fusion parameters of each of the multiple expert models through the fusion model, and performing classification operation processing on the fused features to obtain a first classification result; and returning the first classification result to the client.

[0133] In addition to the aforementioned embodiments, Figure 6The server shown can also execute the following model training method. The first model includes: multiple expert models, and a fusion model connected to the multiple expert models; the multiple expert models include at least two expert models with different network structures. The processor 602 is used to: obtain image features obtained by extracting features from sample images by the multiple expert models after convergence; the sample image is marked with a true label value; through the fusion model, according to the feature fusion parameters of the multiple expert models, the image features corresponding to each of the multiple expert models are fused, and the fused features are classified and processed to obtain a first classification result; with the learning goal of making the gap between the first classification result and the true label value converge to a first specified range, the feature fusion parameters of each of the multiple expert models are adjusted.

[0134] Further optionally, before obtaining the image features obtained by extracting features from the sample images by the multiple expert models after convergence, the processor 602 is also used to: extract features from the sample images by using the multiple expert models to obtain image features corresponding to each of the multiple expert models; perform classification operations on the image features corresponding to each of the multiple expert models by using the classifiers corresponding to each of the multiple expert models to obtain classification results of each of the multiple expert models; and adjust the parameters of each of the multiple expert models with the learning goal of making the gap between the classification results of each of the multiple expert models and the true value of the label converge to a second specified range.

[0135] Further optionally, any expert model among the multiple expert models includes multiple feature calculation modules; when the processor 602 extracts features of the sample image through the multiple expert models to obtain image features corresponding to each of the multiple expert models, it is specifically used to: for any expert model among the multiple expert models, randomly select some feature calculation modules from the expert model to dynamically set the depth of the expert model; perform feature extraction operations on the input sample image through the some feature calculation modules to obtain image features output by the expert model.

[0136] Further optionally, when the processor 602 adjusts the parameters of the multiple expert models with the learning goal of making the gap between the classification results of each of the multiple expert models and the true value of the label converge to a second specified range, it is specifically used to: calculate the classification loss of each of the multiple expert models according to the classification results of each of the multiple expert models and the true value of the label; calculate the difference metric loss corresponding to the multiple expert models according to the image features corresponding to each of the multiple expert models; determine the learning objectives of the multiple expert models according to the classification loss of each of the multiple expert models and the difference metric loss; according to the learning objectives of the multiple expert models, adjust the parameters of each of the multiple expert models until the learning objectives of the multiple expert models converge to the second specified range.

[0137] Further optionally, when the processor 602 calculates the classification loss of each of the multiple expert models based on the classification results of each of the multiple expert models and the true value of the label, it is specifically used to: obtain a second classification result obtained by classifying the sample image using a preset teacher model; based on the second classification result, smooth the true value of the label marked on the sample image to obtain a smoothed label; and calculate the classification loss of each of the multiple expert models based on the classification results of each of the multiple expert models and the smoothed label.

[0138] Further optionally, when the processor 602 calculates the difference measurement loss corresponding to the multiple expert models according to the image features corresponding to the multiple expert models, it is specifically used to: calculate the average features extracted by the multiple expert models according to the image features corresponding to the multiple expert models; respectively calculate the similarity between the features output by each of the multiple expert models and the average features, to obtain the difference measurement loss corresponding to the multiple expert models.

[0139] Further optionally, the processor 602 is further configured to perform a data enhancement operation on the sample image; the data enhancement operation includes: a data enhancement operation based on a single sample and / or a data enhancement operation based on multiple samples.

[0140] Among them, the communication component 603 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0141] The power supply component 605 provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0142] In this embodiment, the multiple expert models in the first model provided by the server include at least two expert models with different network structures, so that different expert models can be used for knowledge learning, which is conducive to identifying samples of different difficulty levels when the number of samples is limited. The fusion model is trained based on the image features extracted from the sample image after the expert model converges, so that the fusion model can dynamically learn how to reasonably use the knowledge learned by multiple expert models based on the knowledge learned by the expert model, so that expert models with different performances can make different contributions to the final output classification results, greatly improving the overall performance of the first model. At the same time, this training method can reduce the dependence on the number of sample images, and can also perform efficient learning when the number of sample images is small, thereby obtaining better model performance.

[0143] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step of the method embodiment that can be executed by the server in the above method embodiment.

[0144] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0149] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0150] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0152] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that: The first model to be trained includes: a plurality of expert models, and a fusion model connected to the plurality of expert models; the plurality of expert models include at least two expert models with different network structures; the method includes: Extracting features of the sample image using multiple expert models to obtain image features corresponding to each of the multiple expert models; By using the classifiers corresponding to the multiple expert models, the image features corresponding to the multiple expert models are classified and processed to obtain classification results of the multiple expert models; Adjusting the parameters of each of the multiple expert models with the goal of making the gap between the classification results of each of the multiple expert models and the true value of the label converge to a second specified range; Obtaining image features obtained by extracting features of a sample image by the multiple expert models after convergence; the sample image is marked with a true value of a label; By means of the fusion model, according to the feature fusion parameters of the respective expert models, the image features corresponding to the respective expert models are fused, and the fused features are subjected to classification operation processing to obtain a first classification result; Taking the gap between the first classification result and the true value of the label as converging to a first specified range as a learning goal, the feature fusion parameters of each of the multiple expert models are adjusted, and the feature fusion parameters of each of the multiple expert models are located in the fusion model.

2. The method according to claim 1, characterized in that Among the multiple expert models, any expert model includes multiple feature calculation modules; Extracting features of the sample image using the multiple expert models to obtain image features corresponding to the multiple expert models respectively includes: For any expert model among the multiple expert models, randomly select some feature calculation modules from the expert model to dynamically set the depth of the expert model; The partial feature calculation module performs a feature extraction operation on the input sample image to obtain the image features output by the expert model.

3. The method according to claim 1, characterized in that The learning objective is to make the gap between the classification results of each of the multiple expert models and the true value of the label converge to a second specified range, and to adjust the parameters of each of the multiple expert models, including: Calculating the classification losses of the multiple expert models according to the classification results of the multiple expert models and the true value of the label; Calculating the difference metric losses corresponding to the multiple expert models according to the image features corresponding to the multiple expert models respectively; Determining learning objectives of the multiple expert models according to the classification losses of the multiple expert models and the difference metric loss; According to the learning objectives of the multiple expert models, the parameters of the respective expert models are adjusted until the learning objectives of the multiple expert models converge to the second specified range.

4. The method according to claim 3, characterized in that Calculating the classification losses of the multiple expert models according to the classification results of the multiple expert models and the true value of the label, including: Obtaining a second classification result obtained by classifying the sample image using a preset teacher model; According to the second classification result, smoothing the true value of the label marked on the sample image to obtain a smoothed label; According to the classification results of the respective expert models and the smoothed labels, the classification losses of the respective expert models are calculated.

5. The method according to claim 3, characterized in that: Calculating the difference metric losses corresponding to the multiple expert models according to the image features corresponding to the multiple expert models respectively includes: Calculating average features extracted by the multiple expert models according to the image features corresponding to the multiple expert models; The similarities between the features output by each of the multiple expert models and the average feature are respectively calculated to obtain difference metric losses corresponding to the multiple expert models.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Performing a data enhancement operation on the sample image; The data enhancement operation includes: a data enhancement operation based on a single sample and / or a data enhancement operation based on multiple samples.

7. An image classification method, characterized in that: include: Inputting the image to be processed into the first model; The first model includes: a plurality of expert models and a fusion model connected with the plurality of expert models; the plurality of expert models include at least two expert models with different network structures; Extracting features from the image using the multiple expert models to obtain image features corresponding to each of the multiple expert models; By means of the fusion model, according to the feature fusion parameters of the respective expert models, the image features corresponding to the respective expert models are fused; The fused features are subjected to classification operation processing through the fusion model to obtain a first classification result, wherein the first model is trained according to the model training method of claim 1.

8. The method according to claim 7, characterized in that Before fusing the image features corresponding to the plurality of expert models respectively according to the feature fusion parameters of the plurality of expert models respectively, the method further comprises: Through the fusion model, feature fusion parameters of each of the multiple expert models are dynamically learned according to image features obtained by extracting features of the sample image after convergence of the multiple expert models and true values ​​of labels marked on the sample image.

9. A scene classification method, characterized in that: include: Acquire video data obtained by shooting a target location; Input any frame image in the video data into the first model; The first model includes: a plurality of expert models and a fusion model connected with the plurality of expert models; the plurality of expert models include at least two expert models with different network structures; Extracting features from the image using the multiple expert models to obtain image features corresponding to each of the multiple expert models; By means of the fusion model, according to the feature fusion parameters of the respective expert models, the image features corresponding to the respective expert models are fused; The fused features are classified and processed by the fusion model to obtain the scene category corresponding to the target place; wherein the first model is trained according to the model training method of claim 1.

10. An image classification method, characterized in that: include: Responding to the call request of the client to the first interface, obtaining the image to be processed contained in the interface parameters; Inputting the image to be processed into a first model; The first model includes: a plurality of expert models and a fusion model connected with the plurality of expert models; the plurality of expert models include at least two expert models with different network structures; Extracting features from the image using the multiple expert models to obtain image features corresponding to each of the multiple expert models; By means of the fusion model, according to the feature fusion parameters of the respective expert models, the image features corresponding to the respective expert models are fused; Performing classification operation processing on the fused features through the fusion model to obtain a first classification result; The first classification result is returned to the client, wherein the first model is trained according to the model training method of claim 1.

11. A server, characterized in that: include: Memory, processor, and communication components; The memory is used to store one or more computer instructions; The processor is used to execute one or more computer instructions to perform the steps in the method according to any one of claims 1-10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps in the method according to any one of claims 1 to 10 can be implemented.

Images