Data-aware hybrid expert model training method for multi-scenario merged datasets

Abstract

The invention discloses a data-aware hybrid expert model training method for a multi-scene merged data set, relates to the field of artificial intelligence technology, and comprises the following steps: collecting and preprocessing data of multiple scenes to construct a multi-scene merged data set; constructing a hybrid model: the hybrid model adopts a transformer-based end-to-end DINO structure, and replaces the FFN layer in the decoder of the DINO structure with an MOE layer, wherein the MOE layer comprises a plurality of independently set expert networks, each of which is an independent neural network for processing the features of a specified data set; constructing a foreground target loss function of the expert network; setting a routing mechanism for routing the token of each scene data set to the corresponding expert network; inputting the multi-scene merged data set into the hybrid model to train each expert network; and the model training method improves the detection accuracy of the hybrid model under the multi-scene data set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data-aware hybrid expert model training method for multi-scenario merged data sets. Background Art

[0002] The field of computer vision has traditionally focused on training and evaluating detection models on specific datasets. These models have limited adaptability to new environments, resulting in reduced accuracy and usefulness in real-world scenarios. Recognizing the challenges posed by single-dataset models and the availability of diverse datasets collected over time, the industry is developing unified models that incorporate information from multiple datasets into a single framework.

[0003] The process of dataset merging presents several challenges that need to be addressed. First, these datasets are collected at different times and for different purposes, resulting in differences across domains. Second, the distribution of training images and tails may be imbalanced across datasets. This means that some datasets and categories may have a large number of training samples, while other datasets and categories may have only a few training samples.

[0004] The traditional approach is to set up separate detection heads on a public backbone network to process different datasets, but this approach will lead to a significant increase in model parameters, pushing up model training and inference costs, and also fail to fully utilize the complementarity between different datasets. Summary of the invention

[0005] Based on the technical problems existing in the background technology, the present invention proposes a data-aware hybrid expert model training method for multi-scene merged data sets to improve the detection accuracy of the hybrid model under multi-scene data sets.

[0006] The data-aware hybrid expert model training method for multi-scenario merged data sets proposed in the present invention comprises the following steps:

[0007] Step 1: Collect and preprocess data from multiple scenes to build a multi-scene merged data set;

[0008] Step 2: Build a hybrid model: The hybrid model adopts the transformer-based end-to-end DINO structure, and replaces the FFN layer in the decoder of the DINO structure with the MOE layer. The MOE layer includes multiple independently set expert networks, each of which is an independent neural network for processing the features of a specified data set.

[0009] Step 3: Construct the prospect target loss function of the expert network;

[0010] Step 4: Set up a routing mechanism to route the token of each scene dataset to the corresponding expert network;

[0011] Step 5: Input the multi-scenario merged dataset into the hybrid model to train each expert network, so as to achieve accurate result output of the hybrid model for the multi-scenario merged dataset.

[0012] Furthermore, during the hybrid model training process, the number of scene datasets in the multi-scene merged dataset is consistent with the number of expert networks, and one scene dataset is allocated to each expert network.

[0013] Furthermore, in step 3, the construction process of the foreground target loss function is as follows:

[0014] Let x∈R D Indicates the input token. represents a set of expert networks, E is the number of expert networks, i is the index of the expert network, and W r ∈R E×D represents the gate function variable. Then each expert network e i The probability p i for: where g x =W r ·x, g x Represents the token that has passed the gating function, and i is the jth expert network.

[0015] Based on the probability p i Calculate the output y of the expert network, y = ∑ i∈topk p i (x)e i (x), topk is the index of the first k expert models in descending order of probability;

[0016] Expert Network i The importance loss function L importance The formula is as follows:

[0017] in Im is the input image, each output image is divided into m parts, each part is a token, I is the probability of the input image, Var is the variance, and Mean is the mean;

[0018] For expert networks i Load L i , with probability p i The cumulative distribution function Φ(·) is sampled at (x) to obtain Φ(p i (x)), and sum all the tokens in the input image Im, load L i represents the number of assignments between experts, and the load loss L load It can be expressed as: in

[0019] Foreground target loss function L load-balancing The formula is as follows:

[0020]

[0021] Furthermore, in step 4, the routing mechanism is specifically set as follows:

[0022] Assume that the multi-scene dataset is And x as input token belongs to the mth scene dataset d m ;

[0023] Construct a mapping function h:{D}→{E} to assign a specific expert network to each scene dataset;

[0024] Computational scenario dataset selection expert i The logical value of probability log(p i (x)), and use the logical value and the target label h(d m ) performs cross entropy operation to obtain the auxiliary loss function of the routing mechanism: h(d m ) is the scene dataset d m The target expert network of the input x.

[0025] Furthermore, after the hybrid model is trained, the parameters and routing mechanism of the hybrid model are optimized through testing. The testing process is as follows:

[0026] 10% of the sample data is extracted from each scene data set in the multi-scene merged data set in step 1 and merged to form a few-sample scene data set to test the hybrid model;

[0027] Datasets with the same labels but belonging to different scenarios are selected and merged to test the hybrid model for performance domain adaptation comparison test;

[0028] A comparative test of different label sets in the same scene is performed, that is, data sets in the same scene but with different label sets are selected and merged to perform a performance comparison test on the hybrid model.

[0029] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data-aware hybrid expert model training method as described above when executing the computer program.

[0030] A computer-readable storage medium stores a plurality of classification programs, wherein the plurality of classification programs are used to be called by a processor and execute the data-aware hybrid expert model training method as described above.

[0031] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0032] The advantages of the data-aware hybrid expert model training method for multi-scenario merged data sets provided by the present invention are: the data set-specific features are separated in the MoE layer, which promotes better expert utilization, effectively avoiding the high training and reasoning cost problems of traditional practices, and improving the detection accuracy of the hybrid model under multi-scenario data sets. MoE is proposed not only as a model parameter scalability tool, but also as an effective learner for processing mixed data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the structure flow of the present invention;

[0034] Figure 2 Schematic diagram constructed for the MoE layer;

[0035] Figure 3 Schematic diagram of the mapping between scene dataset and expert network. DETAILED DESCRIPTION

[0036] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0037] like Figures 1 to 3 As shown, the data-aware hybrid expert model training method for multi-scenario merged data sets proposed in the present invention includes the following steps:

[0038] Step 1: Collect and preprocess data from multiple scenes to build a multi-scene merged data set;

[0039] Collect and organize data sets of multiple scenarios. Preprocess each scenario data set, including data cleaning, labeling, and format conversion, to ensure that they are suitable for hybrid model training. Here, we use data sets of four mine scenarios, namely automatic ore placement, driving without people, personnel leaving their posts, and unmanned driving; the scene images are different.

[0040] Step 2: Build a hybrid model: The hybrid model adopts the transformer-based end-to-end DINO structure, and replaces the FFN layer in the decoder of the DINO structure with the MOE layer. The MOE layer includes multiple independently set expert networks, each of which is an independent neural network for processing the features of a specified data set.

[0041] This embodiment improves the performance of the hybrid model in a multi-dataset environment by identifying and utilizing the unique features present in different scene datasets. The core idea is to use the "Mixture of Experts" (MoE) architecture to train multiple expert networks to process the features of different scene datasets and integrate the knowledge of these experts in a single model.

[0042] MoE has been used as a model expansion tool in natural language processing and image classification problems, as a means to significantly increase the number of model parameters without increasing the amount of computation. It splits the large model into multiple small models (experts), and activates a part of the experts for calculation according to the sample in each iteration, thus saving computing resources; and introduces a gate (router) mechanism that is trainable and ensures sparsity to ensure the optimization of computing power.

[0043] The MOE (mixture of experts) of this embodiment is not only a tool for expanding the capacity of the hybrid model. MoE is an effective and efficient solution for building a general model suitable for scene dataset mixing. Based on the basic MoE, this embodiment introduces a novel dataset-aware hybrid expert structure that can learn the characteristics of a specific dataset at the MoE layer and summarize the information at the non-MoE layer.

[0044] A MoE layer (mixed experts) consists of two parts: (1) a set of expert networks (experts) distributed on different GPUs, and (2) a gating function (i.e., router) that determines the target GPU for each input token and distributes the tokens to their target expert networks. In addition, the number of scene datasets in the multi-scene merge dataset is consistent with the number of expert networks, and one scene dataset is assigned to each expert network.

[0045] Step 3: Construct the foreground target loss function of the expert network, as follows (a1) to (a5):

[0046] (a1) Let x∈R D Indicates the input token. represents a set of expert networks, E is the number of expert networks, i is the index of the expert network, and W r ∈R E×D represents the gate function variable. Then each expert network e i The probability p i for: where g x =W r ·x, g x represents the token after the gating function, and j is the jth expert network.

[0047] (a2) Based on probability p i Calculate the output y of the expert network, where y is the weighted combination of the processed token and its selection probability, y = ∑ i∈topk p i (x)e i (x), topk is the index of the first k expert models in descending order of probability;

[0048] (a3) Expert Network i The importance loss function L importance The formula is as follows:

[0049] in Im is the input image, each output image is divided into m parts, each part is a token, x is the input token corresponding to Im, I is the probability of the input image, Var is the variance, and Mean is the mean;

[0050] (a4) Let the normal distribution be η(0,σ 2 I), where The cumulative distribution function (CDF) of the normal distribution is denoted as Φ(·). For the expert network e i Load L i , with probability p i The cumulative distribution function Φ(·) is sampled at (x) to obtain Φ(p i (x)), and sum all the tokens in the input image Im, load L i represents the number of assignments between experts, and the load loss L load It can be expressed as: in

[0051] (a5) Foreground target loss function L load-balancing The formula is as follows:

[0052]

[0053] Step 4: Set up a routing mechanism to route the token of each scene dataset to the corresponding expert network;

[0054] The specific settings of the routing mechanism are as follows (b1) to (b3):

[0055] (b1), let the multi-scene dataset be And x as input token belongs to the mth scene dataset d m ;

[0056] (b2) Construct a mapping function h:{D}→{E} to m Assign a specific network of experts i ;

[0057] (b3), calculation scene dataset selection expert e i The logical value of probability log(p i (x)), and use the logical value and the target label h(d m ) performs cross entropy operation to obtain the auxiliary loss function of the routing mechanism: h(d m ) is the scene dataset d m The target expert network of the input x, the routing mechanism can be obtained.

[0058] Step 5: Input the multi-scenario merged dataset into the hybrid model to train each expert network, so as to achieve accurate result output of the hybrid model for the multi-scenario merged dataset.

[0059] Through steps one to five, this embodiment proposes a model training method for the mixed problem of multi-scene data sets, which separates the specific features of the scene data set in the MoE layer, promotes the use of better expert networks, and can effectively avoid the high training and reasoning cost problems of traditional methods, and improve the detection accuracy of the mixed model under multi-scene data sets.

[0060] As an embodiment

[0061] S1, build a multi-scenario merged dataset;

[0062] S2, build a hybrid model;

[0063] In the hybrid model, a MoE layer containing multiple expert networks is constructed. Each expert network is an independent neural network responsible for processing the features of a specific data set. The hybrid model adopts the transformer-based end-to-end DINO structure. The backbone network is the pre-trained swin_T, and its 4 scale features are used as input. For hyperparameters, this embodiment uses a 6-layer Transformer encoder and a 6-layer Transformer decoder, and sets the hidden feature dimension to 256. The construction of the MOE layer is as follows: Figure 2 As shown, the FFN layer in the decoder of DINO is replaced by the MoE layer (including n expert networks, n=4 in this example).

[0064] S3, construct the prospect target loss function of the expert network;

[0065] S4. Set up routing mechanism;

[0066] It is used to route the token of each scene dataset to its corresponding expert. This is achieved based on the dataset label, that is, a mapping function h:{D}→{E} is constructed to route the token of each scene dataset d to its corresponding expert. m Assign a specific network of experts i . Here is the auxiliary loss function: Ready-to-use expert network i The logical value of probability log(p i (x)) and the target label h(d m )(i.e. from the scene dataset d m The target expert network of the input x performs cross entropy operation.

[0067] S5. During the training phase, the hybrid model receives data from different scene datasets and routes the data to the corresponding expert network according to the routing mechanism. Each expert network is trained according to the scene dataset it is responsible for, and learns the features and knowledge specific to the scene dataset. By integrating the knowledge of all expert networks, the hybrid model can achieve effective understanding and processing of multi-scene merged datasets.

[0068] like Figure 3 Area ① shows the scene data sets of four scenarios, namely, automatic ore placement (red frame), driving without people (blue frame), personnel leaving their posts (yellow frame), and unmanned driving (green frame). After preprocessing by S11, we get Figure 3 The dataset of the mixed scenario shown in area ②. Then, according to the routing mechanism of S4, automatic ore placement (red box) is assigned to expert 1, driving without people (blue box) is assigned to expert 2, personnel leaving their posts (yellow box) is assigned to expert 3, and unmanned driving (green box) is assigned to expert 4.

[0069] S6. During the model tuning and testing process, the model parameters and routing mechanism are continuously optimized to improve the performance and generalization ability of the model. Here, the capacity factor f is set to 1.25, the auxiliary expert network balance loss weight is 0.1, and the top-1 expert network is selected. One expert is retained on each GPU and trained on 4 RTX4090 GPUs, with a batch_size of 2 for each GPU. The learning rate is set to 1.4e-4. The performance of the model is evaluated using a mixed dataset detection benchmark. By comparing indicators such as the average precision (AP) score, the performance of the hybrid model on the mixed dataset can be measured. The compared models are DINO (without moe structure) and DINO (with moe, non-dataset-aware expert allocation). The comparative tests used here can include the following (1) limited data mixed comparative tests. Here, 10% of the sample data sets are extracted from the two scenarios of automatic mining and driving without people and then merged; (2) Domain adaptation comparison test, this embodiment merges the two scenario data sets of driving without people and unmanned driving; (3) Comparative test of mixing different label sets in the same field, this embodiment merges the two scenario data sets of personnel leaving their posts and unmanned driving.

[0070] In summary, this embodiment provides a data-aware hybrid expert model training method for multi-scenario merged datasets, which separates dataset-specific features within the MoE layer, promotes better expert utilization, and can effectively avoid the high training and reasoning cost problems of traditional practices, and improve the detection accuracy of hybrid models under multi-scenario datasets. MoE is proposed not only as a model parameter scalability tool, but also as an effective learner for processing mixed datasets.

[0071] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data-aware hybrid expert model training method for multi-scenario merged data sets, characterized in that: The steps include: Step 1: Collect and preprocess data from multiple scenes to build a multi-scene merged data set; Step 2: Build a hybrid model: The hybrid model adopts the transformer-based end-to-end DINO structure, and replaces the FFN layer in the decoder of the DINO structure with the MOE layer. The MOE layer includes multiple independently set expert networks, each of which is an independent neural network for processing the features of a specified data set. Step 3: Construct the prospect target loss function of the expert network; Step 4: Set up a routing mechanism to route the token of each scene dataset to the corresponding expert network; Step 5: Input the multi-scenario merged dataset into the hybrid model to train each expert network, so as to achieve accurate result output of the hybrid model for the multi-scenario merged dataset.

2. The data-aware hybrid expert model training method for multi-scenario merged data sets according to claim 1 is characterized in that: During the hybrid model training process, the number of scene datasets in the multi-scene merged dataset is consistent with the number of expert networks, and one scene dataset is assigned to each expert network.

3. The data-aware hybrid expert model training method for multi-scenario merged data sets according to claim 1 is characterized in that: In step 3, the construction process of the foreground target loss function is as follows: Let x∈R D Indicates the input token. represents a set of expert networks, E is the number of expert networks, i is the index of the expert network, and W r ∈R E×D represents the gate function variable. Then each expert network e i The probability p i for: where g x =W r ·x, g x represents the token after the gating function, and j is the jth expert network. Based on the probability p i Calculate the output y of the expert network, y = ∑ i∈topk p i (x)e i (x), topk is the index of the first k expert models in descending order of probability; Expert Network i The importance loss function L importance The formula is as follows: in Im is the input image, each output image is divided into m parts, each part is a token, I is the probability of the input image, Vsr is the variance, and Mean is the mean; For expert networks i Load L i , with probability p i The cumulative distribution function Φ(·) is sampled at (x) to obtain Φ(p i (x)), and sum all the tokens in the input image Im, load L i represents the number of assignments between experts, and the load loss L load It can be expressed as: in Foreground target loss function L load-balancing The formula is as follows:

4. The data-aware hybrid expert model training method for multi-scenario merged data sets according to claim 1, characterized in that: In step 4, the routing mechanism is specifically set up as follows: Assume that the multi-scene dataset is And x as input token belongs to the mth scene dataset d m ; Construct a mapping function h:{D}→{E} to assign a specific expert network to each scene dataset; Computational scenario dataset selection expert i The logical value of probability log(p i (x)), and use the logical value and the target label h(d m ) performs cross entropy operation to obtain the auxiliary loss function of the routing mechanism: h(d m ) is the scene dataset d m The target expert network of the input x.

5. The data-aware hybrid expert model training method for multi-scenario merged data sets according to claim 1, characterized in that: After the hybrid model is trained, the parameters and routing mechanism of the hybrid model are optimized through testing. The testing process is as follows: 10% of the sample data is extracted from each scene data set in the multi-scene merged data set in step 1 and merged to form a few-sample scene data set to test the hybrid model; Datasets with the same labels but belonging to different scenarios are selected and merged to test the hybrid model for performance domain adaptation comparison test; A comparative test of different label sets in the same scene is performed, that is, data sets in the same scene but with different label sets are selected and merged to perform a performance comparison test on the hybrid model.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the data-aware hybrid expert model training method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the data-aware hybrid expert model training method as described in any one of claims 1-5.

Images