Multi-target detection method and device, electronic equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-03-23
- Publication Date
- 2026-06-02
Smart Images
Figure CN116385850B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a multi-target detection method, apparatus, electronic device, and storage medium. Background Technology
[0002] Object detection has become a research hotspot in both theory and application in recent years. It is an important branch of data processing and computer vision, a core component of intelligent monitoring systems, and a fundamental algorithm in the broader field of identity recognition, playing a crucial role in subsequent tasks such as face recognition, gait recognition, crowd counting, and instance segmentation. Thanks to the development of deep learning CNN network architectures, the performance of object detection tasks has gradually improved. However, existing object detection frameworks are designed for fully labeled supervised learning models. For semi-labeled and unlabeled datasets, existing frameworks struggle to extract effective information and train them effectively. In practical applications, object detection tasks involve changes in the target domain or target category, and place more stringent demands on model size and inference speed. Summary of the Invention
[0003] In view of the above, it is necessary to propose a multi-target detection method, device, electronic equipment and storage medium that can improve the accuracy of multi-target detection.
[0004] A first aspect of the present invention provides a multi-target detection method, the method comprising:
[0005] Initialize the first network structure of the student model and the second network structure of the teacher model;
[0006] The number of samples for a single category in batch training is determined based on a given sample dataset of multiple categories, and multiple training sample data are obtained from the given sample dataset of multiple categories based on the number of samples.
[0007] The multiple training sample data are input into the first network structure for training to obtain multiple first predicted category labels, and input into the second network structure for training to obtain multiple second predicted category labels;
[0008] The network parameters of the student model are updated based on the plurality of first predicted category labels and the plurality of second predicted category labels;
[0009] The network parameters of the teacher model are updated based on the updated network parameters of the student model.
[0010] When the iterative training termination condition is met, the training process for the student model and the teacher model ends, and a multi-object detector is obtained.
[0011] The target data is detected using the multi-target detector.
[0012] According to an optional embodiment of the present invention, the initialization of the first network structure of the student model and the second network structure of the teacher model includes:
[0013] The first network structure and the second network structure are initialized with the same network structure, wherein the first network structure and the second network structure adopt a shared backbone network;
[0014] The backbone network of the first network structure is connected to multiple first detection branches;
[0015] The backbone network of the second network structure is connected to multiple second detection branches;
[0016] The number of the plurality of first detection branches and the number of the plurality of second detection branches are consistent with the number of categories in the sample dataset, and each of the first detection branches and each of the second detection branches is used to detect one of the categories.
[0017] According to an optional embodiment of the present invention, determining the number of samples for a single class in batch training based on a given sample dataset of multiple classes includes:
[0018] Calculate the number of samples in the sample dataset for each category;
[0019] The minimum sample size is determined based on the multiple sample sizes mentioned above;
[0020] The number of samples is determined based on the minimum number of samples, wherein the number of samples is less than or equal to the minimum number of samples.
[0021] According to an optional embodiment of the present invention, updating the network parameters of the student model based on the plurality of first predicted category labels and the plurality of second predicted category labels includes:
[0022] For each of the first detection branches of the student model, determine whether there is target training sample data corresponding to the first detection branch in the plurality of training sample data;
[0023] When the target training sample data exists, calculate the first loss function value based on the first predicted category label output by the first detection branch and the labeled category label of the target training sample data;
[0024] Obtain multiple remaining sample data from the plurality of training sample data, excluding the target training sample data;
[0025] For each remaining sample data, a second loss function value is calculated based on the second predicted category label output by the second detection branch corresponding to the remaining sample data and the first predicted category label output by the first detection branch corresponding to the remaining sample data.
[0026] The first detection branch is iteratively trained based on the first loss function value and the second loss function value, and the network parameters of the student model are updated.
[0027] According to an optional embodiment of the present invention, the step of iteratively training the corresponding first detection branch based on the first loss function value and the second loss function value, and updating the network parameters of the student model, includes:
[0028] Obtain the gradient of the network parameters of the student model;
[0029] The gradient is compared with a preset gradient threshold to obtain a comparison result;
[0030] The gradient is updated to the target gradient based on the comparison result;
[0031] Calculate the total loss function value based on the first loss function value and the second loss function value;
[0032] The network parameters of the student model are updated based on the total loss function value, the target gradient, and the obtained preset learning rate.
[0033] According to an optional embodiment of the present invention, updating the gradient to the target gradient based on the comparison result includes:
[0034] When the comparison result is that the gradient is greater than the preset gradient threshold, the norm of the gradient is calculated;
[0035] Calculate the ratio of the norm to the preset gradient threshold to determine the scaling factor;
[0036] The target gradient is obtained by multiplying the scaling factor and the gradient.
[0037] According to an optional embodiment of the present invention, updating the network parameters of the teacher model based on the updated network parameters of the student model includes:
[0038] Obtain the first network parameters of the student model after the end of the current batch training, and the second network parameters of the teacher model after the end of the previous batch training;
[0039] Calculate the exponential moving average based on the first network parameters and the second network parameters;
[0040] The network parameters of the teacher model are updated to the exponential moving average when the current batch training ends.
[0041] A second aspect of the present invention provides a multi-target detection device, the device comprising:
[0042] An initialization module is used to initialize the first network structure of the student model and the second network structure of the teacher model.
[0043] The acquisition module is used to determine the number of samples for a single category in batch training based on a given sample dataset of multiple categories, and to acquire multiple training sample data from the given sample dataset of multiple categories based on the number of samples.
[0044] The training module is used to input the multiple training sample data into the first network structure for training to obtain multiple first predicted category labels, and input the data into the second network structure for training to obtain multiple second predicted category labels;
[0045] The first update module is used to update the network parameters of the student model based on the plurality of first predicted category labels and the plurality of second predicted category labels;
[0046] The second update module is used to update the network parameters of the teacher model based on the updated network parameters of the student model.
[0047] The training module is also used to terminate the training process of the student model and the teacher model when the iterative training termination condition is met, so as to obtain a multi-object detector;
[0048] The detection module is used to detect target data using the multi-target detector.
[0049] A third aspect of the present invention provides an electronic device comprising a processor and a memory, the processor being configured to implement the multi-target detection method by executing a computer program stored in the memory.
[0050] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-target detection method described above.
[0051] The multi-object detection method, apparatus, electronic device, and storage medium provided by this invention treat each detection task as a semi-supervised learning task. In joint training across datasets (datasets of multiple categories), an average teacher algorithm is used. Student and teacher models with identical network structures are employed. In each iteration, the student model updates its network parameters by calculating the loss function value and using gradient descent. The teacher model's network parameters are obtained by calculating an exponential moving average from the student model's network parameters. Training a multi-object detector using multiple datasets of different categories, compared to training multiple different detection models, not only reduces the number of model parameters but also prevents overfitting and improves the model's generalization ability. This invention flexibly utilizes annotations and the teacher model's output as training supervision in different detection branches of the student model, fully leveraging the information in the dataset and improving detection accuracy. Attached Figure Description
[0052] Figure 1 This is a flowchart of the multi-target detection method provided in Embodiment 1 of the present invention.
[0053] Figure 2 This is a schematic diagram of the network structure of the teacher-student model provided in an embodiment of the present invention.
[0054] Figure 3 This is a structural diagram of the multi-target detection device provided in Embodiment 2 of the present invention.
[0055] Figure 4 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation
[0056] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.
[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing an embodiment in one alternative implementation and is not intended to be limiting of the invention.
[0058] The multi-target detection method provided in this embodiment of the invention is executed by an electronic device, and correspondingly, the multi-target detection device operates in the electronic device.
[0059] This invention provides a standardized approach to symptom processing based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.
[0060] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0061] Object detection is one of the most important applications of artificial intelligence in video surveillance. It aims to detect foreground objects in data or video and outline them with bounding boxes. Existing object detection algorithms, after being trained on a dataset with a fixed number of labeled object categories, can quickly and accurately detect all objects belonging to that category set, which has greatly promoted the development of intelligent security monitoring.
[0062] With social development and technological advancements, people are paying increasing attention to home security. Surveillance cameras have gradually moved from public places into homes, leading to more diverse and customized demands in the field of intelligent monitoring, such as monitoring children and pets at home. However, the number of categories that can be labeled in a single dataset is limited. If some target categories are missing from the dataset, manually adding the labeled categories is very labor-intensive.
[0063] The most direct approach is to train different models for different datasets to meet various detection needs. However, the more models there are, the more resources are consumed, and the slower the detection speed becomes, making it unsuitable for scenarios with high real-time requirements. To address this issue, existing technologies employ a shared backbone network and add a detection branch network for each dataset. Each branch is responsible for detecting the target categories labeled in its corresponding dataset. During training, for each detection branch, only the loss function value of the images in its corresponding dataset is calculated. For example, for two datasets labeled pets and humans respectively, two branches are set up to detect pets and humans. When the input is an image from the pet dataset, only the loss function value of the pet branch is calculated, while the human branch is ignored, and vice versa. Compared to multiple independent detection models, this method reduces the number of model parameters, but it doesn't fully utilize the data. For a detection branch, its corresponding dataset is fully labeled, while other datasets are unlabeled. For example, in the pet dataset, humans are unlabeled; and in the human dataset, pets are unlabeled. Directly ignoring the loss function value of unlabeled targets is not the optimal solution.
[0064] Example 1
[0065] Figure 1 This is a flowchart of a multi-target detection method provided in Embodiment 1 of the present invention. The multi-target detection method specifically includes the following steps. Depending on different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.
[0066] S11, initialize the first network structure of the student model and the second network structure of the teacher model.
[0067] Knowledge distillation adopts a teacher-student model: the complex and large model is used as the teacher, while the student model has a simpler structure. The teacher is used to assist in the training of the student model. The teacher has a strong learning ability and can transfer the knowledge it has learned to the student model with a relatively weak learning ability, thereby enhancing the generalization ability of the student model.
[0068] To perform multi-target detection, electronic devices can leverage the concept of knowledge distillation to jointly train teacher-student models, resulting in a detection classifier. This classifier is then used for multi-target detection. Before jointly training the teacher-student models, the electronic device needs to initialize the network structures of both the student and teacher models. For clarity, the student model's network structure is designated as the first network structure, and the teacher model's network structure as the second network structure. Initializing the first and second network structures includes initializing network parameters, such as the student model's first learning rate, the teacher model's second learning rate, the student model's first network weights, and the teacher model's second network weights. The first and second learning rates are pre-set parameters.
[0069] In an optional implementation, the initialization of the first network structure for the student model and the second network structure for the teacher model includes:
[0070] The first network structure and the second network structure are initialized with the same network structure, wherein the first network structure and the second network structure adopt a shared backbone network;
[0071] The backbone network of the first network structure is connected to multiple first detection branches;
[0072] The backbone network of the second network structure is connected to multiple second detection branches.
[0073] like Figure 2 The diagram shows the first network structure of the initialized student model and the second network structure of the teacher model. The student model includes a first backbone network and multiple first detection branches connected to it. Each first detection branch corresponds to a class in the sample dataset, and different first detection branches correspond to different classes in the sample dataset. The teacher model includes a second backbone network and multiple second detection branches connected to it. Each second detection branch corresponds to a class in the sample dataset, and different second detection branches correspond to different classes in the sample dataset. The first and second backbone networks have the same network structure; that is, they are shared networks. For example, the first and second backbone networks can be a ResNet network structure or other network structures, depending on the specific requirements.
[0074] In this model, the number of first detection branches is the same as the number of second detection branches. For example, assuming the sample dataset has N categories, then the number of first detection branches and the number of second detection branches are both N. That is, the first backbone network of the student model connects to N first detection branches, and the second backbone network of the teacher model connects to N second detection branches.
[0075] S12, determine the number of samples for a single category in batch training based on the given sample dataset of multiple categories, and obtain multiple training sample data from the given sample dataset of multiple categories based on the number of samples.
[0076] The electronic device can acquire a sample dataset of multiple categories after initializing the first network structure of the student model and the second network structure of the teacher model; or it can acquire a sample dataset of multiple categories before initializing the first network structure of the student model and the second network structure of the teacher model.
[0077] A given multi-class sample dataset refers to a collection of sample data with multiple classes, where each sample data corresponds to a class and is uniquely labeled with a class label.
[0078] Sample data can include pedestrian images, vehicle images, and text images. When the sample data consists of pedestrian images, each pedestrian represents a category. Multiple pedestrian images are captured using an image acquisition device, and each set of images is labeled with a category label, resulting in multiple pedestrian image sets. When the sample data consists of vehicle images, each brand represents a category. Multiple vehicle images of each brand are captured using an image acquisition device, and each set of images is labeled with a category label, resulting in multiple vehicle image sets. When the sample data consists of text images, each application area represents a category. Multiple text images of each application area are captured using an image acquisition device, and each set of text images is labeled with a category label, resulting in multiple text image sets.
[0079] The electronic device can first determine the number of samples for a single category in batch training, and then randomly select multiple sample data from the given sample dataset of multiple categories as training sample data according to the number of samples. The multiple training sample data obtained shall have at least 2 categories, that is, training sample data of multiple categories are obtained.
[0080] In an optional implementation, after obtaining a sample dataset of a given number of categories, the method further includes:
[0081] The sample datasets of the multiple categories are preprocessed.
[0082] A given sample data set can be called the original sample data set, and the sample data in the original sample data set is the original sample data. After the electronic device obtains the original sample data sets of multiple categories, it preprocesses the original sample data therein, including but not limited to: preprocessing such as scaling, cropping, mirror transformation, normalization, etc., so as to achieve the expansion of the original sample data set, thereby achieving the purpose of enhancing the robustness of the model.
[0083] It should be understood that after preprocessing the sample data sets of the multiple categories, the sample sampling quantity of a single category in batch training is determined according to the preprocessed sample data sets of the multiple categories, and multiple training sample data are obtained in batches from the preprocessed sample data sets of the multiple categories according to the sample sampling quantity.
[0084] In an optional implementation manner, the determining the sample sampling quantity of a single category in batch training according to the given sample data sets of multiple categories includes:
[0085] Calculating the sample quantity of the sample data set of each category;
[0086] Determining the minimum sample quantity according to the multiple sample quantities;
[0087] Determining the sample sampling quantity according to the minimum sample quantity, where the sample sampling quantity is less than or equal to the minimum sample quantity.
[0088] For example, assume that the sample data sets of multiple categories are X = (X1, X2, Xi,..., XC), where Xi is the sample data set of the i-th category, 1 ≤ i ≤ C, C is the number of categories of the sample data set, the total number of sample data in the X set is N, where the number of sample data in Xi is xi, that is, x1 + x2 + x3 +... + xC = N. Among x1, x2,..., xC, the minimum sample quantity is the number of sample data xi in the i-th category of sample data set (xi < xj, j ≠ i), then the sample sampling quantity of each category can be less than or equal to the minimum sample quantity xi. Thus, the sample sampling quantity of each category can be made balanced.
[0089] In each batch training, for each category, samples are respectively sampled from the sample data set of each category according to the sample sampling quantity, a sampling sample set of each category is generated, batch training sample data are generated according to the sampling sample set of each category, and the student model and the teacher model are trained with the batch training sample data until the training is completed.
[0090] By dynamically sampling the sample dataset for each category, the number of samples sampled for each category can be balanced, meaning that each category receives the same or approximately the same number of training samples. This ensures the efficiency and accuracy of the trained model in detecting each category.
[0091] S13, input the multiple training sample data into the first network structure for training to obtain multiple first predicted category labels, and input the data into the second network structure for training to obtain multiple second predicted category labels.
[0092] After acquiring multiple training sample data, these data are simultaneously input into the first network structure of the student model and the second network structure of the teacher model for training. The first network structure of the student model outputs multiple first predicted category labels, and the second network structure of the teacher model outputs multiple second predicted category labels. The first and second predicted category labels include the categories of the training sample data.
[0093] S14, update the network parameters of the student model according to the plurality of first predicted category labels and the plurality of second predicted category labels.
[0094] The electronic device updates the network parameters of the student model based on multiple first predicted category labels obtained from the student model and multiple second predicted category labels obtained from the teacher model.
[0095] In an optional implementation, updating the network parameters of the student model based on the plurality of first predicted category labels and the plurality of second predicted category labels includes:
[0096] For each of the first detection branches of the student model, determine whether there is target training sample data corresponding to the first detection branch in the plurality of training sample data;
[0097] When the target training sample data exists, calculate the first loss function value based on the first predicted category label output by the first detection branch and the labeled category label of the target training sample data;
[0098] Obtain multiple remaining sample data from the plurality of training sample data, excluding the target training sample data;
[0099] For each remaining sample data, a second loss function value is calculated based on the second predicted category label output by the second detection branch corresponding to the remaining sample data and the first predicted category label output by the first detection branch corresponding to the remaining sample data.
[0100] The first detection branch is iteratively trained based on the first loss function value and the second loss function value, and the network parameters of the student model are updated.
[0101] Assuming the number of training sample data obtained is B, for each first detection branch H of the student model i The output, if there exists a data point belonging to dataset D among the B training samples. i If the target training sample data is B, then the first loss function value is calculated based on the labeled category label of the target training sample data and the predicted category label of the first detection branch; the remaining data in the B training sample data do not belong to dataset D. i The training sample data is then used to detect the second detection branch H of the teacher model. i The output is used as pseudo-labels to calculate the second loss function value. All loss function values (i.e., the first and second loss function values) are summed, and then the network parameters of the student model are updated using the gradient descent algorithm. The first loss function value L1 can be calculated using the cross-entropy loss function, L1 = -SUM(y*logy'), where y represents the labeled class label, and y' represents the predicted class label output by the first detection branch. The second loss function value L2 can also be calculated using the cross-entropy loss function, L2 = -SUM(y'*logy”), where y' represents the predicted class label output by the first detection branch, and y” represents the predicted class label output by the second detection branch.
[0102] In practice, the electronic device can sequentially traverse each first detection branch of the student model. For each traversed first detection branch, it determines whether there is target training sample data corresponding to the traversed first detection branch among multiple training sample data. Since each first / second detection branch corresponds to a specified category label, a branch identifier can be set for the corresponding first / second detection branch according to the specified category label. For example, if the category label is "1", the branch identifier "1" can be set for the first first detection branch in the student model and the first second detection branch in the teacher model. In this way, the electronic device can determine whether there is target training sample data corresponding to the traversed first detection branch among multiple training sample data based on the branch identifier and the labeled category label of the sample data. When the labeled category label of the sample data is the same as the branch identifier of the traversed first detection branch, it is determined that there is target training sample data corresponding to the traversed first detection branch among the multiple training sample data. When the labeled category label of the sample data is different from the branch identifier of the traversed first detection branch, it is determined that there is no target training sample data corresponding to the traversed first detection branch among the multiple training sample data.
[0103] For example, assuming that each batch randomly selects one or more samples from each of the N sample datasets as training samples, the category of the obtained training samples is N. When traversing to the first detection branch of the student model (branch identifier H1), if it is possible to obtain sample data labeled with category D1 from the multiple training samples as target training samples, then based on the first predicted category label output by the first detection branch and the labeled category label of the target training samples, the first loss function value L1 = -SUM(y1*logy1') is calculated, where y1 represents the labeled category label of the target training samples and y1' represents the student model. The first detection branch outputs the predicted category label of the target training sample data; the sample data in the multiple training sample data whose category label is not D1 is taken as the remaining sample data. For each remaining sample data, the second loss function value L2 = -SUM(y2'*logy2”) is calculated according to the first predicted category label output by the first detection branch with the same branch identifier and the second predicted category label output by the second detection branch. Here, y2' represents the predicted category label of the remaining sample data output by the j-th second detection branch of the teacher model, and y2” represents the predicted category label of the remaining sample data output by the j-th first detection branch of the student model. The value of j is 2, 3, ..., N.
[0104] When traversing to the second first detection branch of the student model (branch identifier H2), if sample data labeled with category label D2 can be obtained from multiple training sample data as target training sample data, then the first loss function value L1 = -SUM(y1*logy1') is calculated based on the first predicted category label output by the first detection branch and the labeled category label of the target training sample data, where y1 represents the labeled category label of the target training sample data and y1' represents the predicted category label of the target training sample data output by the second first detection branch of the student model; the sample data with labeled category label other than D2 in the multiple training sample data are taken as the remaining sample data. For each remaining sample data, the second loss function value L2 = -SUM(y2'*logy2") is calculated based on the first predicted category label output by the first detection branch with the same branch identifier and the second predicted category label output by the second detection branch, where y2' represents the predicted category label of the remaining sample data output by the j-th second detection branch of the teacher model and y2" represents the predicted category label of the remaining sample data output by the j-th first detection branch of the student model, and j takes the value 1, 3, 4, ..., N.
[0105] This process is repeated for each of the first detection branches of the student model.
[0106] It should be understood that for each first detection branch traversed, a first loss function value and multiple second loss function values are obtained. Based on the first loss function value and multiple second loss function values, the total loss function value is calculated, and the gradient descent algorithm is used to iteratively train the corresponding first detection branch, thereby updating the network parameters of the student model.
[0107] In an optional implementation, the step of iteratively training the corresponding first detection branch based on the first loss function value and the second loss function value, and updating the network parameters of the student model, includes:
[0108] Obtain the gradient of the network parameters of the student model;
[0109] The gradient is compared with a preset gradient threshold to obtain a comparison result;
[0110] The gradient is updated to the target gradient based on the comparison result;
[0111] Calculate the total loss function value based on the first loss function value and the second loss function value;
[0112] The network parameters of the student model are updated based on the total loss function value, the target gradient, and the obtained preset learning rate.
[0113] The gradient of the network parameters can be any of the parameters in the student model or a specified parameter; this invention does not impose any restrictions.
[0114] The preset gradient threshold is set according to actual needs.
[0115] In each batch, the network parameters of the student model are updated based on the relationship between the gradient and the preset gradient threshold. After training in multiple batches, the target student model is determined.
[0116] The gradient is compared with a preset gradient threshold. The comparison results are: gradient greater than the preset threshold, and gradient less than the preset threshold. If the comparison result is that the gradient is greater than the preset threshold, it indicates that the student model's gradient descent is slow and needs to be updated. If the comparison result is that the gradient is less than the preset threshold, it indicates that the student model's gradient descent is moderate and does not need to be updated. It should be understood that the condition of gradient equal to the preset threshold applies to both cases where the gradient is greater than the preset threshold and cases where the gradient is less than the preset threshold.
[0117] In the above implementation method, when updating the network parameters of the student model, it is necessary not only to calculate the total loss function value, but also to include the first learning rate and the comparison result between the gradient and the preset gradient threshold. The gradient is updated by using the relationship between the gradient and the preset gradient threshold, and the student model is trained according to the updated target gradient. This can avoid the gradient explosion problem under a large learning rate, ensure the reliability of the training process, ensure that the student model converges quickly, and thus improve the training efficiency of the student model.
[0118] In some alternative implementations, updating the gradient to the target gradient based on the comparison result includes:
[0119] When the comparison result is that the gradient is greater than the preset gradient threshold, the norm of the gradient is calculated;
[0120] Calculate the ratio of the norm to the preset gradient threshold to determine the scaling factor;
[0121] The target gradient is obtained by multiplying the scaling factor and the gradient.
[0122] The norm corresponding to the gradient can be L0 norm, L1 norm, L2 norm, etc.
[0123] The ratio of the norm to a preset gradient threshold is used as a scaling factor. If the norm is greater than the preset gradient threshold, the scaling factor is greater than 1. The target gradient is obtained by multiplying the scaling factor by the gradient. If the norm is less than the preset gradient threshold, the scaling factor is less than 1. The target gradient is obtained by multiplying the value 1 by the gradient.
[0124] S15, update the network parameters of the teacher model according to the updated network parameters of the student model.
[0125] During the current batch training, once the network parameters of the student model are updated, the network parameters of the teacher model can be updated based on the updated network parameters of the student model.
[0126] In an optional implementation, updating the network parameters of the teacher model based on the updated network parameters of the student model includes:
[0127] Obtain the first network parameters of the student model after the end of the current batch training, and the second network parameters of the teacher model after the end of the previous batch training;
[0128] Calculate the exponential moving average based on the first network parameters and the second network parameters;
[0129] The network parameters of the teacher model are updated to the exponential moving average when the current batch training ends.
[0130] Specifically, a moving average coefficient 'a' is set, and the update process of the network parameters for the current batch of teacher models is as follows: F' t =a*F' t-1 +(1-a)*F t , of which F t F' represents the network parameters after the model update for the t-th batch of students. t This represents the network parameters after the teacher model is updated in the t-th batch.
[0131] S16, When the iterative training termination condition is met, the training process for the student model and the teacher model ends, and a multi-object detector is obtained.
[0132] The termination condition for iterative training can be that the total loss function value is less than a preset loss threshold, or the number of batch training iterations is greater than a preset number of iterations threshold. When the termination condition for iterative training is met, the training process for the student model and the teacher model ends, resulting in the target student model and the target teacher model, which are then used as the final multi-object detector.
[0133] S17, Use the multi-target detector to detect the target data.
[0134] In this context, target data refers to data that requires multi-target detection, such as multiple faces contained in a face image, which are detected using a multi-target detector.
[0135] The target data is simultaneously input into both the target student model and the target teacher model for detection. Multiple first-predicted category labels are output through multiple first-detection branches of the target student model, and multiple second-predicted category labels are output through multiple second-detection branches of the target teacher model. The first-predicted category labels corresponding to the first-detection branches with the same branch identifier and the second-predicted category labels corresponding to the second-detection branches are obtained. If the first-predicted category label and the second-predicted category label are the same, the target predicted category label for the target data is determined to be the first-predicted category label / second-predicted category label. If the first-predicted category label and the second-predicted category label are different, the target predicted category label for the target data is determined to be the predicted category label with the higher prediction probability.
[0136] The multi-object detector trained in the above embodiments can be embedded in a mobile terminal, allowing the mobile terminal to perform object detection on the acquired object data. For example, by installing a corresponding APP on a mobile phone, when object detection is needed, the APP can be accessed, and object data can be acquired through functions such as taking a photo. Then, object detection can be performed on the object data to determine the predicted object category label.
[0137] This invention treats each detection task as a semi-supervised learning task, employing an average teacher algorithm in joint training across multiple datasets (datasets of multiple categories). It uses student and teacher models with identical network structures. In each iteration, the student model updates its network parameters by calculating the loss function and using gradient descent, while the teacher model's network parameters are obtained by calculating an exponential moving average from the student model's network parameters. Training a multi-object detector using multiple datasets of different categories not only reduces the number of model parameters compared to training multiple different detection models, but also prevents overfitting and improves the model's generalization ability. This invention flexibly utilizes annotations and the teacher model's output as training supervision in different detection branches of the student model, fully leveraging the information in the dataset and improving detection accuracy.
[0138] Example 2
[0139] Figure 3 This is a structural diagram of the multi-target detection device provided in Embodiment 2 of the present invention.
[0140] In some embodiments, the multi-target detection device 30 may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the multi-target detection device 30 may be stored in the memory of an electronic device and executed by at least one processor to perform (see details). Figure 1 (Description) The function of multi-target detection.
[0141] In this embodiment, the multi-target detection device 30 can be divided into multiple functional modules according to its functions. These functional modules may include: an initialization module 301, an acquisition module 302, a training module 303, a first update module 304, a second update module 305, and a detection module 306. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.
[0142] The initialization module 301 is used to initialize the first network structure of the student model and the second network structure of the teacher model.
[0143] Knowledge distillation adopts a teacher-student model: the complex and large model is used as the teacher, while the student model has a simpler structure. The teacher is used to assist in the training of the student model. The teacher has a strong learning ability and can transfer the knowledge it has learned to the student model with a relatively weak learning ability, thereby enhancing the generalization ability of the student model.
[0144] To perform multi-target detection, electronic devices can leverage the concept of knowledge distillation to jointly train teacher-student models, resulting in a detection classifier. This classifier is then used for multi-target detection. Before jointly training the teacher-student models, the electronic device needs to initialize the network structures of both the student and teacher models. For clarity, the student model's network structure is designated as the first network structure, and the teacher model's network structure as the second network structure. Initializing the first and second network structures includes initializing network parameters, such as the student model's first learning rate, the teacher model's second learning rate, the student model's first network weights, and the teacher model's second network weights. The first and second learning rates are pre-set parameters.
[0145] In an optional implementation, the initialization of the first network structure for the student model and the second network structure for the teacher model includes:
[0146] The first network structure and the second network structure are initialized with the same network structure, wherein the first network structure and the second network structure adopt a shared backbone network;
[0147] The backbone network of the first network structure is connected to multiple first detection branches;
[0148] The backbone network of the second network structure is connected to multiple second detection branches.
[0149] like Figure 2The diagram shows the first network structure of the initialized student model and the second network structure of the teacher model. The student model includes a first backbone network and multiple first detection branches connected to it. Each first detection branch corresponds to a category, and different first detection branches correspond to different categories. The teacher model includes a second backbone network and multiple second detection branches connected to it. Each second detection branch corresponds to a category, and different second detection branches correspond to different categories. The first backbone network and the second backbone network have the same network structure; that is, they are shared networks. For example, the first backbone network and the second backbone network can be a ResNet network structure or other network structures, depending on the specific requirements.
[0150] In this model, the number of first detection branches is the same as the number of second detection branches. For example, assuming the dataset has N categories, then the number of first detection branches and the number of second detection branches are both N. That is, the first backbone network of the student model connects to N first detection branches, and the second backbone network of the teacher model connects to N second detection branches.
[0151] The acquisition module 302 is used to determine the number of samples for a single category in batch training based on a given sample dataset of multiple categories, and to acquire multiple training sample data from the given sample dataset of multiple categories based on the number of samples.
[0152] The electronic device can acquire a sample dataset of multiple categories after initializing the first network structure of the student model and the second network structure of the teacher model; or it can acquire a sample dataset of multiple categories before initializing the first network structure of the student model and the second network structure of the teacher model.
[0153] A given multi-class sample dataset refers to a collection of sample data with multiple classes, where each sample data corresponds to a class and is uniquely labeled with a class label.
[0154] Sample data can include pedestrian images, vehicle images, and text images. When the sample data consists of pedestrian images, each pedestrian represents a category. Multiple pedestrian images are captured using an image acquisition device, and each set of images is labeled with a category label, resulting in multiple pedestrian image sets. When the sample data consists of vehicle images, each brand represents a category. Multiple vehicle images of each brand are captured using an image acquisition device, and each set of images is labeled with a category label, resulting in multiple vehicle image sets. When the sample data consists of text images, each application area represents a category. Multiple text images of each application area are captured using an image acquisition device, and each set of text images is labeled with a category label, resulting in multiple text image sets.
[0155] The electronic device can first determine the number of samples for a single category in batch training, and then randomly select multiple sample data from the given sample dataset of multiple categories as training sample data according to the number of samples. The multiple training sample data obtained shall have at least 2 categories, that is, training sample data of multiple categories are obtained.
[0156] In an optional implementation, after obtaining a sample dataset of a given number of categories, the method further includes:
[0157] The sample datasets of the multiple categories are preprocessed.
[0158] The given sample dataset can be called the original sample dataset. The sample data in the original sample dataset is the original sample data. After the electronic device obtains the original sample datasets of multiple categories, it performs preprocessing on the original sample data, including but not limited to: scaling, cropping, mirroring transformation, normalization and other preprocessing, in order to expand the original sample dataset and thereby enhance the robustness of the model.
[0159] It should be understood that after preprocessing the sample datasets of the multiple categories, the number of samples for a single category in batch training is determined based on the preprocessed sample datasets of the multiple categories, and multiple training sample data are obtained in batches from the preprocessed sample datasets of the multiple categories according to the number of samples.
[0160] In an optional implementation, determining the number of samples for a single class in batch training based on a given dataset of samples from multiple classes includes:
[0161] Calculate the number of samples in the sample dataset for each category;
[0162] The minimum sample size is determined based on the multiple sample sizes mentioned above;
[0163] Determine the sample sampling quantity according to the minimum sample quantity, where the sample sampling quantity is less than or equal to the minimum sample quantity.
[0164] For example, assume that the sample data sets of multiple categories are X = (X1, X2, Xi,..., XC), where Xi is the sample data set of the i-th category, 1 ≤ i ≤ C, C is the number of categories of the sample data sets, the total number of sample data in the X set is N, where the number of sample data in Xi is xi, that is, x1 + x2 + x3 +... + xC = N. Among x1, x2,..., xC, the minimum sample quantity is the number of sample data xi in the i-th category of sample data set (xi < xj, j ≠ i). Then the sample sampling quantity for each category can be less than or equal to the minimum sample quantity xi. Thus, the sample sampling quantity for each category can be balanced.
[0165] In each batch training, for each category, sample from the sample data set of each category respectively according to the sample sampling quantity to generate a sampling sample set for each category, generate batch training sample data according to the sampling sample set of each category, and use the batch training sample data to train the student model and the teacher model until the training is completed.
[0166] By dynamically sampling the sample data sets of each category through the sample sampling quantity, the sample sampling quantity for each category can be balanced, that is, each category samples the same number or approximately the same number of training sample data, so as to ensure the detection efficiency and detection accuracy of the trained model for each category.
[0167] The training module 303 is used to input the multiple training sample data into the first network structure for training to obtain multiple first predicted category labels, and input them into the second network structure for training to obtain multiple second predicted category labels.
[0168] After obtaining multiple training sample data, input the multiple training sample data into the first network structure of the student model for training and into the second network structure of the teacher model for training at the same time. Output multiple first predicted category labels through the first network structure of the student model, and output multiple second predicted category labels through the second network structure of the teacher model. The first predicted category labels and the second predicted category labels include the categories of the training sample data.
[0169] The first update module 304 is used to update the network parameters of the student model according to the multiple first predicted category labels and the multiple second predicted category labels.
[0170] The electronic device updates the network parameters of the student model based on multiple first predicted category labels obtained from the student model and multiple second predicted category labels obtained from the teacher model.
[0171] In an optional implementation, updating the network parameters of the student model based on the plurality of first predicted category labels and the plurality of second predicted category labels includes:
[0172] For each of the first detection branches of the student model, determine whether there is target training sample data corresponding to the first detection branch in the plurality of training sample data;
[0173] When the target training sample data exists, calculate the first loss function value based on the first predicted category label output by the first detection branch and the labeled category label of the target training sample data;
[0174] Obtain multiple remaining sample data from the plurality of training sample data, excluding the target training sample data;
[0175] For each remaining sample data, a second loss function value is calculated based on the second predicted category label output by the second detection branch corresponding to the remaining sample data and the first predicted category label output by the first detection branch corresponding to the remaining sample data.
[0176] The first detection branch is iteratively trained based on the first loss function value and the second loss function value, and the network parameters of the student model are updated.
[0177] Assuming the number of training sample data obtained is B, for each first detection branch H of the student model i The output, if there exists a data point belonging to dataset D among the B training samples. i If the target training sample data is B, then the first loss function value is calculated based on the labeled category label of the target training sample data and the predicted category label of the first detection branch; the remaining data in the B training sample data do not belong to dataset D. i The training sample data is then used to detect the second detection branch H of the teacher model. iThe output is used as pseudo-labels to calculate the second loss function value. All loss function values (i.e., the first and second loss function values) are summed, and then the network parameters of the student model are updated using the gradient descent algorithm. The first loss function value L1 can be calculated using the cross-entropy loss function, L1 = -SUM(y*logy'), where y represents the labeled class label, and y' represents the predicted class label output by the first detection branch. The second loss function value L2 can also be calculated using the cross-entropy loss function, L2 = -SUM(y'*logy”), where y' represents the predicted class label output by the first detection branch, and y” represents the predicted class label output by the second detection branch.
[0178] In practice, the electronic device can sequentially traverse each first detection branch of the student model. For each traversed first detection branch, it determines whether there is target training sample data corresponding to the traversed first detection branch among multiple training sample data. Since each first / second detection branch corresponds to a specified category label, a branch identifier can be set for the corresponding first / second detection branch according to the specified category label. For example, if the category label is "1", the branch identifier "1" can be set for the first first detection branch in the student model and the first second detection branch in the teacher model. In this way, the electronic device can determine whether there is target training sample data corresponding to the traversed first detection branch among multiple training sample data based on the branch identifier and the labeled category label of the sample data. When the labeled category label of the sample data is the same as the branch identifier of the traversed first detection branch, it is determined that there is target training sample data corresponding to the traversed first detection branch among the multiple training sample data. When the labeled category label of the sample data is different from the branch identifier of the traversed first detection branch, it is determined that there is no target training sample data corresponding to the traversed first detection branch among the multiple training sample data.
[0179] For example, assuming that each batch randomly selects one or more samples from each of the N sample datasets as training samples, the category of the obtained training samples is N. When traversing to the first detection branch of the student model (branch identifier H1), if it is possible to obtain sample data labeled with category D1 from the multiple training samples as target training samples, then based on the first predicted category label output by the first detection branch and the labeled category label of the target training samples, the first loss function value L1 = -SUM(y1*logy1') is calculated, where y1 represents the labeled category label of the target training samples and y1' represents the student model. The first detection branch outputs the predicted category label of the target training sample data; the sample data in the multiple training sample data whose category label is not D1 is taken as the remaining sample data. For each remaining sample data, the second loss function value L2 = -SUM(y2'*logy2”) is calculated according to the first predicted category label output by the first detection branch with the same branch identifier and the second predicted category label output by the second detection branch. Here, y2' represents the predicted category label of the remaining sample data output by the j-th second detection branch of the teacher model, and y2” represents the predicted category label of the remaining sample data output by the j-th first detection branch of the student model. The value of j is 2, 3, ..., N.
[0180] When traversing to the second first detection branch of the student model (branch identifier H2), if sample data labeled with category label D2 can be obtained from multiple training sample data as target training sample data, then the first loss function value is calculated based on the first predicted category label output by the first detection branch and the labeled category label of the target training sample data; the sample data with label label not equal to D2 from the multiple training sample data are taken as the remaining sample data L1 = -SUM(y1*logy1'), where y1 represents the labeled category label of the target training sample data, and y1' represents the predicted category label of the target training sample data output by the second first detection branch of the student model. For each remaining sample data, the second loss function value L2 = -SUM(y2'*logy2") is calculated based on the first predicted category label output by the first detection branch with the same branch identifier and the second predicted category label output by the second detection branch, where y2'
[0181] y1" represents the predicted class label of the remaining sample data output by the j-th second detection branch of the teacher model, and y2" represents the predicted class label of the remaining sample data output by the j-th first detection branch of the student model. The value of j is 1, 3, 4, ..., N.
[0182] This process is repeated for each of the first detection branches of the student model.
[0183] It should be understood that for each first detection branch traversed, a first loss function value and multiple second loss function values are obtained. Based on the first loss function value and multiple second loss function values, the total loss function value is calculated, and the gradient descent algorithm is used to iteratively train the corresponding first detection branch, thereby updating the network parameters of the student model.
[0184] In an optional implementation, the step of iteratively training the corresponding first detection branch based on the first loss function value and the second loss function value, and updating the network parameters of the student model, includes:
[0185] Obtain the gradient of the network parameters of the student model;
[0186] The gradient is compared with a preset gradient threshold to obtain a comparison result;
[0187] The gradient is updated to the target gradient based on the comparison result;
[0188] Calculate the total loss function value based on the first loss function value and the second loss function value;
[0189] The network parameters of the student model are updated based on the total loss function value, the target gradient, and the obtained preset learning rate.
[0190] The gradient of the network parameters can be any of the parameters in the student model or a specified parameter; this invention does not impose any restrictions.
[0191] The preset gradient threshold is set according to actual needs.
[0192] In each batch, the network parameters of the student model are updated based on the relationship between the gradient and the preset gradient threshold. After training in multiple batches, the target student model is determined.
[0193] The gradient is compared with a preset gradient threshold. The comparison results are: gradient greater than the preset threshold, and gradient less than the preset threshold. If the comparison result is that the gradient is greater than the preset threshold, it indicates that the student model's gradient descent is slow and needs to be updated. If the comparison result is that the gradient is less than the preset threshold, it indicates that the student model's gradient descent is moderate and does not need to be updated. It should be understood that the condition of gradient equal to the preset threshold applies to both cases where the gradient is greater than the preset threshold and cases where the gradient is less than the preset threshold.
[0194] In the above implementation method, when updating the network parameters of the student model, it is necessary not only to calculate the total loss function value, but also to include the first learning rate and the comparison result between the gradient and the preset gradient threshold. The gradient is updated by using the relationship between the gradient and the preset gradient threshold, and the student model is trained according to the updated target gradient. This can avoid the gradient explosion problem under a large learning rate, ensure the reliability of the training process, ensure that the student model converges quickly, and thus improve the training efficiency of the student model.
[0195] In some alternative implementations, updating the gradient to the target gradient based on the comparison result includes:
[0196] When the comparison result is that the gradient is greater than the preset gradient threshold, the norm of the gradient is calculated;
[0197] Calculate the ratio of the norm to the preset gradient threshold to determine the scaling factor;
[0198] The target gradient is obtained by multiplying the scaling factor and the gradient.
[0199] The norm corresponding to the gradient can be L0 norm, L1 norm, L2 norm, etc.
[0200] The ratio of the norm to a preset gradient threshold is used as a scaling factor. If the norm is greater than the preset gradient threshold, the scaling factor is greater than 1. The target gradient is obtained by multiplying the scaling factor by the gradient. If the norm is less than the preset gradient threshold, the scaling factor is less than 1. The target gradient is obtained by multiplying the value 1 by the gradient.
[0201] The second update module 305 is used to update the network parameters of the teacher model based on the updated network parameters of the student model.
[0202] During the current batch training, once the network parameters of the student model are updated, the network parameters of the teacher model can be updated based on the updated network parameters of the student model.
[0203] In an optional implementation, updating the network parameters of the teacher model based on the updated network parameters of the student model includes:
[0204] Obtain the first network parameters of the student model after the end of the current batch training, and the second network parameters of the teacher model after the end of the previous batch training;
[0205] Calculate the exponential moving average based on the first network parameters and the second network parameters;
[0206] The network parameters of the teacher model are updated to the exponential moving average when the current batch training ends.
[0207] Specifically, a moving average coefficient 'a' is set, and the update process of the network parameters for the current batch of teacher models is as follows: F' t =a*F' t-1 +(1-a)*F t , of which F t F' represents the network parameters after the model update for the t-th batch of students. t This represents the network parameters after the teacher model is updated in the t-th batch.
[0208] The training module 303 is used to terminate the training process of the student model and the teacher model when the iterative training termination condition is met, thereby obtaining a multi-object detector.
[0209] The termination condition for iterative training can be that the total loss function value is less than a preset loss threshold, or the number of batch training iterations is greater than a preset number of iterations threshold. When the termination condition for iterative training is met, the training process for the student model and the teacher model ends, resulting in the target student model and the target teacher model, which are then used as the final multi-object detector.
[0210] The detection module 306 is used to detect target data using the multi-target detector.
[0211] In this context, target data refers to data that requires multi-target detection, such as multiple faces contained in a face image, which are detected using a multi-target detector.
[0212] The target data is simultaneously input into both the target student model and the target teacher model for detection. Multiple first-predicted category labels are output through multiple first-detection branches of the target student model, and multiple second-predicted category labels are output through multiple second-detection branches of the target teacher model. The first-predicted category labels corresponding to the first-detection branches with the same branch identifier and the second-predicted category labels corresponding to the second-detection branches are obtained. If the first-predicted category label and the second-predicted category label are the same, the target predicted category label for the target data is determined to be the first-predicted category label / second-predicted category label. If the first-predicted category label and the second-predicted category label are different, the target predicted category label for the target data is determined to be the predicted category label with the higher prediction probability.
[0213] The multi-object detector trained in the above embodiments can be embedded in a mobile terminal, allowing the mobile terminal to perform object detection on the acquired object data. For example, by installing a corresponding APP on a mobile phone, when object detection is needed, the APP can be accessed, and object data can be acquired through functions such as taking a photo. Then, object detection can be performed on the object data to determine the predicted object category label.
[0214] This invention treats each detection task as a semi-supervised learning task, employing an average teacher algorithm in joint training across multiple datasets (datasets of multiple categories). It uses student and teacher models with identical network structures. In each iteration, the student model updates its network parameters by calculating the loss function and using gradient descent, while the teacher model's network parameters are obtained by calculating an exponential moving average from the student model's network parameters. Training a multi-object detector using multiple datasets of different categories not only reduces the number of model parameters compared to training multiple different detection models, but also prevents overfitting and improves the model's generalization ability. This invention flexibly utilizes annotations and the teacher model's output as training supervision in different detection branches of the student model, fully leveraging the information in the dataset and improving detection accuracy.
[0215] Example 3
[0216] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps described in the multi-target detection method embodiment above, for example... Figure 1 S11-S17 as shown:
[0217] S11, Initialize the first network structure of the student model and the second network structure of the teacher model;
[0218] S12, determine the number of samples for a single category in batch training based on the given sample dataset of multiple categories, and obtain multiple training sample data from the given sample dataset of multiple categories based on the number of samples.
[0219] S13, input the multiple training sample data into the first network structure for training to obtain multiple first predicted category labels, and input the data into the second network structure for training to obtain multiple second predicted category labels;
[0220] S14, Update the network parameters of the student model according to the plurality of first predicted category labels and the plurality of second predicted category labels;
[0221] S15, Update the network parameters of the teacher model according to the updated network parameters of the student model;
[0222] S16, When the iterative training termination condition is met, the training process of the student model and the teacher model ends, and a multi-object detector is obtained;
[0223] S17, Use the multi-target detector to detect the target data.
[0224] Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 Modules 301-306 in the document:
[0225] The initialization module 301 is used to initialize the first network structure of the student model and the second network structure of the teacher model.
[0226] The acquisition module 302 is used to determine the number of samples for a single category in batch training based on a given sample dataset of multiple categories, and to acquire multiple training sample data from the given sample dataset of multiple categories based on the number of samples.
[0227] The training module 303 is used to input the multiple training sample data into the first network structure for training to obtain multiple first predicted category labels, and input the data into the second network structure for training to obtain multiple second predicted category labels.
[0228] The first update module 304 is used to update the network parameters of the student model according to the plurality of first predicted category labels and the plurality of second predicted category labels;
[0229] The second update module 305 is used to update the network parameters of the teacher model according to the updated network parameters of the student model;
[0230] The training module 303 is also used to terminate the training process of the student model and the teacher model when the iterative training termination condition is met, so as to obtain a multi-object detector;
[0231] The detection module 306 is used to detect target data using the multi-target detector.
[0232] Example 4
[0233] See Figure 4 The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 4 includes a memory 41, at least one processor 42, at least one communication bus 43, and a transceiver 44.
[0234] Those skilled in the art should understand that Figure 4The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention. It can be a bus structure or a star structure. The electronic device 4 may also include more or fewer other hardware or software than shown, or different component arrangements.
[0235] In some embodiments, the electronic device 4 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 4 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.
[0236] The electronic device 4 described herein is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.
[0237] In some embodiments, the memory 41 stores a computer program that, when executed by the at least one processor 42, implements all or part of the steps in the multi-target detection method described above. The memory 41 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.
[0238] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application required for a function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.
[0239] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0240] In some embodiments, the at least one processor 42 is the control unit of the electronic device 4, connecting various components of the electronic device 4 via various interfaces and lines. It executes programs or modules stored in the memory 41 and calls data stored in the memory 41 to perform various functions and process data of the electronic device 4. For example, when the at least one processor 42 executes a computer program stored in the memory, it implements all or part of the steps of the multi-target detection method described in this embodiment of the invention; or it implements all or part of the functions of the multi-target detection device. The at least one processor 42 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.
[0241] In some embodiments, the at least one communication bus 43 is configured to enable communication between the memory 41 and the at least one processor 42, etc.
[0242] Although not shown, the electronic device 4 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 42 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power sources, a recharging device, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components. The electronic device 4 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0243] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of the present invention.
[0244] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0245] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0246] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0247] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements, and the singular does not exclude the plural. Multiple elements or devices recited in the specification may also be implemented by a single element or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.
[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A multi-target detection method, characterized in that, The method includes: Initializing the first network structure of the student model and the second network structure of the teacher model includes: initializing the first network structure and the second network structure with the same network structure, wherein the first network structure and the second network structure adopt a shared backbone network, connecting the backbone network of the first network structure with multiple first detection branches, and connecting the backbone network of the second network structure with multiple second detection branches; The number of samples for a single category in batch training is determined based on a given sample dataset of multiple categories. Multiple training sample data are obtained from the given sample dataset of multiple categories based on the number of samples. The number of the multiple first detection branches and the number of the multiple second detection branches are consistent with the number of categories in the sample dataset, and each of the first detection branches and each of the second detection branches is used to detect one category. The sample dataset is an image dataset. The multiple training sample data are input into the first network structure for training to obtain multiple first predicted category labels, and input into the second network structure for training to obtain multiple second predicted category labels; Updating the network parameters of the student model based on the plurality of first predicted category labels and the plurality of second predicted category labels includes: for each first detection branch of the student model, determining whether there is target training sample data corresponding to the first detection branch in the plurality of training sample data; when the target training sample data exists, calculating a first loss function value based on the first predicted category label output by the first detection branch and the labeled category label of the target training sample data; obtaining a plurality of remaining sample data in the plurality of training sample data excluding the target training sample data; for each remaining sample data, calculating a second loss function value based on the second predicted category label output by the second detection branch corresponding to the remaining sample data and the first predicted category label output by the first detection branch corresponding to the remaining sample data; iteratively training the corresponding first detection branch based on the first loss function value and the second loss function value; and updating the network parameters of the student model; each first detection branch has a corresponding branch identifier, and the labeled category label of the target training sample data is the same as the branch identifier of the corresponding first detection branch; The network parameters of the teacher model are updated based on the updated network parameters of the student model. When the iterative training termination condition is met, the training process for the student model and the teacher model ends, and a multi-object detector is obtained. The target data is detected using the multi-target detector.
2. The multi-target detection method as described in claim 1, characterized in that, Determining the number of samples for a single class in batch training based on a given dataset of samples from multiple classes includes: Calculate the number of samples in the sample dataset for each category; The minimum sample size is determined based on the multiple sample sizes mentioned above; The number of samples is determined based on the minimum number of samples, wherein the number of samples is less than or equal to the minimum number of samples.
3. The multi-target detection method as described in claim 1, characterized in that, The step of iteratively training the corresponding first detection branch based on the first loss function value and the second loss function value, and updating the network parameters of the student model, includes: Obtain the gradient of the network parameters of the student model; The gradient is compared with a preset gradient threshold to obtain a comparison result; The gradient is updated to the target gradient based on the comparison result; Calculate the total loss function value based on the first loss function value and the second loss function value; The network parameters of the student model are updated based on the total loss function value, the target gradient, and the obtained preset learning rate.
4. The multi-target detection method as described in claim 3, characterized in that, The step of updating the gradient to the target gradient based on the comparison result includes: When the comparison result is that the gradient is greater than the preset gradient threshold, the norm of the gradient is calculated; Calculate the ratio of the norm to the preset gradient threshold to determine the scaling factor; The target gradient is obtained by multiplying the scaling factor and the gradient.
5. The multi-target detection method according to any one of claims 1 to 4, characterized in that, The step of updating the network parameters of the teacher model based on the updated network parameters of the student model includes: Obtain the first network parameters of the student model after the end of the current batch training, and the second network parameters of the teacher model after the end of the previous batch training; Calculate the exponential moving average based on the first network parameters and the second network parameters; The network parameters of the teacher model are updated to the exponential moving average when the current batch training ends.
6. A multi-target detection device, said device being used to implement the multi-target detection method as described in any one of claims 1 to 5, characterized in that, The device includes: An initialization module is used to initialize the first network structure of the student model and the second network structure of the teacher model. The acquisition module is used to determine the number of samples for a single category in batch training based on a given sample dataset of multiple categories, and to acquire multiple training sample data from the given sample dataset of multiple categories based on the number of samples. The training module is used to input the multiple training sample data into the first network structure for training to obtain multiple first predicted category labels, and input the data into the second network structure for training to obtain multiple second predicted category labels; The first update module is used to update the network parameters of the student model based on the plurality of first predicted category labels and the plurality of second predicted category labels; The second update module is used to update the network parameters of the teacher model based on the updated network parameters of the student model. The training module is also used to terminate the training process of the student model and the teacher model when the iterative training termination condition is met, so as to obtain a multi-object detector; The detection module is used to detect target data using the multi-target detector.
7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the multi-target detection method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-target detection method as described in any one of claims 1 to 5.