Model training method and device based on knowledge distillation, electronic equipment and readable storage medium

Abstract

The invention provides a model training method and device based on knowledge distillation, electronic equipment and a readable storage medium, and relates to the technical field of artificial intelligence such as deep learning, computer vision and large language models. The knowledge distillation-based model training method comprises the steps of determining a target domain according to a target task, and obtaining a target teacher model corresponding to the target domain; performing knowledge distillation according to a teacher backbone network in the target teacher model, and training an initial student backbone network to obtain a target student backbone network; and obtaining a target student model corresponding to the target task according to the target student backbone network and a target head network corresponding to the target task. According to the invention, on one hand, the performance of the target student backbone network during feature extraction for the target task is effectively improved, and on the other hand, the acquisition steps of the target student model can be simplified, and the acquisition efficiency of the target student model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as deep learning, computer vision, and large language models. A model training method, device, electronic device, and readable storage medium based on knowledge distillation are provided. Background Art

[0002] With the development of information technology, neural network models are widely used in different machine learning tasks. In order to achieve better processing effects, neural network models often have large-scale parameters and complex model structures, that is, in the training and application stages, neural network models will occupy a lot of computing resources. Therefore, such large neural network models cannot be deployed accordingly on some resource-constrained devices. Summary of the invention

[0003] According to a first aspect of the present disclosure, a model training method based on knowledge distillation is provided, comprising: determining a target domain according to a target task, and acquiring a target teacher model corresponding to the target domain; performing knowledge distillation according to a teacher backbone network in the target teacher model, training an initial student backbone network, and obtaining a target student backbone network; and obtaining a target student model corresponding to the target task according to the target student backbone network and a target head network corresponding to the target task.

[0004] According to a second aspect of the present disclosure, a model training device based on knowledge distillation is provided, comprising: an acquisition unit, used to determine a target domain according to a target task, and acquire a target teacher model corresponding to the target domain; a distillation unit, used to perform knowledge distillation according to a teacher backbone network in the target teacher model, train an initial student backbone network, and obtain a target student backbone network; a processing unit, used to obtain a target student model corresponding to the target task according to the target student backbone network and a target head network corresponding to the target task.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0006] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0008] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0010] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0011] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0014] Figure 5 It is a block diagram of an electronic device used to implement the model training method based on knowledge distillation of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0016] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the model training method based on knowledge distillation in this embodiment specifically includes the following steps:

[0017] S101, determining a target domain according to a target task, and acquiring a target teacher model corresponding to the target domain;

[0018] S102, performing knowledge distillation according to the teacher backbone network in the target teacher model, training an initial student backbone network, and obtaining a target student backbone network;

[0019] S103. Obtain a target student model corresponding to the target task based on the target student backbone network and the target head network corresponding to the target task.

[0020] The model training method based on knowledge distillation of this embodiment, on the one hand, uses the teacher backbone network in the target teacher model corresponding to the target field to perform knowledge distillation to obtain the target student backbone network corresponding to the target task, so that the knowledge of the large and complex teacher backbone network can be transferred to the small and simple target student backbone network, thereby effectively improving the performance of the target student backbone network in feature extraction for the target task. On the other hand, after obtaining the target student backbone network, it is only necessary to combine the target head network corresponding to the target task to obtain the target student model corresponding to the target task, without the need to train other networks in the target student model, which can simplify the steps of obtaining the target student model and improve the efficiency of obtaining the target student model.

[0021] In this embodiment, the teacher model (Teacher Model) is a large neural network model (i.e., large model) with excellent performance, a large number of parameters, and a relatively complex structure. It is responsible for providing learning knowledge so that this knowledge can be learned by another model (i.e., student model); the student model (Student Model) is a small neural network model (i.e., small model) with a small number of parameters and a relatively simple structure. It improves its own performance by learning the knowledge provided by the teacher model; usually, the student model or the small model is suitable for deployment on resource-constrained devices.

[0022] In this embodiment, knowledge distillation (KD) is a model compression method in deep learning, which is used to effectively transfer the knowledge of a large model (i.e., teacher model) to a small model (i.e., student model), thereby improving the performance of the small model.

[0023] In this embodiment, the backbone network is the basic component of the neural network model, which is responsible for extracting features from the input data. The extracted features will then be used for various tasks, such as classification, detection, segmentation, etc.

[0024] When executing S101, this embodiment can obtain the field corresponding to the target task in the correspondence relationship according to the preset correspondence relationship between tasks and fields as the target field; that is, the target field is the field to which the target task belongs, and the target task corresponds to a specific business in the actual application scenario.

[0025] For example, if the corresponding relationship includes roadside gun detection tasks, roadside fisheye detection tasks, pedestrian detection tasks, motor vehicle detection tasks, non-motor vehicle detection tasks and license plate detection tasks, the field corresponding to the above tasks is the traffic field; if the target task in this embodiment is the license plate detection task, then this embodiment will obtain the traffic field as the target field.

[0026] For another example, if the corresponding relationship includes product category detection tasks, product identification tasks, product recommendation tasks, product evaluation feedback tasks, etc., the field corresponding to the above tasks is the e-commerce field; if the target task in this embodiment is the product recommendation task, then this embodiment will obtain the e-commerce field as the target field.

[0027] In this embodiment, after executing S101 to determine the target domain, the target teacher model corresponding to the target domain can be obtained; in this embodiment, the obtained target teacher model is a multi-task large model.

[0028] Specifically, when executing S101 to obtain the target teacher model corresponding to the target domain, the implementation method that can be adopted in this embodiment is: obtain a target training data set corresponding to the target domain, the obtained target training data set includes target training data corresponding to different preset tasks, and the preset tasks are at least two tasks included in the target domain; use the obtained target training data set to train the initial teacher model to obtain the target teacher model; wherein the initial teacher model includes a backbone network.

[0029] That is to say, this embodiment uses a target training data set corresponding to the target field for real-time training to obtain a target teacher model. Since the target training data set includes training data corresponding to different preset tasks, the trained target teacher model can handle a variety of preset tasks, thereby effectively improving the model performance of the target teacher model and ensuring that the target teacher backbone network in the target teacher model can perform feature extraction for different tasks.

[0030] For example, if the target field is the traffic field, and if the preset tasks are all tasks corresponding to the traffic field, the target training data set obtained by executing S101 in this embodiment includes training data corresponding to the roadside gun detection task, training data corresponding to the roadside fisheye detection task, training data corresponding to the pedestrian detection task, training data corresponding to the motor vehicle detection task, training data corresponding to the non-motor vehicle detection task, and training data corresponding to the license plate detection task, etc.

[0031] For another example, the training data in the target training data set includes input data (such as image data) and its corresponding annotation results; if the target training data set corresponds to the traffic field, the training data in this embodiment may include license plate images (input data) and license plate numbers (annotation results) corresponding to the "license plate detection task", and may also include images (input data) corresponding to the "vehicle detection task" and vehicle detection results in the images (annotation results), etc.

[0032] The preset tasks in this embodiment may also be at least two tasks including the target task. For example, when the target task is a pedestrian detection task, the preset tasks in this embodiment may be a pedestrian detection task, a motor vehicle detection task, and a non-motor vehicle detection task. They may also be a pedestrian detection task and a license plate detection task.

[0033] In addition, the preset tasks in this embodiment may also be at least two arbitrary tasks among the tasks included in the target domain, that is, the preset tasks may not include the target task.

[0034] It can be understood that when executing S101 to obtain the target teacher model corresponding to the target domain, this embodiment can also obtain a pre-trained teacher model corresponding to the target domain as the target teacher model; wherein the pre-training process is the same as the above-mentioned real-time training process.

[0035] After executing S101 to obtain the target teacher model corresponding to the target domain, this embodiment executes S102 to perform knowledge distillation according to the teacher backbone network in the target teacher model, train the initial student backbone network, and obtain the target student backbone network.

[0036] In this embodiment, the knowledge of the teacher backbone network in the target teacher model is transferred to the initial student backbone network by knowledge distillation, so that the trained target student backbone network can obtain feature extraction performance close to that of the teacher backbone network.

[0037] In this embodiment, the teacher backbone network is a backbone network with excellent performance, a large number of parameters, and a relatively complex structure; the initial student backbone network is a backbone network with a small number of parameters and a relatively simple structure; the target student backbone network trained by knowledge distillation is a backbone network with excellent performance, a small number of parameters, and a relatively simple structure.

[0038] When executing S102 to obtain the initial student backbone network, the implementation method that can be adopted in this embodiment is: obtain business information corresponding to the target task. The business information obtained in this embodiment can be scenario information of business deployment, equipment information of business deployment, etc.; obtain the student backbone network corresponding to the business information as the initial student backbone network.

[0039] That is to say, this embodiment can obtain the initial student backbone network in combination with the business information corresponding to the target task, so that the obtained initial student backbone network can correspond to the specific business information, thereby improving the accuracy of the obtained initial student backbone network.

[0040] In addition, when executing S102, this embodiment can also use the student backbone network corresponding to the target task in the correspondence relationship as the initial student backbone network according to the correspondence relationship between the preset tasks and the student backbone network.

[0041] It can be understood that when executing S102, this embodiment can obtain one initial student network or multiple initial student backbone networks. Different initial student backbone networks can correspond to different device computing powers or different parameter scales; for example, the initial student backbone network A corresponding to computing power A, the initial student backbone network B corresponding to parameter scale B, etc., that is, when executing S102, this embodiment can obtain a backbone network cluster composed of multiple initial student backbone networks.

[0042] In addition, when executing S102, this embodiment may also directly obtain a preset student backbone network as the initial student backbone network without the need for business information or task information.

[0043] After executing S102 to obtain the target student backbone network, this embodiment executes S103 to obtain the target student model corresponding to the target task based on the target student backbone network and the target head network corresponding to the target task.

[0044] In this embodiment, the target student model corresponding to the target task is a small model that can be actually deployed on a resource-constrained device; after the target student model is deployed on the resource-constrained device, the device can complete the services corresponding to the target task, such as license plate detection services, pedestrian detection services, commodity category detection services, etc.

[0045] In this embodiment, the target head network (Head) corresponding to the target task is used to complete the specific task according to the features extracted by the backbone network, thereby outputting the corresponding processing results, such as pedestrian detection results, license plate detection results, commodity category recognition results, etc.

[0046] When executing S103, this embodiment splices the target student backbone network and the target head network in the order of the target student backbone network first and the target head network later, and then uses the splicing result as the target student model corresponding to the target task.

[0047] In addition, when executing S103, this embodiment can further obtain a target neck network (Neck) corresponding to the target task, and then obtain a target student model corresponding to the target task based on the target student backbone network, the target neck network and the target head network.

[0048] It can be understood that after executing S103 to obtain the target student model, this embodiment can also obtain training data corresponding to the target task, and then use the acquired training data to verify or further optimize the target student model (for example, when the target task is not included in the preset tasks), so that after completing the verification or optimization of the target student model, the target student model is deployed to the corresponding device so that the device can complete the business corresponding to the target task.

[0049] In this embodiment, since multiple preset tasks corresponding to the target field have been combined in the process of obtaining the target student backbone network, the target student backbone network will also have a certain accuracy when performing feature extraction for related tasks in the target field (other tasks except the preset tasks, such as the target tasks).

[0050] Therefore, even if the present embodiment does not use the training data related to the target task in the knowledge distillation process (for example, when the target task is a new task and there is currently no relevant training data), by subsequently acquiring the training data corresponding to the target task to optimize the obtained target student model, the target student model can have a higher accuracy when processing the target task.

[0051] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown in , when executing S102 "performing knowledge distillation according to the teacher backbone network in the target teacher model, training the initial student backbone network, and obtaining the target student backbone network" in this embodiment, the implementation method that can be adopted is:

[0052] S201, obtaining a target training data set corresponding to the target field;

[0053] S202, inputting the input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, to obtain the teacher prediction features output by the teacher backbone network and the student prediction features output by the initial student backbone network;

[0054] S203, obtaining a target loss function value according to the teacher prediction feature and the student prediction feature;

[0055] S204. Use the target loss function value to adjust the parameters of the initial student backbone network to obtain the target student backbone network.

[0056] That is to say, this embodiment uses the input data (such as image data) corresponding to different preset tasks in the target training data set, and combines it with the teacher backbone network to train the initial student backbone network. Through knowledge distillation, during the training process, the initial student backbone network continuously learns the knowledge of the teacher backbone network when performing feature extraction, so that the trained target student backbone network has feature extraction performance close to that of the teacher backbone network.

[0057] In addition, this embodiment uses knowledge distillation to train only the backbone network used for feature extraction, without the need to train the corresponding head network or neck network. This can simplify the steps in obtaining the target student model and improve the efficiency in obtaining the target student model.

[0058] The target training data set obtained by executing S201 in this embodiment includes training data corresponding to different preset tasks, and each training data is composed of input data (such as image data) and its corresponding annotation result.

[0059] When executing S202, this embodiment can input the same input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, so as to obtain the teacher prediction features output by the teacher backbone network and the student prediction features output by the initial student backbone network.

[0060] It can be understood that if multiple initial student backbone networks are obtained, this embodiment can train each initial student backbone network separately to obtain multiple target student backbone networks. Different target student backbone networks can correspond to different device computing powers, different parameter scales, etc.

[0061] When executing S203 in this embodiment to obtain the target loss function value according to the teacher prediction features and the student prediction features, the implementation method that can be adopted is: obtain multiple teacher prediction sub-features according to the teacher prediction features and multiple feature scales; obtain multiple student prediction sub-features according to the student prediction features and multiple feature scales; obtain sub-loss function values ​​corresponding to different feature scales according to the teacher prediction sub-features and the student prediction sub-features corresponding to the same feature scales; obtain the target loss function value according to the multiple sub-loss function values.

[0062] That is to say, this embodiment combines multiple preset feature scales to calculate the target loss function value, so that the obtained target loss function value includes information at different feature scales, which can improve the accuracy of the obtained target loss function value, and then improve the applicability of the obtained target student backbone network to different tasks in the target field.

[0063] It can be understood that when executing S203 to obtain the target loss function value, this embodiment can also normalize the teacher prediction feature or teacher prediction sub-feature, that is, normalize the feature values ​​corresponding to different positions in the teacher prediction feature or teacher prediction sub-feature to values ​​between 0 and 1, and then obtain the target loss function value based on the normalized results of the teacher prediction feature or teacher prediction sub-feature.

[0064] This embodiment can reduce the convergence difficulty of the initial student backbone network during training by normalizing the teacher prediction features or teacher prediction sub-features, thereby improving the efficiency in obtaining the target student backbone network.

[0065] After executing S204 to obtain the target student backbone network, this embodiment may further include the following contents: obtaining a detection data set corresponding to the detection task, such as a COCO (Common Objects in Context) data set; and fine-tuning the target student backbone network using the obtained detection data set.

[0066] In actual application scenarios, the feature extraction accuracy of the target student backbone network trained by the above-mentioned knowledge distillation method for the detection task is lower than that of other tasks. Therefore, this embodiment obtains the detection data set corresponding to the detection task to fine-tune the target student backbone network, thereby improving the performance of the target student backbone network on the detection task.

[0067] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure. Figure 3, a flowchart of the present embodiment when obtaining a target student backbone network is shown in FIG. 1 : S301, inputting the input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, for example, inputting the image data corresponding to the traffic field respectively; S302, the teacher backbone network obtains the teacher prediction feature according to the input data input; S303, the initial student backbone network obtains the student prediction feature according to the input data input; S304, according to the teacher prediction feature and different feature scales, a plurality of teacher prediction sub-features are obtained, for example, the teacher prediction feature is subjected to (1 / 32, 1 / 16, 1 / 8) and other scales; S305, according to the student prediction features and different feature scales, multiple student prediction sub-features are obtained, for example, the student prediction features are downsampled to (1 / 32, 1 / 16, 1 / 8) and other scales; S306, according to the teacher prediction sub-features and the student prediction sub-features corresponding to the same feature scales, the sub-loss function values ​​corresponding to different feature scales are obtained; S307, according to the multiple sub-loss function values, the target loss function value is obtained; S308, according to the target loss function value, the parameters of the initial student backbone network are adjusted to obtain the target student backbone network.

[0068] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 As shown, the model training device 400 based on knowledge distillation of this embodiment includes:

[0069] An acquisition unit 401 is used to determine a target domain according to a target task, and acquire a target teacher model corresponding to the target domain;

[0070] A distillation unit 402 is used to perform knowledge distillation according to the teacher backbone network in the target teacher model, train the initial student backbone network, and obtain a target student backbone network;

[0071] The processing unit 403 is used to obtain a target student model corresponding to the target task according to the target student backbone network and the target head network corresponding to the target task.

[0072] The acquisition unit 401 can acquire the field corresponding to the target task in the correspondence relationship according to the preset correspondence relationship between tasks and fields as the target field; that is, the target field is the field to which the target task belongs, and the target task corresponds to a specific business in an actual application scenario.

[0073] After determining the target domain, the acquisition unit 401 can acquire the target teacher model corresponding to the target domain; in this embodiment, the acquired target teacher model is a multi-task large model.

[0074] Specifically, when the acquisition unit 401 acquires the target teacher model corresponding to the target domain, the implementation method that can be adopted is: acquiring a target training data set corresponding to the target domain; using the acquired target training data set to train the initial teacher model to obtain the target teacher model.

[0075] That is to say, the acquisition unit 401 uses a target training data set corresponding to the target field for real-time training to obtain a target teacher model. Since the target training data set includes training data corresponding to different preset tasks, the trained target teacher model can handle a variety of preset tasks, thereby effectively improving the model performance of the target teacher model and ensuring that the target teacher backbone network in the target teacher model can perform feature extraction for different tasks.

[0076] The preset tasks in this embodiment may also be at least two tasks including the target task; or may be at least two arbitrary tasks included in the target domain, that is, the preset tasks may not include the target task.

[0077] It can be understood that when acquiring the target teacher model corresponding to the target domain, the acquisition unit 401 can also acquire a pre-trained teacher model corresponding to the target domain as the target teacher model; wherein the pre-training process is the same as the above-mentioned real-time training process.

[0078] In this embodiment, after the acquisition unit 401 acquires the target teacher model corresponding to the target domain, the distillation unit 402 performs knowledge distillation based on the teacher backbone network in the target teacher model, trains the initial student backbone network, and obtains the target student backbone network.

[0079] In this embodiment, the knowledge of the teacher backbone network in the target teacher model is transferred to the initial student backbone network by knowledge distillation, so that the trained target student backbone network can obtain feature extraction performance close to that of the teacher backbone network.

[0080] In this embodiment, the teacher backbone network is a backbone network with excellent performance, a large number of parameters, and a relatively complex structure; the initial student backbone network is a backbone network with a small number of parameters and a relatively simple structure; the target student backbone network trained by knowledge distillation is a backbone network with excellent performance, a small number of parameters, and a relatively simple structure.

[0081] When the distillation unit 402 obtains the initial student backbone network, the implementation method that can be adopted is: obtain business information corresponding to the target task. The business information obtained in this embodiment may be scenario information of business deployment, equipment information of business deployment, etc.; obtain the student backbone network corresponding to the business information as the initial student backbone network.

[0082] That is to say, the distillation unit 402 can obtain the initial student backbone network in combination with the business information corresponding to the target task, so that the obtained initial student backbone network can correspond to the specific business information, thereby improving the accuracy of the obtained initial student backbone network.

[0083] In addition, the distillation unit 402 can also use the student backbone network corresponding to the target task in the correspondence relationship between the preset tasks and the student backbone network as the initial student backbone network.

[0084] It can be understood that the distillation unit 402 can obtain one initial student network or multiple initial student backbone networks. Different initial student backbone networks can correspond to different device computing powers or different parameter scales, that is, the distillation unit 402 can obtain a backbone network cluster composed of multiple initial student backbone networks.

[0085] In addition, the distillation unit 402 may also directly obtain a preset student backbone network as the initial student backbone network without the need for business information or task information.

[0086] Specifically, when the distillation unit 402 performs knowledge distillation according to the teacher backbone network in the target teacher model, trains the initial student backbone network, and obtains the target student backbone network, the implementation method that can be adopted is: obtain a target training data set corresponding to the target field; input the input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, and obtain the teacher prediction features output by the teacher backbone network and the student prediction features output by the initial student backbone network; obtain the target loss function value according to the teacher prediction features and the student prediction features; use the target loss function value to adjust the parameters of the initial student backbone network to obtain the target student backbone network.

[0087] That is to say, the distillation unit 402 uses the input data (such as image data) corresponding to different preset tasks in the target training data set, and combines it with the teacher backbone network to train the initial student backbone network. Thus, through the knowledge distillation method, during the training process, the initial student backbone network continuously learns the knowledge of the teacher backbone network when performing feature extraction, so that the trained target student backbone network has feature extraction performance close to that of the teacher backbone network.

[0088] In addition, this embodiment uses knowledge distillation to train only the backbone network used for feature extraction, without the need to train the corresponding head network or neck network. This can simplify the steps in obtaining the target student model and improve the efficiency in obtaining the target student model.

[0089] The target training data set acquired by the distillation unit 402 includes training data corresponding to different preset tasks, and each training data is composed of input data (such as image data) and its corresponding annotation results.

[0090] The distillation unit 402 can input the same input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, thereby obtaining the teacher prediction features output by the teacher backbone network and the student prediction features output by the initial student backbone network.

[0091] It is understandable that if multiple initial student backbone networks are obtained, the distillation unit 402 can train each initial student backbone network separately to obtain multiple target student backbone networks. Different target student backbone networks can correspond to different device computing powers, different parameter scales, etc.

[0092] When the distillation unit 402 obtains the target loss function value based on the teacher prediction features and the student prediction features, the implementation method that can be adopted is: according to the teacher prediction features and multiple feature scales, multiple teacher prediction sub-features are obtained; according to the student prediction features and multiple feature scales, multiple student prediction sub-features are obtained; according to the teacher prediction sub-features and the student prediction sub-features corresponding to the same feature scales, sub-loss function values ​​corresponding to different feature scales are obtained; according to the multiple sub-loss function values, the target loss function value is obtained.

[0093] That is to say, the distillation unit 402 calculates the target loss function value in combination with multiple preset feature scales, so that the obtained target loss function value includes information at different feature scales, which can improve the accuracy of the obtained target loss function value, and thus improve the applicability of the obtained target student backbone network to different tasks in the target field.

[0094] It can be understood that when the distillation unit 402 obtains the target loss function value, it can also normalize the teacher prediction feature or the teacher prediction sub-feature, that is, normalize the feature values ​​corresponding to different positions in the teacher prediction feature or the teacher prediction sub-feature to values ​​between 0 and 1, and then obtain the target loss function value based on the normalized results of the teacher prediction feature or the teacher prediction sub-feature.

[0095] This embodiment can reduce the convergence difficulty of the initial student backbone network during training by normalizing the teacher prediction features or teacher prediction sub-features, thereby improving the efficiency in obtaining the target student backbone network.

[0096] After obtaining the target student backbone network, the distillation unit 402 may further include the following contents: obtaining a detection data set corresponding to the detection task; and fine-tuning the target student backbone network using the obtained detection data set.

[0097] In actual application scenarios, the feature extraction accuracy of the target student backbone network trained by the above-mentioned knowledge distillation method for the detection task is lower than that of other tasks. Therefore, the distillation unit 402 obtains the detection data set corresponding to the detection task to fine-tune the target student backbone network, thereby improving the performance of the target student backbone network on the detection task.

[0098] In this embodiment, after the distillation unit 402 obtains the target student backbone network, the processing unit 403 obtains the target student model corresponding to the target task based on the target student backbone network and the target head network corresponding to the target task.

[0099] In this embodiment, the target student model corresponding to the target task is a small model that can be actually deployed on a resource-constrained device; after the target student model is deployed on the resource-constrained device, the device can complete the services corresponding to the target task, such as license plate detection services, pedestrian detection services, commodity category detection services, etc.

[0100] In this embodiment, the target head network (Head) corresponding to the target task is used to complete the specific task according to the features extracted by the backbone network, thereby outputting the corresponding processing results, such as pedestrian detection results, license plate detection results, commodity category recognition results, etc.

[0101] The processing unit 403 splices the target student backbone network with the target head network in the order of the target student backbone network in front and the target head network in the back, and then uses the splicing result as the target student model corresponding to the target task.

[0102] In addition, the processing unit 403 can further obtain a target neck network (Neck) corresponding to the target task, and then obtain a target student model corresponding to the target task according to the target student backbone network, the target neck network and the target head network.

[0103] It can be understood that after obtaining the target student model, the processing unit 403 can also obtain training data corresponding to the target task, and then use the obtained training data to verify or further optimize the target student model (for example, when the target task is not included in the preset tasks), so that after completing the verification or optimization of the target student model, the target student model is deployed to the corresponding device so that the device can complete the business corresponding to the target task.

[0104] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0106] like Figure 5 , is a block diagram of an electronic device according to a model training method based on knowledge distillation according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0107] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0108] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a model training method based on knowledge distillation. For example, in some embodiments, the model training method based on knowledge distillation may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508.

[0110] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the model training method based on knowledge distillation described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the model training method based on knowledge distillation in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable knowledge distillation-based model training device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method based on knowledge distillation, comprising: Determine a target domain according to the target task, and obtain a target teacher model corresponding to the target domain; Performing knowledge distillation according to the teacher backbone network in the target teacher model, training an initial student backbone network, and obtaining a target student backbone network; According to the target student backbone network and the target head network corresponding to the target task, a target student model corresponding to the target task is obtained.

2. The method according to claim 1, wherein: The acquiring of the target teacher model corresponding to the target domain comprises: Acquire a target training data set corresponding to the target domain; The target training data set is used to train the initial teacher model to obtain the target teacher model.

3. The method according to claim 1, wherein: Acquiring the initial student backbone network includes: Acquire business information corresponding to the target task; A student backbone network corresponding to the business information is obtained as the initial student backbone network.

4. The method according to claim 1, wherein: The performing of knowledge distillation according to the teacher backbone network in the target teacher model, training the initial student backbone network, and obtaining the target student backbone network comprises: Acquire a target training data set corresponding to the target domain; Inputting the input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, to obtain the teacher prediction features output by the teacher backbone network and the student prediction features output by the initial student backbone network; Obtaining a target loss function value according to the teacher prediction feature and the student prediction feature; The target loss function value is used to adjust the parameters of the initial student backbone network to obtain the target student backbone network.

5. The method according to claim 4, wherein: The obtaining of the target loss function value according to the teacher prediction feature and the student prediction feature includes: According to the teacher prediction feature and multiple feature scales, multiple teacher prediction sub-features are obtained; According to the student prediction feature and a plurality of feature scales, a plurality of student prediction sub-features are obtained; According to the teacher prediction sub-features and the student prediction sub-features corresponding to the same feature scale, the sub-loss function values ​​corresponding to different feature scales are obtained; The target loss function value is obtained according to multiple sub-loss function values.

6. The method according to claim 1, further comprising: After obtaining the target student backbone network, obtaining a detection data set corresponding to the detection task; Using the detection dataset, the target student backbone network is fine-tuned.

7. A model training device based on knowledge distillation, comprising: An acquisition unit, used to determine a target domain according to a target task, and acquire a target teacher model corresponding to the target domain; A distillation unit, used to perform knowledge distillation according to the teacher backbone network in the target teacher model, train the initial student backbone network, and obtain the target student backbone network; A processing unit is used to obtain a target student model corresponding to the target task based on the target student backbone network and the target head network corresponding to the target task.

8. The device according to claim 7, wherein: When acquiring the target teacher model corresponding to the target domain, the acquisition unit specifically performs: Acquire a target training data set corresponding to the target domain; The target training data set is used to train the initial teacher model to obtain the target teacher model.

9. The device according to claim 7, wherein: When obtaining the initial student backbone network, the distillation unit specifically performs: Acquire business information corresponding to the target task; A student backbone network corresponding to the business information is obtained as the initial student backbone network.

10. The device according to claim 7, wherein: When the distillation unit performs knowledge distillation according to the teacher backbone network in the target teacher model, trains the initial student backbone network, and obtains the target student backbone network, the distillation unit specifically performs: Acquire a target training data set corresponding to the target domain; Inputting the input data in the target training data set into the teacher backbone network and the initial student backbone network respectively, to obtain the teacher prediction features output by the teacher backbone network and the student prediction features output by the initial student backbone network; Obtaining a target loss function value according to the teacher prediction feature and the student prediction feature; The target loss function value is used to adjust the parameters of the initial student backbone network to obtain the target student backbone network.

11. The device according to claim 10, wherein: When the distillation unit obtains the target loss function value according to the teacher prediction feature and the student prediction feature, it specifically performs: According to the teacher prediction feature and multiple feature scales, multiple teacher prediction sub-features are obtained; According to the student prediction feature and a plurality of feature scales, a plurality of student prediction sub-features are obtained; According to the teacher prediction sub-features and the student prediction sub-features corresponding to the same feature scale, the sub-loss function values ​​corresponding to different feature scales are obtained; The target loss function value is obtained according to multiple sub-loss function values.

12. The apparatus according to claim 7, wherein the distillation unit is further configured to perform: After obtaining the target student backbone network, obtaining a detection data set corresponding to the detection task; Using the detection dataset, the target student backbone network is fine-tuned.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images