Model distillation training method and related device and equipment, readable storage medium

Abstract

A method for model distillation training includes: before a first device distills and trains a neural network model, it is necessary to establish a communication connection for distillation training with a second device. Through the interaction of distillation training information between the first device and the second device, the second device uses the first configuration information sent by the first device to configure a first reference neural network model. After the second device configures the first reference neural network model, it uses the first data information to calculate and process the first sample data in the first data information through the configured first reference neural network model, obtains the first indication information, and sends it to the first device. The first device uses the first indication information to train the first neural network model it designed. In this way, the second device guides the first device to perform distillation training by transferring "knowledge".

Description

Technical Field

[0001] The present application relates to the technical field of model distillation training, and in particular to a model distillation training method and related devices and equipment, and a readable storage medium. Background Art

[0002] In information technology, central network elements and edge network elements are common application scenarios in the network. For example, in a wireless scenario, the Radio Access Network (RAN) can be used as a central network element and the User Equipment (UE) can be used as an edge network element. Similarly, in 3GPP, a standardized Network Data Analytics Function (NWDAF) is introduced, and there are also deployment scenarios for central NWDAF and edge NWDAF. The central network element can send a trained neural network model to the edge network element. Figure 1 This is a schematic diagram of the deployment of neural networks between central network elements and edge network elements, such as Figure 1 As shown in the figure, since the central network element has powerful computing power and stores a large amount of data resources, the central network element usually deploys a large neural network (large network) for all categories of scenarios and trains the neural network. The edge network element has limited storage resources and computing resources, and the edge network element usually only focuses on a few specific categories, so the edge network element usually deploys a small neural network (small network).

[0003] At present, there are two main ways for edge network elements to deploy neural networks: one way is that the central network element directly sends the large network to the edge network element, and the edge network element deploys the large network. Since the storage resources and computing resources of edge network elements are limited, the edge network element will occupy the limited storage resources of the edge network element when deploying the neural network in this way. Another way is that the central network element performs network model distillation to obtain a unified small network, and sends the small network to the edge network element, and the edge network element deploys the small network. Since the central network element does not know the storage resources, computing power, and categories of interest of each edge network element, it is impossible to distill a small network model that is adapted to each edge network element. The problem caused by the incompatibility between the small network model obtained by unified distillation training of the central network element and the edge network element is that the categories supported by the small network model are all categories, while the edge network element only pays attention to a few specific categories. When the computing power and storage resources required by the small network model of all categories exceed the capabilities of the edge network element, the small network model will occupy the storage space and running space of the edge network element, affecting the edge network element to complete other tasks, thereby reducing the working performance of the edge network element; in addition, due to the limited computing power and storage resources of the edge network element, the edge network element may not be able to fully run the small network model of all categories, resulting in low completion and accuracy when the small network model performs functions such as portrait recognition and traffic recognition. Summary of the invention

[0004] The present application provides a method for model distillation training, which solves the problem that the edge network element deploys a large network sent by the central network element and occupies its limited storage resources, and also solves the problem that the small network obtained by the unified distillation training sent by the central network element is not compatible with the edge network element.

[0005] In a first aspect, an embodiment of the present application provides a method for model distillation training, comprising:

[0006] If the second device supports the distillation training neural network model of the first device, the first device designs the first neural network model;

[0007] The first device sends first configuration information to the second device; wherein the first configuration information is used by the second device to configure the first reference neural network model;

[0008] The first device sends first data information to the second device; wherein the first data information includes first sample data used by the first reference neural network model for distillation training;

[0009] The first device receives first indication information returned by the second device, and uses the first indication information to train the first neural network model; the first indication information is information obtained by inputting the first sample data into the first reference neural network model.

[0010] In an embodiment of the present application, a first device designs a first neural network model, and sends first configuration information and first data information to a second device, so that the first device can personalize the design and training of the first neural network model, achieve a balance between the size of the first neural network model and the accuracy of the first neural network model, improve the efficiency of distillation training, solve the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and solve the problem that the first device deploys a small network sent by the second device and obtained through unified distillation training and is not compatible with the first device. That is to say, the problems caused by the incompatibility between the small neural network model obtained by unified distillation training of the central network element and the edge network element in the prior art can be solved, including: the categories supported by the small neural network model are all categories, while the edge network element only pays attention to a few specific categories. When the computing power and storage resources required by the small neural network model of all categories exceed the capabilities of the edge network element, the small neural network model will occupy the storage space and running space of the edge network element, affecting the edge network element to complete other tasks, thereby reducing the working performance of the edge network element. Technical problem; in addition, due to the limited computing power and storage resources of the edge network element, the edge network element may not be able to fully run the small neural network model of all categories, resulting in the small neural network model performing functions such as portrait recognition and traffic recognition. The degree of completion and accuracy are not high. Technical problem.

[0011] In a possible implementation manner, before the first device designs the first neural network model, the method further includes:

[0012] The first device sends a first training request to the second device; wherein the first training request includes a first training type ID, and the first training type ID is used to indicate the function type of the neural network model to be distilled and trained by the first device;

[0013] If the second training type ID is consistent with the first training type ID, the first device receives a first response sent by the second device; wherein the first response includes first storage information; the first storage information is used to indicate the size of the storage space occupied by the first reference neural network model; the second training type ID is used to indicate the function type of the neural network model supported by the second device for distillation training;

[0014] The first device sends first inquiry information to the second device based on the first storage information; wherein the first inquiry information includes first distillation inquiry information and first distillation capability information; the first distillation inquiry information is used by the first device to inquire whether the second device supports distillation training of the neural network model and, if supporting the distillation training of the neural network model of the first device, the way to support distillation training of the neural network model; the first distillation capability information includes one or more of the first device's design capability for the neural network model, the distillation training capability for the neural network model, and whether there is distillation training data;

[0015] The first device receives a second response from the second device; wherein the second response is used to indicate whether the second device supports the distillation training neural network model of the first device and, if supporting the distillation training neural network model of the first device, a method of supporting the distillation training neural network model.

[0016] In the embodiment of the present application, the first device sends a first training request to the second device, and the second device provides the first device with a corresponding reference neural network model according to the first training request, which is beneficial for the first device to design the first neural network model. In addition, a balance is achieved between the size of the first neural network model and the accuracy of the first neural network model, and the efficiency of distillation training is improved. The problem that the first device cannot store the large network sent by the second device due to its limited storage resources is solved, and the problem that the first device deploys the small network obtained by the unified distillation training sent by the second device and is not compatible with the first device is also solved.

[0017] In a possible implementation manner, before the first device sends the first configuration information to the second device, the method further includes:

[0018] The first device sends a first focus category list to the second device; wherein the first focus category list includes a set of categories focused on by the first device's distillation training; the set of categories focused on by the first device's distillation training in the first focus category list is a subset of the set of categories in the first category list; the first category list includes a set of preset categories for the first reference neural network model.

[0019] In an embodiment of the present application, a first device sends a first focus category list to a second device, which is beneficial for the second device to obtain data for distillation training based on the first data information sent by the first device, and then filter the data obtained after distillation training according to the categories in the first focus category list to obtain first indication information. The first indication information corresponds to the categories in the first focus category list, which is beneficial for the first device to distill and train an adapted first neural network model based on the first indication information, and further reduce communication overhead without affecting the training effect of the first neural network model, thereby saving network resources for separate distillation training.

[0020] In a possible implementation, the first indication information is obtained by the second device calculating and processing the first sample data through the first reference neural network model, and filtering the processed first sample data based on the first focus category list.

[0021] In an embodiment of the present application, the second device filters the data obtained after distillation training according to the categories in the first focus category list to obtain first indication information, and the first indication information corresponds to the categories in the first focus category list, which is conducive to the first device to distill and train an adapted first neural network model based on the first indication information, and further reduces the communication overhead without affecting the training effect of the first neural network model, saving network resources for separate distillation training.

[0022] In a possible implementation manner, the first device designs a first neural network model, including:

[0023] The first device sends a first network structure request to the second device; wherein the first network structure request is used by the first device to obtain structural information of the first reference neural network model from the second device;

[0024] The first device receives a first structure request response sent by the second device; wherein the first structure request response includes structure information of the first reference neural network model;

[0025] The first device designs a first neural network model based on structural information of the first reference neural network model.

[0026] In the embodiment of the present application, after the first device sends a first network structure request to the second device, it receives a first structure request response sent by the second device and obtains the structure information of the first reference neural network model. This is beneficial for the first device to design the first neural network model according to the structure information of the first reference neural network model, achieves a balance between the size of the first neural network model and the accuracy of the first neural network model, improves the efficiency of distillation training, solves the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and solves the problem that the first device deploys the small network obtained by the unified distillation training sent by the second device and is not compatible with the first device.

[0027] In a possible implementation manner, after the first device sends the first data information to the second device, the method further includes:

[0028] The first device sends a first training result notification to the second device; wherein the first training result notification includes completion information of the first neural network model trained by the first device through distillation and feedback information of the first neural network model trained by distillation.

[0029] In an embodiment of the present application, a first device sends a first training result notification to a second device, wherein the first training result notification is used to inform the second device whether the training of the first neural network model is completed and to feed back information about the trained first neural network model to the second device. The second device determines whether to release the communication resources for the distillation training between the first device and the second device based on the first training result notification to avoid occupying other channel resources of the first device and the second device.

[0030] In a second aspect, an embodiment of the present application provides a method for model distillation training, comprising:

[0031] The second device receives the first configuration information sent by the first device, and uses the first configuration information to configure the first reference neural network model;

[0032] The second device receives first data information sent by the first device; wherein the first data information includes first sample data used for distillation training by the first reference neural network model;

[0033] The second device sends first indication information to the first device; wherein the first indication information is used by the first device to train a first neural network model; the first indication information is information obtained by inputting the first sample data into the first reference neural network model.

[0034] In the embodiment of the present application, the second device receives the first configuration information and the first data information sent by the first device, which is conducive to the second device obtaining the first indication information based on the first configuration information and the first data information sent by the first device, so that the first device uses the first indication information to train the first neural network model. This achieves a balance between the size of the first neural network model and the accuracy of the first neural network model, improves the efficiency of distillation training, solves the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and solves the problem that the first device deploys the small network obtained by the unified distillation training sent by the second device and is not compatible with the first device.

[0035] In a possible implementation manner, before the second device receives the first configuration information sent by the first device and uses the first configuration information to configure the first reference neural network model, the method further includes:

[0036] The second device receives a first training request sent by the first device; wherein the first training request includes a first training type ID, and the first training type ID is used to indicate a function type of a neural network model to be distilled and trained by the first device;

[0037] If the second training type ID is consistent with the first training type ID, the second device generates a first response based on the first training request, and sends the first response to the first device; wherein the second training type ID is used to indicate the function type of the distillation training neural network model supported by the second device; the first response includes first storage information; the first storage information is used to indicate the size of the storage space occupied by the first reference neural network model;

[0038] The second device receives the first inquiry information sent by the first device; wherein the first inquiry information includes first distillation inquiry information and first distillation capability information; the first distillation inquiry information is used by the first device to inquire whether the second device supports distillation training of the neural network model and, if supporting the distillation training of the neural network model of the first device, the way to support distillation training of the neural network model; the first distillation capability information includes one or more of the first device's design capability for the neural network model, the distillation training capability for the neural network model, and whether there is distillation training data;

[0039] The second device generates a second response based on the first distillation query information, and sends the second response to the first device; wherein the second response is used to indicate whether the second device supports the distillation training neural network model of the first device and, if so, the way to support the distillation training neural network model.

[0040] In the embodiment of the present application, the second device receives the first training request and the first query information sent by the first device, and sends a corresponding response to the first device, which is beneficial for the first device to design the first neural network model. In addition, a balance is achieved between the size of the first neural network model and the accuracy of the first neural network model, the efficiency of distillation training is improved, and the problem that the first device cannot store the large network sent by the second device due to its limited storage resources is solved. It also solves the problem that the first device deploys the small network obtained by the unified distillation training sent by the second device and is not compatible with the first device.

[0041] In a possible implementation manner, before the second device receives the first configuration information sent by the first device, the method further includes:

[0042] The second device receives a first focus category list sent by the first device; wherein the first focus category list includes a set of categories focused on by the first device's distillation training; in the first focus category list, the set of categories focused on by the first device's distillation training is a subset of the set of categories in the first category list; the first category list includes a set of preset categories for the first reference neural network model.

[0043] In an embodiment of the present application, a second device receives a first attention category list sent by a first device, which facilitates the second device to obtain first indication information based on the first attention category list, wherein the first indication information includes indication information related to the first device attention category, and the first device uses the first indication information to train a first neural network model. After the training, the first neural network model is a neural network model that is compatible with the first device, thereby achieving a balance between the size of the first neural network model and the accuracy of the first neural network model, improving the efficiency of distillation training, and solving the problem that the first device cannot store a large network sent by the second device due to its limited storage resources, and also solving the problem that the first device deploys a small network sent by the second device and obtained through unified distillation training, which is not compatible with the first device.

[0044] In a possible implementation, the first indication information is obtained by the second device calculating and processing the first sample data through the first reference neural network model, and filtering the processed first sample data based on the first focus category list.

[0045] In an embodiment of the present application, the second device sends the first indication information obtained based on the first focus category list to the first device, which is conducive to the first device obtaining a more adapted first neural network model based on the first indication information, thereby solving the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and also solving the problem that the small network obtained by the unified distillation training sent by the second device deployed by the first device is not compatible with the first device.

[0046] In a possible implementation manner, before the second device receives the first configuration information sent by the first device, the method further includes:

[0047] The second device receives a first network structure request sent by the first device; wherein the first network structure request is used by the first device to obtain structural information of the first reference neural network model from the second device;

[0048] The second device sends a first structure request response to the first device based on the first network structure request; wherein the first structure request response includes structural information of the first reference neural network model.

[0049] In an embodiment of the present application, the second device sends structural information of the first reference neural network model to the first device, which is beneficial for the first network to design the first neural network model according to the structural information of the first reference neural network model, achieves a balance between the size of the first neural network model and the accuracy of the first neural network model, improves the efficiency of distillation training, solves the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and solves the problem that the small network obtained by the unified distillation training sent by the second device deployed by the first device is not compatible with the first device.

[0050] In a possible implementation manner, after the second device receives the first data information sent by the first device, the method further includes:

[0051] The second device receives a first training result notification sent by the first device; wherein the first training result notification includes completion information of the first neural network model trained by the first device through distillation and feedback information of the first neural network model trained by the first device through distillation.

[0052] In the embodiment of the present application, the second device receives the first result notification, which is beneficial for the second device to determine whether to release the distillation training communication resources between the first device and the second device based on the first training result notification, so as to avoid occupying other channel resources of the first device and the second device.

[0053] In a third aspect, an embodiment of the present application provides a method for model distillation training, comprising:

[0054] The third device receives registration information sent by the second device; wherein the registration information includes a third training type ID, a third neural network model ID, second storage information, a second category list, and training response information; the training response information is used to indicate whether the second device supports the distillation training neural network model and, if the distillation training neural network model is supported, the way in which the distillation training neural network model is supported; the third training type ID is used to indicate the function type of the distillation training neural network model supported by the second device;

[0055] The third device receives a second training request sent by the first device; wherein the second training request includes a fourth training type ID, second distillation query information and second distillation capability information; the fourth training type ID is used to indicate the function type of the neural network model to be distilled and trained by the first device;

[0056] If the fourth training type ID is consistent with the third training type ID, the third device generates a third response based on the second training request, and sends the third response to the first device; wherein the third response includes training response information, a third neural network model ID, second storage information, and a second category list;

[0057] The third device receives a distillation notification sent by the first device, wherein the distillation result notification is used to indicate whether the first device and the second device are matched successfully.

[0058] In the embodiment of the present application, the third device receives the registration information sent by the second device and the third device receives the second training request sent by the first device, which is beneficial for the third device to establish a distillation training communication connection for the first device and the second device by analyzing the registration information and the second training request, thereby achieving a balance between the size of the second neural network model and the accuracy of the second neural network model, thereby improving the efficiency of distillation training.

[0059] In a fourth aspect, an embodiment of the present application provides a method for model distillation training, comprising:

[0060] The first device sends a second training request to the third device; wherein the second training request includes a fourth training type ID, second distillation query information, and second distillation capability information; wherein the fourth training type ID is used to indicate the function type of the neural network model to be distilled and trained by the first device;

[0061] If the fourth training type ID is consistent with the third training type ID, the first device receives a third response sent by the third device; wherein the third response includes training response information, a third neural network model ID, second storage information and a second category list; the third training type ID is used to indicate that the second device supports the function type of the distillation training neural network model;

[0062] The first device sends a distillation notification to the third device; wherein the distillation result notification is used to indicate whether the first device and the second device are matched successfully.

[0063] In an embodiment of the present application, the first device sends a second training request to the third device, and the third device sends a third response to the first device based on the second training request, which is beneficial for the third device to establish a distillation training communication connection for the first device and the second device, thereby improving the efficiency of distillation training.

[0064] In a possible implementation manner, after the first device sends the distillation notification to the third device, the method further includes:

[0065] The first device designs a second neural network model;

[0066] The first device sends second configuration information to the second device; wherein the second configuration information is used by the second device to configure the second reference neural network model;

[0067] The first device sends second data information to the second device; wherein the second data information includes second sample data used for distillation training by the second reference neural network model;

[0068] The first device receives second indication information returned by the second device, and uses the second indication information to train the second neural network model; the second indication information is information obtained by inputting the second sample data into the second reference neural network model.

[0069] In the embodiment of the present application, after the first device and the second device establish a distillation training communication connection, the first device designs a second neural network model, and the first device and the second device perform a distillation training communication process of sending and receiving second configuration information and second data information. This is beneficial for the second device to obtain the second indication information that is compatible with the first device, thereby facilitating the first device to use the second indication information to train a second neural network model that is compatible with it.

[0070] In a possible implementation manner, before the first device sends the second configuration information to the second device, the method further includes:

[0071] The first device sends a second focus category list to the second device; wherein the second focus category list includes a set of categories focused on by the distillation training of the first device; in the second focus category list, the set of categories focused on by the distillation training of the first device is a subset of the set of categories in the second category list; the second category list includes a set of preset categories of the second reference neural network model.

[0072] In an embodiment of the present application, a first device sends a second focus category list to a second device, which is beneficial for the second device to obtain second indication information based on the second focus category list, the second indication information corresponds to the category in the second focus category list, and the first device uses the second indication information to train a second neural network model to obtain a neural network model that is compatible with the first device, thereby achieving a balance between the size of the second neural network model and the accuracy of the second neural network model, improving the efficiency of distillation training, and solving the problem that the first device cannot store a large network issued by the second device due to its limited storage resources, and also solving the problem that the first device deploys a small network issued by the second device and obtained through unified distillation training, which is not compatible with the first device.

[0073] In a possible implementation, the second indication information is obtained by the second device calculating and processing the second sample data through the second reference neural network model, and filtering the processed second sample data based on the second focus category list.

[0074] In an embodiment of the present application, the second device obtains second indication information based on the second focus category list, and the second indication information corresponds to a category in the second focus category list. The first device uses the second indication information to train the second neural network model, and can obtain a neural network model that is compatible with the first device, thereby achieving a balance between the size of the second neural network model and the accuracy of the second neural network model, improving the efficiency of distillation training, and solving the problem that the first device cannot store the large network issued by the second device due to its limited storage resources. It also solves the problem that the first device deploys a small network issued by the second device and obtained through unified distillation training, which is not compatible with the first device.

[0075] In a possible implementation, the first device designs a second neural network model, including:

[0076] The first device sends a second network structure request to the second device; wherein the second network structure request is used by the first device to obtain structural information of a second reference neural network model from the second device;

[0077] The first device receives a second structure request response sent by the second device; wherein the second structure request response includes structure information of the second reference neural network model;

[0078] The first device designs a second neural network model based on structural information of a second reference neural network model.

[0079] In the embodiment of the present application, after the first device sends a second network structure request to the second device, it receives a second structure request response sent by the second device and obtains the structure information of the second reference neural network model. This is beneficial for the first device to design the second neural network model according to the structure information of the first reference neural network model, achieves a balance between the size of the second neural network model and the accuracy of the second neural network model, improves the efficiency of distillation training, solves the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and solves the problem that the first device deploys the small network obtained by the unified distillation training sent by the second device and is not compatible with the first device.

[0080] In a possible implementation manner, after the first device sends the second data information to the second device, the method further includes:

[0081] The first device sends a second training result notification to the second device; wherein the second training result notification includes completion information of the distillation training of the second neural network model by the first device and feedback information of the distillation training of the second neural network model.

[0082] In an embodiment of the present application, the first device sends a second training result notification to the second device, and the second training result notification is used to inform the second device whether the second neural network model has been trained and to feed back information about the trained second neural network model to the second device. The second device determines whether to release the distillation training communication resources between the first device and the second device based on the second training result notification to avoid occupying other channel resources of the first device and the second device.

[0083] In a fifth aspect, an embodiment of the present application provides a method for model distillation training, comprising:

[0084] The second device sends registration information to the third device; wherein the registration information includes a third training type ID, a third neural network model ID, second storage information, a second category list, and training response information; the training response information is used to indicate whether the second device supports the distillation training neural network model and, if the distillation training neural network model is supported, the way in which the distillation training neural network model is supported;

[0085] The second device receives second configuration information sent by the first device; wherein the second configuration information is used by the second device to configure the second reference neural network model;

[0086] The second device receives second data information sent by the first device; wherein the second data information includes second sample data used for distillation training by the second reference neural network model;

[0087] The second device sends second indication information to the first device; wherein the second indication information is used by the first device to train a second neural network model; the second indication information is information obtained by inputting the second data into the second reference neural network model.

[0088] In the embodiment of the present application, the second device sends registration information to the third device, which is conducive to the third device to quickly and effectively establish a distillation training connection between the first device and the second device, thereby improving the efficiency of distillation training. The second device receives the second configuration information and the second data information sent by the first device, which is conducive to the second device obtaining the second indication information based on the second configuration information and the second data information, so that the first device uses the second indication information to train the second neural network model. Thereby, a balance is achieved between the size of the second neural network model and the accuracy of the second neural network model, the efficiency of distillation training is improved, and the problem that the first device cannot store the large network sent by the second device due to its limited storage resources is solved. It also solves the problem that the small network obtained by the first device deploying the unified distillation training sent by the second device is not compatible with the first device.

[0089] In a possible implementation manner, before the second device receives the second configuration information sent by the first device, the method further includes:

[0090] The second device receives a second focus category list sent by the first device; wherein the second focus category list includes a set of categories focused on by the first device for distillation training; in the second focus category list, the set of categories focused on by the first device for distillation training is a subset of the set of categories in the second category list; the second category list includes a set of preset categories for the second reference neural network model.

[0091] In an embodiment of the present application, a second device receives a second attention category list sent by a first device, which facilitates the second device to obtain second indication information based on the second attention category list, wherein the second indication information includes indication information related to the first device attention category, and the first device uses the second indication information to train a first neural network model. After the training, the first neural network model is a neural network model that is compatible with the first device, thereby achieving a balance between the size of the first neural network model and the accuracy of the second neural network model, improving the efficiency of distillation training, and solving the problem that the first device cannot store a large network sent by the second device due to its limited storage resources, and also solving the problem that the first device deploys a small network sent by the second device and obtained through unified distillation training, which is not compatible with the first device.

[0092] In a possible implementation, the second indication information is obtained by the second device calculating and processing the second sample data through the second reference neural network model, and filtering the processed second sample data based on the second focus category list.

[0093] In an embodiment of the present application, the second device sends the second indication information obtained based on the second focus category list to the first device, which is beneficial for the first device to obtain a more suitable second neural network model based on the second indication information, thereby solving the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and also solves the problem that the small network obtained by the unified distillation training sent by the second device deployed by the first device is not compatible with the first device.

[0094] In a possible implementation manner, before the second device receives the second configuration information sent by the first device, the method further includes:

[0095] The second device receives a second network structure request sent by the first device; wherein the second network structure request is used by the first device to obtain structural information of the second reference neural network model from the second device;

[0096] The second device sends a second structure request response to the first device based on the second network structure request; wherein the second structure request response includes structural information of the second reference neural network model.

[0097] In an embodiment of the present application, the second device sends structural information of the second reference neural network model to the first device, which is beneficial for the second network to design the second neural network model according to the structural information of the second reference neural network model, achieves a balance between the size of the second neural network model and the accuracy of the second neural network model, improves the efficiency of distillation training, solves the problem that the first device cannot store the large network sent by the second device due to its limited storage resources, and solves the problem that the small network obtained by the unified distillation training sent by the second device deployed by the first device is not compatible with the first device.

[0098] In a possible implementation manner, after the second device receives the second data information sent by the first device, the method further includes:

[0099] The second device receives a second training result notification sent by the first device; wherein the second training result notification includes completion information of the distillation training of the second neural network model by the first device and feedback information of the distillation training of the second neural network model.

[0100] In the embodiment of the present application, the second device receives the second result notification, which is beneficial for the second device to determine whether to release the distillation training communication resources between the first device and the second device based on the second training result notification, so as to avoid occupying other channel resources of the first device and the second device.

[0101] In a sixth aspect, an embodiment of the present application provides a model distillation training device, wherein the model distillation training device is a first device, comprising:

[0102] A first neural network model design unit, configured to design a first neural network model if the second device supports the distillation training neural network model of the first device;

[0103] A first configuration information sending unit, configured to send first configuration information to the second device;

[0104] A first data information sending unit, configured to send first data information to the second device;

[0105] A first indication information receiving unit, configured to receive first indication information returned by the second device;

[0106] The first neural network model training unit is used to train the first neural network model using the first indication information.

[0107] In a possible implementation, the model distillation training device further includes:

[0108] A first training request sending unit, configured to send a first training request to the second device before the first neural network model design unit designs the first neural network model;

[0109] A first response receiving unit, configured to receive a first response sent by the second device after the first training request sending unit sends the first training request to the second device if the second training type ID is consistent with the first training type ID;

[0110] A first inquiry information sending unit, configured to send first inquiry information to the second device based on the first storage information;

[0111] The second response receiving unit is configured to receive a second response from the second device after the first inquiry information sending unit sends the first inquiry information to the second device.

[0112] In a possible implementation, the model distillation training device further includes:

[0113] The first attention category list sending unit is configured to send a first attention category list to the second device before the first configuration information sending unit sends the first configuration information to the second device.

[0114] In a possible implementation, the first neural network model design unit further includes:

[0115] A first network structure request sending unit, configured to send a first network structure request to the second device before the first neural network model design unit designs the first neural network model;

[0116] The first structure request response receiving unit is used to receive the first structure request response sent by the second device after the first network structure request sending unit sends the first network structure request to the second device.

[0117] In a seventh aspect, an embodiment of the present application provides a model distillation training device, wherein the model distillation training device is a second device, comprising:

[0118] A first configuration information receiving unit, configured to receive first configuration information sent by the first device;

[0119] A first configuration unit, configured to configure the first reference neural network model using the first configuration information;

[0120] A first data information receiving unit, configured to receive first data information sent by the first device;

[0121] The first indication information sending unit is used to send first indication information to the first device.

[0122] In a possible implementation, the model distillation training device further includes:

[0123] A first training request receiving unit, configured to receive a first training request sent by the first device before the first parameter configuration unit configures the first reference neural network model;

[0124] a first response sending unit, configured to generate a first response based on the first training request if the second training type ID is consistent with the first training type ID, and send the first response to the first device;

[0125] A first inquiry information receiving unit, configured to receive first inquiry information sent by the first device;

[0126] The second response sending unit is configured to send the second response to the first device after the first inquiry information receiving unit receives the first inquiry information sent by the first device.

[0127] In a possible implementation, the model distillation training device further includes:

[0128] The first attention category list receiving unit is configured to receive a first attention category list sent by the first device before the first configuration information receiving unit receives the first configuration information sent by the first device.

[0129] In a possible implementation, the model distillation training device further includes:

[0130] A first network structure request receiving unit, configured to receive a first network structure request sent by the first device before the first configuration information receiving unit receives the first configuration information sent by the first device;

[0131] The first structure request response sending unit is configured to send a first structure request response to the first device.

[0132] In an eighth aspect, an embodiment of the present application provides a model distillation training device, wherein the model distillation training device is a third device, comprising:

[0133] A registration information receiving unit, configured to receive registration information sent by a second device;

[0134] A second training request receiving unit, configured to receive a second training request sent by the first device;

[0135] a third response sending unit, configured to generate a third response based on the second training request if the fourth training type ID is consistent with the third training type ID, and send the third response to the first device;

[0136] The distillation notification receiving unit is used to receive the distillation notification sent by the first device.

[0137] In a ninth aspect, an embodiment of the present application provides a model distillation training device, wherein the model distillation training device is a first device, comprising:

[0138] A second training request sending unit, configured to send a second training request to a third device;

[0139] a third response receiving unit, configured to receive a third response sent by the third device if the fourth training type ID is consistent with the third training type ID;

[0140] A distillation notification sending unit is used to send a distillation notification to the third device.

[0141] In a possible implementation, the model distillation training device further includes:

[0142] A second neural network model design unit, configured to design a second neural network model after the distillation notification sending unit sends the distillation notification to the third device;

[0143] A second configuration information sending unit, used to send second configuration information to the second device;

[0144] A second data information sending unit, configured to send second data information to the second device;

[0145] The second indication information receiving unit is configured to receive second indication information returned by the second device.

[0146] In a possible implementation, the model distillation training device further includes:

[0147] The second attention category list sending unit is configured to send a second attention category list to the second device after the second configuration information sending unit sends the second configuration information to the second device.

[0148] In a possible implementation, the second neural network model design unit further includes:

[0149] A second network structure request sending unit, used to send a second network structure request to the second device;

[0150] The second structure request response receiving unit is configured to receive a second structure request response sent by the second device.

[0151] In a tenth aspect, an embodiment of the present application provides a model distillation training device, wherein the model distillation training device is a second device, comprising:

[0152] A registration information sending unit, used to send registration information to a third device;

[0153] A second configuration information receiving unit, configured to receive second configuration information sent by the first device;

[0154] A second data information receiving unit, configured to receive second data information sent by the first device;

[0155] The second indication information sending unit is used to send second indication information to the first device.

[0156] In a possible implementation, the model distillation training device further includes:

[0157] The second attention category list receiving unit is configured to receive the second attention category list sent by the first device before the second configuration information receiving unit receives the second configuration information sent by the first device.

[0158] In a possible implementation, the model distillation training device further includes:

[0159] A second network structure request receiving unit, configured to receive a second network structure request sent by the first device before the second configuration information receiving unit receives the second configuration information sent by the first device;

[0160] The second structure request response sending unit is used to send a second structure request response to the first device based on the second network structure request.

[0161] In an eleventh aspect, an embodiment of the present application provides a model distillation training device, including a processor and a memory;

[0162] In which, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method in the above-mentioned first aspect and its various possible implementation methods.

[0163] In a twelfth aspect, an embodiment of the present application provides a model distillation training device, including a processor and a memory;

[0164] In which, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method in the above-mentioned second aspect and its various possible implementation methods.

[0165] In a thirteenth aspect, an embodiment of the present application provides a model distillation training device, including a processor and a memory;

[0166] In which, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method in the above-mentioned third aspect and its various possible implementation methods.

[0167] In a fourteenth aspect, an embodiment of the present application provides a model distillation training device, including a processor and a memory;

[0168] In which, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method in the above-mentioned fourth aspect and its various possible implementation methods.

[0169] In a fifteenth aspect, an embodiment of the present application provides a model distillation training device, including a processor and a memory;

[0170] In which, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method in the above-mentioned fifth aspect and its various possible implementation methods.

[0171] In the sixteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the first device implements the model distillation training method in the above-mentioned first and fourth aspects and their various possible implementations, or the second device implements the model distillation training method in the above-mentioned second and fifth aspects and their various possible implementations, or the third device implements the model distillation training method in the above-mentioned third aspect and its various possible implementations.

[0172] In the seventeenth aspect, an embodiment of the present application provides a computer program, which includes instructions. When the computer program is executed by a computer, it enables a first device to execute the process executed by the first device in the above-mentioned first and fourth aspects and their various possible implementations, or a second device to execute the process executed by the second device in the above-mentioned second and fifth aspects and their various possible implementations, or enables a third device to execute the process executed by the third device in the above-mentioned third aspect and its various possible implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Figure 1 It is a schematic diagram of deploying a neural network in a central network element and an edge network element provided in an embodiment of the present application;

[0174] Figure 2 This is a schematic diagram of a model distillation provided in an embodiment of the present application;

[0175] Figure 3 It is a schematic diagram of a model distillation training system architecture provided in an embodiment of the present application;

[0176] Figure 4 It is a schematic diagram of a flow chart of model distillation training provided in an embodiment of the present application;

[0177] Figure 5 is a schematic diagram of another flow chart of model distillation training provided in an embodiment of the present application;

[0178] Figure 6 is a schematic diagram of another flow chart of model distillation training provided in an embodiment of the present application;

[0179] Figure 7 is a schematic diagram of another flow chart of model distillation training provided in an embodiment of the present application;

[0180] Figure 8 is a structural schematic diagram of a model distillation training device provided in an embodiment of the present application;

[0181] Fig. 9 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0182] Fig.10 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0183] Fig.11 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0184] Fig.12 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0185] Fig.13 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0186] Fig.14 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0187] Fig.15 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0188] Fig.16 is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application;

[0189] Fig.17 It is a structural schematic diagram of another model distillation training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0190] The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0191] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0192] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0193] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0194] The first device and the second device described in the claims and the specification of this application are network devices or user devices that support AI model training and neural model distillation training, and the third device is a switching device that supports inter-device communication processes and receives device registration information.

[0195] See also Figure 2 , Figure 2 Schematic diagram of a model distillation provided in the embodiment of the present application. Figure 2As shown, the model distillation schematic diagram includes a teacher network and a student network. The teacher network is a trained neural network with a complex network structure, usually with more nodes and layers, and has strong computing and processing capabilities and a large amount of data resources. The student network is a neural network to be trained, which is a small network with a simpler network structure and usually with fewer nodes and layers. The teacher network and the student network input the same X, and the X is processed and calculated by the teacher network, and after being processed by the normalized exponential function Softmax 1 function, the instruction information of the distillation training is obtained. In this embodiment, the instruction information of the distillation training is the soft target 1 output by the teacher network, which can also be understood as the "knowledge" obtained by the teacher network processing X. The X is processed and calculated by the student network, and after being processed by the Softmax2 function, soft target 2 is output. The student network compares and analyzes soft target 1 and soft target 2 to obtain a loss function Fn1. The X is processed and calculated by the student network, and after being processed by the Softmax3 function, hard target 1 is output. The student network compares and analyzes hard target 1 and hard target 2 to obtain a loss function Fn2; wherein the hard target 2 is the preset output result of X. The loss functions Fn1 and Fn2 are used to indicate the degree of deviation between soft target 1 and soft target 2, and hard target 1 and hard target 2. The student network adjusts the student network according to the loss functions Fn1 and Fn2, so that after the student network processes and calculates X, the loss functions Fn1 and Fn2 are reduced to a preset value, so that when the input is the same X, the output of the student network is close to the output of the teacher network, achieving the purpose of training the student network by teacher model distillation. In this way, the teacher network cultivates the generalization ability of the student network and realizes the transfer of the "knowledge" of the teacher network to the student network. The process of neural network training of the teacher network on the student network for a certain functional type is as follows:

[0196] The teacher network is a trained neural network that can realize multiple functions such as image recognition and traffic recognition. This embodiment takes the teacher network as an example that it can realize image recognition and perform model distillation so that the student network can realize the function of image recognition. This embodiment takes the input X of the teacher network and the student network as an example of a car picture. X is input to the teacher network, and the X is processed by the relevant algorithm through each network layer in the teacher network. The processing process is mainly to compare X with each category preset in the teacher network. Among them, the preset categories can be things of different types from the input X, things with a large difference in appearance and shape from the input X, or other categories such as things with different attributes from the input X. This embodiment takes the teacher network preset category as a thing of different types from the input X as an example. The preset categories in the teacher network are four categories of vehicles, trains, airplanes, and ships. The preset categories in the teacher network are recorded as X1, X2, X3, and X4 respectively. The teacher network compares the input X with the preset categories X1, X2, X3 and X4, and then outputs the similarities Y1, Y2, Y3 and Y4 between X and X1, X and X2, X and X3 and X and X4 respectively. Since the teacher network is a trained neural network, the teacher network will determine that X and X1 belong to the same category (corresponding to the same category of cars and vehicles). Therefore, the similarity between X and X1 output by the teacher network is the highest, that is, the value of Y1 is the highest (close to 100%); the similarities between X and X2, X and X3 and X and X4 output by the teacher network are very low, that is, the values ​​of Y2, Y3 and Y4 are extremely low (close to 0%), the difference between the values ​​of Y1 and Y2, Y3 and Y4 is too large, the information output by the teacher network is almost only the information of Y1, and the information of Y2, Y3 and Y4 is almost 0, so the amount of information output by the teacher network is small. In order to increase the amount of information output by the teacher network, Y1, Y2, Y3 and Y4 are processed by the Softmax 1 function to reduce the numerical difference between Y1 and Y2, Y3 and Y4. The expression of the Softmax 1 function is:

[0197]

[0198] Among them, Z i Yes i After being processed by the Softmax 1 function, the output information of the teacher network; T is the distillation temperature information (the larger T is, the higher the output Z is). i The flatter the distribution is, the more it is smoothed, which helps retain similar information. If T approaches infinity, it is a uniform distribution. The value of T is based on the difference in network structure between the teacher network and the student network.

[0199] After Y1, Y2, Y3 and Y4 are processed by the Softmax 1 function, Z1, Z2, Z3 and Z4 are output respectively. The distribution of Z1, Z2, Z3 and Z4 is relatively gentle (the value of Z1 is the highest, and the difference between the values ​​of Z2, Z3 and Z4 and the value of Z1 is narrowed). By processing Y1, Y2, Y3 and Y4 by the Softmax 1 function, the amount of information of Y2, Y3 and Y4 is increased, and then the amount of information output by the teacher network is increased. Z1, Z2, Z3 and Z4 are soft targets 1 output by the teacher network.

[0200] It should be noted that the teacher network is a trained neural network, and the similarity between the teacher network output X and its similar categories has a high confidence level (in this embodiment, cars and vehicles have a high similarity), so that Y1 has a large numerical difference from Y2, Y3, and Y4. For example, in this embodiment, for the input of a car picture, the teacher network will determine that the similarity between the car picture and the vehicle category is very high, and the teacher network output Y1 may be 95%; the teacher network determines that the similarity between the car picture and the train, airplane, and ship is very low, and the teacher network outputs Y2, Y3, and Y4 may be 3%, 1%, and 1%, respectively. In this case, the "knowledge" of the comparison between the teacher network input X and the categories of X1, X2, X3, and X4 is difficult to transfer to the student network, because their similarity is almost close to 0 and has very little information. Training the student network in this way is not conducive to strengthening the generalization ability of the student network and affects the effect of distillation training. Therefore, the teacher network processes Y1, Y2, Y3 and Y4 through the Softmax 1 function, which can make the numerical distribution of Y1, Y2, Y3 and Y4 smoother, increase the amount of information of Y2, Y3 and Y4, and transfer the "knowledge" of the teacher network learning the comparison between X and X1, X2, X3 and X4 to the student network, thereby enhancing the generalization ability of the student network.

[0201] Similarly, X is input to the student network, and X is processed by the relevant algorithms in each network layer of the student network and compared with each category preset in the student network. The student network compares the input X with the preset categories X1, X2, X3 and X4 respectively, and then outputs the similarities H1, H2, H3 and H4 between X and X1, X and X2, X and X3 and X and X4 respectively. Since the student network is a neural network to be trained, the student network is not capable of comparing the input X with its preset category, so that the similarity between X output by the student network and its preset category is greatly different from the preset result (for example, the value of H4 is extremely large, close to 100%; the values ​​of H2, H3 and H1 are extremely small, close to 0). Similar to the teacher network, since the numerical differences between H4 and H2, H3 and H1 are too large, the information output by the student network is almost only the information of H4, and the amount of information of H2, H3 and H1 is almost 0. In order to increase the amount of information output by the student network, H1, H2, H3 and H4 are processed by the Softmax 2 function to reduce the numerical difference between H4 and H2, H3 and H1, thereby increasing the amount of information output by the student network; wherein the distillation temperature information T is the same as that used by the teacher network. After being processed by the Softmax 2 function, H1, H2, H3 and H4 output M1, M2, M3 and M4 respectively, and the distribution of M1, M2, M3 and M4 is relatively flat (the value of M4 is the highest, but the difference between the values ​​of M2, M3 and M1 and M4 is reduced), and M1, M2, M3 and M4 are soft targets 2 output by the student network.

[0202] The student network compares the soft target 2 it outputs with the soft target 1 output by the teacher network, and obtains the loss function Fn1 based on the degree of deviation between the soft target 2 and the soft target 1. In addition, the student network processes its outputs H1, H2, H3 and H4 through the Softmax 3 function to obtain B4. Among them, the distillation temperature information T used in Softmax 3 is 1, and the B4 obtained by processing H1, H2, H3 and H4 through the Softmax 3 function only contains the information of H4, and B4 is the hard target 1 output by the student network. The student network compares and analyzes the hard target 1 it outputs with the hard target 2, and obtains the loss function Fn2 based on the degree of deviation between the hard target 1 and the hard target 2. Among them, the hard target 2 is a preset result that should be obtained after inputting X, and the preset relationship between X and hard target 2 is stored in the student network as local data.

[0203] The student network adjusts its neural network model based on the loss functions Fn1 and Fn2 to reduce the degree of deviation between the output information of the student network and the teacher network, thereby reducing the values ​​of the loss functions Fn1 and Fn2, and then enhancing the generalization ability of the student network and realizing the "knowledge" transfer of the teacher network.

[0204] See also Figure 3 , Figure 3 It is a schematic diagram of the architecture of a model distillation training system provided in an embodiment of the present application. Figure 3 It means that in a wireless scenario, the second device is a radio access network (Radio Access Network, RAN), the first device is a user equipment (User Equipment, UE), and the RAN accesses multiple user equipments. Among them, the large network deployed by the RAN is the first reference neural network model, and the first reference neural network model is a full-category model. The categories in the first reference neural network model include: 3D video, high-definition video transmission, and augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR); the small network deployed by UE1 is the first neural network model, and the categories in the first neural network model are a subset of the categories included in the first reference neural network model, including: 3D video and high-definition video transmission. This embodiment takes UE1 distillation training of the first neural network model as an example so that the first neural network model has the function of traffic identification. The steps of UE1 distillation training of the first neural network model are as follows:

[0205] 1. UE1 sends a first training request to RAN, where the first training request includes a first training type ID, where the first training type ID is used to indicate that the function type of the first neural network model of the distillation training of UE1 is traffic identification.

[0206] 2. RAN receives the first training request and compares the first training type ID with the second training type ID in RAN. If the two are consistent, RAN supports UE1 to distill and train the first neural network model, and sends a first response to UE1 according to the second training type ID. The first response includes first storage information, which is the size of the memory or storage space occupied by the first reference neural network model with traffic identification function.

[0207] 3. UE1 sends a first inquiry message to RAN based on the first response, asking RAN whether it supports UE1 to perform distillation training on the first neural network model, so that the first neural network model has a traffic identification function, and reports the distillation training capability of UE1.

[0208] 4. Based on the first inquiry information, RAN sends a second response to UE1 to inform UE1 that RAN supports UE1 to distill and train the first neural network model so that it can obtain traffic identification function; the method of supporting distillation training is that RAN sends a first indication information to UE1 to guide UE1 to distill and train the first neural network model.

[0209] 5. UE1 designs the first neural network model.

[0210] 6. UE1 sends the first configuration information to RAN, and RAN configures the first reference neural network model based on the first configuration information. The first configuration information includes distillation temperature information T and knowledge type KnowledgeType; the distillation temperature information T is obtained based on the first reference neural network model deployed by RAN and the first neural network model designed and deployed by UE1, and can also be obtained based on an experience library or other methods. The T is used to distill out the information obtained by comparing and analyzing the traffic data to be identified by the first reference neural network model with the preset traffic category in the first reference neural network model. RAN configures the normalized Softmax(T) function based on the distillation temperature information T in the first configuration information, and learns from the knowledge type KnowledgeType that the type of the first indication information is the soft target output by the first reference neural network model after traffic identification. Then, the first indication information is sent to UE1 (this embodiment takes the knowledge type KnowledgeType as the soft target Soft target output by the first reference neural network model as an example).

[0211] 7. UE1 sends first data information to RAN, and the first data information includes first sample data; the first sample data is the training data required for UE1 to perform distillation training on the first neural network model for traffic identification. In addition, the first sample data includes an index Index used to indicate each traffic data. For example, the first sample data includes: traffic data of 3D movies, traffic data of movie transmission, and traffic data of human-computer interaction, and its indexes Index are 1, 2, and 3, respectively, which are used to indicate the traffic data of 3D movies, the traffic data of movie transmission, and the traffic data of human-computer interaction.

[0212] 8. RAN processes the first sample data through the first reference neural network model to obtain first indication information, and sends the first indication information to UE1; the first indication information is information obtained by the first reference neural network model in the process of traffic identification based on the first sample data. The process of RAN obtaining the first indication information is as follows: RAN inputs the first sample data information (traffic data of 3D movies, traffic data of movie transmission, and traffic data of human-computer interaction) into the first reference neural network model, and the first reference neural network model compares and analyzes the traffic data of 3D movies, traffic data of movie transmission, and traffic data of human-computer interaction with the three preset categories of 3D video, high-definition video transmission, and AR / VR in the first reference neural network model, and outputs the similarities A1, A2, and A3 between the traffic data of 3D movies and the three preset categories of traffic of 3D video, high-definition video transmission, and AR / VR; the similarities B1, B2, and B3 between the traffic data of movie transmission and the four preset categories of traffic of 3D video, high-definition video transmission, and AR / VR; and the similarities C1, C2, and C3 between the traffic data of human-computer interaction and the three preset categories of traffic of 3D video, high-definition video transmission, and AR / VR. Since the first reference neural network model is a trained neural network model, it can accurately identify which preset category the input traffic data has the highest similarity with. Therefore, among A1, A2 and A3, the value of A1 is as high as 100%, and the values ​​of A2 and A3 are very small, close to 0; among B1, B2 and B3, the value of B2 is as high as 100%, and the values ​​of B1 and B3 are very small, close to 0; among C1, C2 and C3, the value of C3 is as high as 100%, and the values ​​of C1 and C2 are very small, close to 0. In order to increase the amount of information output by the first reference neural network model and make the numerical distribution of the similarity between the traffic data in each sample data information and the preset category smoother, A1, A2 and A3 are used as the input of the normalized Softmax(T) function, and a1, a2 and a3 are output. The a1, a2 and a3 are soft target 1 output by the first reference neural network model, and the index Index 1 is used to indicate soft target 1 (the index Index of the soft target corresponds to the index Index of the traffic data in the first sample data information). Similarly, the same method is used to process B1~B3, C1~C3 to obtain soft target 2 (b1~b3) and soft target 3 (c1~c3), and the indexes Index2 and 3 are used to indicate soft target 2 and soft target 3 respectively. The soft target 1, soft target 2 and soft target 3 are the first indication information. Then, RAN sends the first indication information to UE1.

[0213] 9. UE1 receives the first indication information, and uses the first indication information to distill and train the first neural network model for the traffic identification function of each category contained therein. The specific process of UE1 using the first indication information to distill and train the first neural network model is as follows: UE1 inputs the first sample data information (traffic data of 3D movies, traffic data of movie transmission, and traffic data of human-computer interaction) into the first neural network model, and the first neural network model compares and analyzes the traffic data of 3D movies, the traffic data of movie transmission, and the traffic data of human-computer interaction with the two preset categories of 3D video and high-definition video transmission in the first neural network model, and outputs the similarities D1 and D2 between the traffic data of 3D movies and the two preset categories of traffic of 3D video and high-definition video transmission; the similarities E1 and E2 between the traffic data of movie transmission and the two preset categories of traffic of 3D video and high-definition video transmission; and the similarities F1 and F2 between the traffic data of human-computer interaction and the two preset categories of traffic of 3D video and high-definition video transmission. Since the first neural network model is a distillation-trained neural network model, the confidence in the output result of traffic identification is very low. For example, the first neural network model may believe that the traffic data of 3D movies is highly similar to the traffic data of high-definition video transmission, the traffic data of movie transmission is highly similar to the traffic data of 3D video, and the traffic data of human-computer interaction is highly similar to the traffic data of high-definition video transmission. Therefore, the value of D2 is very high, close to 100%, and the value of D1 is very low, close to 0; the value of E1 is very high, close to 100%, and the value of E2 is very low, close to 0; the value of F2 is very high, close to 100%, and the value of F1 is very low, close to 0. In order to increase the amount of information output by the first neural network model and make the numerical distribution of the similarity between the flow data in each sample data information and the preset category smoother, D1 and D2 are used as the input of the normalized Softmax(T) function (the distillation temperature information T in the Softmax function is consistent with the T used in the first reference neural network model), and d1 and d2 are output. The d1 and d2 are the soft target 4 output by the first neural network model, and the index Index 1 is used to indicate the soft target 4 (the index Index of the soft target corresponds to the index Index of the flow data in the first sample data information). Similarly, the same method is used to process E1~E2, F1~F2 to obtain soft target 5 (e1~e2) and soft target 6 (f1~f2), and the indexes Index 2 and 3 are used to indicate soft target 5 and soft target 6 respectively.Then UE1 compares and analyzes soft target 4 with soft target 1 in the first indication information, compares and analyzes soft target 5 with soft target 2 in the first indication information, and compares and analyzes soft target 6 with soft target 3 in the first indication information, so as to know the difference between the result of traffic identification output by the first neural network model and the result of traffic identification output by the first reference neural network model, and obtain the loss function Fn1. In addition, the edge network element uses D1~D2, E1~E2 and F1~F2 as the input of Softmax (T=1) respectively, and obtains D2', E1' and F2' (D2' only contains the information of D2, E1' only contains the information of E1, and F3' only contains the information of F3), and D2', E1' and F3' are hard targets 1, hard targets 2 and hard targets 3 output by the first neural network model respectively. Then the hard target 1 is compared and analyzed with the preset result 1, the hard target 2 is compared and analyzed with the preset result 2, and the hard target 3 is compared and analyzed with the preset result 3 to obtain the loss function Fn2. Among them, the preset result 1, preset result 2 and preset result 3 are traffic data of preset categories corresponding to the traffic data information in the first data information stored in UE1. For example, preset result 1 is that the traffic data identification of 3D movies belongs to the 3D video traffic category, preset result 2 is that the traffic data of movie transmission belongs to the traffic category of high-definition video transmission, and preset result 3 is that the traffic data of human-computer interaction belongs to the traffic category of AR / VR. When the preset results are used for the first neural network model to perform traffic identification tasks, the output results thereof are compared to determine whether the results output by the first neural network model are correct. For example, when the output result of the first neural network model is: identifying the traffic data of 3D movies as traffic of the AR / VR category, it is obviously incorrect, because according to preset result 1, the traffic data of 3D movies is identified as belonging to the 3D video traffic category. Then UE1 adjusts the first neural network model based on the loss functions Fn1 and Fn2 to reduce the values ​​of the loss functions Fn1 and Fn2, thereby strengthening the generalization ability of the first neural network model for traffic identification and realizing the distillation training of the first neural network model by UE1.

[0214] 10. UE1 sends a first training result notification to RAN, which is used to notify RAN that the distillation training of the first neural network model is completed, and to feedback information about the first neural network model after the distillation training is completed.

[0215] 11. RAN releases the channel resources of the distillation training communication connection between RAN and UE1.

[0216] In this embodiment, before UE1 distills and trains the neural network model, it needs to establish a communication connection for distillation training with RAN. Through the interaction of distillation training information between UE1 and RAN, RAN uses the first configuration information sent by UE1 to configure the first reference neural network model; wherein, the first reference neural network model is a trained neural network model. After RAN configures the first reference neural network model, it uses the first data information to calculate and process the first sample data in the first data information through the configured first reference neural network model to obtain the first indication information, and the first indication information is the "knowledge" obtained by the distillation training of the first reference neural network model. RAN sends the first indication information to UE1, and UE1 uses the first indication information to train the first neural network model it designed. In this way, RAN guides the distillation training of UE1 by migrating "knowledge". This method enables RAN and UE1 to perform separate distillation training, so that UE1 can design and train a personalized small network, achieve a balance between the size of the neural network model and the accuracy of the neural network model, improve the efficiency of distillation training, and save the storage space of UE1.

[0217] See also Figure 4 , Figure 4 This is a flow chart of a model distillation training provided by an embodiment of the present application. Before the first device performs distillation training on the neural network model, it establishes a communication connection with the second device for distillation training, and realizes the transfer of the "knowledge" of the second device by interacting with the second device for relevant distillation training information, thereby guiding the first device to perform distillation training on the neural network model. Figure 4 In the example, taking the first device as an edge network element and the second device as a central network element, the steps for the edge network element to perform distillation training on the neural network model are as follows:

[0218] Step S401: an edge network element sends a first training request to a central network element.

[0219] Specifically, before the edge network element distills and trains the neural network model, it must establish a distillation communication connection with the central network element. The edge network element first sends a first training request to the central network element, and the first training request is used by the edge network element to inform the central network element that the edge network element has a need to distill and train a neural network model of a certain functional type. In addition, the first training request includes a first training type ID, and the first training ID is used to indicate the functional type of the edge network element distillation training neural network model; the functional types include image recognition, face recognition, etc.

[0220] Step S402: The central network element sends a first response to the edge network element.

[0221] Specifically, after the central network element receives the first training request sent by the edge network element, the central network element compares the first training type ID with the second training type ID of the central network element. If the first training type ID is consistent with the second training type ID, it means that the central network element has a reference neural network model that supports distillation training of the edge network element, and the reference neural network model has the functional type indicated by the first training type ID. Among them, the second training type ID is the functional type that the central network element supports distillation training. Based on the second training type ID, the central network element finds the first reference neural network model from multiple reference neural network models deployed by the central network element, and the first reference neural network model has the functional type indicated by the second training type ID. The central network element sends a first response to the edge network element, and the first response includes the ID of the first reference neural network model and first storage information. Among them, the first storage information is used to indicate the size of the storage space occupied by the first reference neural network model.

[0222] If the first training type ID is inconsistent with the second training type ID, it means that the central network element does not have a reference neural network model corresponding to the first training type ID. Then, the central network element cannot obtain the first indication information by distilling the first data information sent by the edge network element through the reference neural network model. Since the edge network element lacks the first indication information, the edge network element cannot use the first indication information to distill and train the neural network model. Furthermore, the central network element cannot guide the edge network element to distill and train the neural network model. The embodiment of the present application takes the case where the first training type ID is consistent with the second training type ID as an example.

[0223] Step S403: the edge network element sends first query information to the central network element based on the first stored information.

[0224] Specifically, the first query information includes first distillation query information and first distillation capability information. The first distillation query information is sent by the edge network element to the central network element based on the first storage information, specifically: when the resource storage and other capabilities of the edge network element exceed a certain threshold, and the threshold is larger than the storage space occupied by the first reference neural network model indicated by the first storage information, the first distillation query information is used to inquire whether the central network element supports the distillation training of the edge network element by sending the first indication information to the edge network element; when the resource storage capacity of the edge network element is lower than the storage space occupied by the first reference neural network model indicated by the first storage information, the first distillation query information is used to inquire whether the central network element supports the distillation training of the edge network element by obtaining a small network through distillation training of the central network element, and sending the small network to the edge network element. This embodiment takes the case that when the resource storage and other capabilities of the edge network element exceed a certain threshold, the design of the small network can be performed as an example. Therefore, the first distillation query information is used to inquire whether the central network element supports the distillation training of the edge network element, and in what way to support the training.

[0225] The first distillation capability information is used to indicate the distillation training capability of the edge network element, including one or more of the following: the design capability of the neural network model, the distillation training capability of the neural network model, and whether the edge network element has distillation training data information. The design capability of the neural network model is used to indicate whether the edge network element has the ability to independently design a small neural network model. The distillation training capability of the neural network model is used to indicate whether the edge network element has the ability to distill and train the first neural network model when the edge network element can independently design the first neural network model. The distillation training capability of the neural network model includes one or more of the following: whether the edge network element supports the training package for algorithm training, whether it has a GPU chip and a tensor core of the GPU chip, the number of floating-point operations per second, the video memory bandwidth, the CPU's main frequency size, the CPU's cache size, the architecture category, and the storage space and other hardware capabilities. The distillation training data is the first data information of the edge network element, which is used for the central network element to obtain the first indication information based on the first sampled data in the first data information, and guide the edge network element to perform distillation training. The embodiment of the present application takes the edge network element having the neural network model design capability, the neural network model distillation training capability, and the distillation training data information as an example.

[0226] Step S404: the central network element generates a second response based on the first distillation query information, and sends the second response to the edge network element.

[0227] Specifically, after receiving the first distillation query information, the central network element sends a second response to the edge network element based on its own storage resources, hardware performance and other parameters. The second response is used to indicate that the central network element supports distillation training for the edge network element, and supports the edge network element to perform distillation training by sending the first indication information to the edge network element.

[0228] Step S405: The edge network element designs a first neural network model.

[0229] Specifically, if the edge network element needs to refer to the structural information of the first reference neural network model when designing the first neural network model and the structural information of the first reference neural network model has been loaded, the edge network element designs a small network based on the structural information of the first reference neural network model, and the small network is the first neural network model.

[0230] In one possible implementation, if the edge network element needs to refer to the structural information of the first reference neural network model when designing the first neural network model, and the structural information of the first reference neural network model is not loaded, the edge network element sends a first network structure request to the central network element, and the first network structure request is used by the edge network element to request the central network element to provide the structural information of the first reference network structure model. The central network element sends a first structure request response to the edge network element based on the first network structure request, and the first structure request response includes the structural information of the first reference neural network model. The edge network element designs a small network with reference to the structural information of the first reference neural network model, and the small network is the first neural network model.

[0231] Step S406: The edge network element sends first configuration information to the central network element.

[0232] Specifically, the first configuration information sent by the edge network element to the central network element is used for the central network element to configure the first reference neural network model. The first configuration information includes first distillation temperature information. The first distillation temperature information is the temperature parameter T of the normalized exponential function Softmax. The temperature parameter T is obtained according to the size difference between the first neural network model and the first reference neural network model, and is used to increase the amount of information output by the first reference neural network model after distillation. The first configuration information also includes a knowledge type KnowledgeType and an ID of the first reference neural network model. The knowledge type KnowledgeType is used to indicate the type of first indication information required by the edge network element, wherein the type of the first indication information includes: soft targets output by the first reference neural network model, intermediate output feature information of the first reference neural network model, and the relationship between network layers within the first reference neural network model. The ID of the first reference neural network model is used to associate the first configuration information with the first reference neural network model to avoid confusion with other neural network models of the central network element.

[0233] Step S407: The edge network element sends the first data information to the central network element.

[0234] Specifically, the first data information sent by the edge network element to the central network element is used by the central network element to process the first data information through the first reference neural network model to obtain first indication information. The first data information includes first sample data, which is sample data used for distillation training by the first reference neural network model; the first sample information also includes a sample data index Index and an ID of the first reference neural network model. Among them, the sample data index Index is a label for each first sample data, which is used to indicate the first sample data; the ID of the first reference neural network model is used to associate the first configuration information with the first reference neural network model to avoid confusion with other neural network models of the central network element.

[0235] Step S408: The central network element calculates and processes the first sample data through a first reference neural network model to obtain first indication information, and sends the first indication information to the edge network element.

[0236] Specifically, the first indication information includes distillation knowledge DistillationKnowledge and the ID of the first reference neural network model, and the first indication information is used to guide the edge network element to train the first neural network model. The central network element inputs the first sample data into the first reference neural network model one by one according to the Index for calculation and processing. The specific process is as follows: The input first sample data is compared and analyzed with the preset categories in the first reference neural network model to obtain the first indication information, and the sample data index Index is associated with the distillation knowledge DistillationKnowledge. Then, the first indication information is sent to the edge network element. The ID of the first reference neural network model is used to associate the distillation knowledge DistillationKnowledge with the first reference neural network model to avoid confusion with other neural network models of the central network element.

[0237] Step S409: The edge network element uses the first indication information to perform distillation training on the first neural network model.

[0238] Specifically, if the type of the first indication information received by the edge network element is a soft target output by the first reference neural network model, that is, the first indication information is the similarity between the first sample data and the preset category of the first reference neural network model, the process of the edge network element using the first indication information to perform distillation training on the first neural network model is as follows: the edge network element inputs the first sample data into the first neural network model one by one according to the Index, and then compares and analyzes the first sample data with the preset categories in the first neural network model respectively to obtain the similarity between the first sample data and the preset category. Then, the edge network element outputs a soft target through the Softmax function (the temperature parameter T configured by the Softmax function of the first neural network model is the same as the temperature parameter T configured by the Softmax function of the first reference neural network model) and compares and analyzes the soft target with the soft target output by the first reference neural network model to obtain a loss function Fn1. In addition, the edge network element processes the similarity between the first sample data and the preset category through another Softmax function (the configured temperature parameter T=1) to obtain a hard target, and compares and analyzes the hard target with the preset result of the first sample data to obtain a loss function Fn2. The edge network element adjusts the first neural network model based on Fn1 and Fn2 to reduce the values ​​of the loss functions Fn1 and Fn2, enhance the generalization ability of the first neural network model, and realize the distillation training of the first neural network model by the edge network element.

[0239] In a possible implementation, if the type of the first indication information received by the edge network element is the intermediate output feature information of the first reference neural network model, that is, the first indication information is the first feature information output by the intermediate network layer of the first reference neural network model when the first sample data is compared and analyzed with the preset category of the first reference neural network model, the process of the edge network element using the first indication information to perform distillation training on the first neural network model is as follows: the edge network element inputs the first sample data into the first neural network model one by one according to the Index, and obtains the second feature information output by the intermediate layer of the first neural network model when the first sample data is compared and analyzed with the preset category of the first neural network model. Then, the second feature information is compared and analyzed with the first feature information to obtain the loss function Fn3. The edge network element adjusts the parameters of the first neural network model based on Fn3 to reduce the value of the loss function Fn3, enhance the generalization ability of the first neural network model, and realize the distillation training of the first neural network model by the edge network element.

[0240] In one possible implementation, if the type of the first indication information received by the edge network element is the relationship between network layers of the first reference neural network model, the process of the edge network element using the first indication information to perform distillation training on the first neural network model is as follows: the edge network element inputs the first sample data into the first neural network model one by one according to the Index, and then compares and analyzes the first sample data with the preset categories in the first neural network model to obtain the relationship between each network layer in the first neural network model, and compares and analyzes the relationship between each network layer in the first neural network model with the first indication information to obtain the loss function Fn4. The edge network element adjusts the parameters of the first neural network model based on Fn4 to reduce the value of the loss function Fn4, enhance the generalization ability of the first neural network model, and realize the distillation training of the first neural network model by the edge network element.

[0241] Step S410: The edge network element sends a first training result notification to the central network element.

[0242] Specifically, after the edge network element completes the distillation training of the first neural network model, it sends a first training result notification to the central network element, where the first training result notification is used to inform the central network element whether the distillation training is completed; when the distillation training of the edge network element has been completed, information about the trained first neural network model by the edge network element is fed back to the central network element (for example, whether the edge network element receives the first neural network model trained by distillation, etc.).

[0243] It should be noted that, since the neural network model of distillation training has other requirements such as accuracy and generalization ability, the training of the first neural network model of the edge network element cannot be completed in one training, and steps S407 to S409 often need to be repeated. Whether the distillation training of the edge network element is completed is determined by the convergence of the loss function Fn of the edge network element in the distillation training or the reaching of the preset maximum number of iterations. For example, when the loss function Fn of the edge network element in the distillation training reaches the preset value y, it can be considered that the first neural network model of the distillation training of the edge network element has reached the required accuracy, then the distillation training of the edge network element has been completed. Alternatively, when the distillation training has reached the preset number of times of the edge network element, the distillation training is terminated.

[0244] Step S411: the central network element releases the channel resources of the distillation training communication connection between the central network element and the edge network element.

[0245] Specifically, after receiving the first training result notification, the central network element determines whether the distillation training of the edge network element has been completed; if the distillation training of the edge network element has been completed, the communication connection resources for the distillation training between the central network element and the edge network element are released to avoid occupying other channel resources of the central network element and the edge network element.

[0246] In this embodiment, before the edge network element distills and trains the neural network model, it is necessary to establish a communication connection for distillation training with the central network element. Through the interaction of distillation training information between the edge network element and the central network element, the central network element uses the first configuration information sent by the edge network element to configure the first reference neural network model; wherein, the first reference neural network model is a trained neural network model. After the central network element configures the first reference neural network model, it uses the first data information to calculate and process the first sample data in the first data information through the configured first reference neural network model to obtain the first indication information, and the first indication information is the "knowledge" obtained by the distillation training of the first reference neural network model. The central network element sends the first indication information to the edge network element, and the edge network element uses the first indication information to train the first neural network model it designed. In this way, the central network element guides the distillation training of the edge network element by migrating "knowledge". This method enables the central network element and the edge network element to perform separate distillation training, so that the edge network element can design and train personalized small networks, achieve a balance between the size of the neural network model and the accuracy of the neural network model, improve the efficiency of distillation training, and save the storage space of the edge network element.

[0247] See also Figure 5 , Figure 5 This is another flow chart of model distillation training provided by an embodiment of the present application. Before the first device sends the first configuration information to the second device, it sends the first attention category list to the second device. The first indication information received by the first device is the information filtered by the second device based on the first attention category list. Figure 5 In the example, taking the first device as an edge network element and the second device as a central network element, the steps for the edge network element to perform distillation training on the neural network model are as follows:

[0248] Please refer to step S401 to step S405 for steps S501 to S505, which will not be described again here.

[0249] Step S506: the edge network element sends the first concerned category list to the central network element.

[0250] Specifically, the first list of concerned categories sent by the edge network element to the central network element includes a set of categories concerned by the distillation training of the edge network element; wherein the set of categories concerned by the distillation training of the edge network element is a subset of the set of categories in the first category list; wherein the first category list includes a set of preset categories of the first reference neural network model. By sending the first list of concerned categories to the central network element, the edge network element can obtain the first indication information corresponding to the category concerned by the distillation training of the edge network element, which is conducive to realizing personalized training of the first neural network model by the edge network element.

[0251] Please refer to step S406 to step S407 for step S507 to step S508, which will not be described in detail here.

[0252] Step S509: the central network element calculates and processes the first sample data through the first reference neural network model, and filters the processed first sample data based on the first focus category list to obtain first indication information, and sends the first indication information to the edge network element.

[0253] Specifically, similar to step S408, the central network element inputs the first sample data into the first reference neural network model one by one according to the Index, compares and analyzes the first sample data with the preset categories in the first reference neural network model to obtain indication information, and the central network element filters out the indication information corresponding to the categories in the first focus category list based on the categories in the first focus category list from the indication information to obtain the first indication information, and then the central network element sends the first indication information to the edge network element.

[0254] For steps S510 to S512, please refer to steps S409 to S411, which will not be repeated here.

[0255] In this embodiment, before the central network element sends the first indication information, the edge network element sends the first attention category list to the central network element, which is used to instruct the central network element to filter the indication information obtained by passing the first sample data through the first reference neural network model based on the attention categories in the first attention category list. By sending the first attention category list to the central network element, the central network element filters the indication information obtained by comparing and analyzing the first sample data with the preset categories in the first reference neural network model. The obtained first indication information better meets the requirements of the distillation training of the edge network element, further reduces the communication overhead without affecting the effect of the distillation training of the first neural network model, saves network resources for separate distillation training, and saves storage space of the edge network element.

[0256] See also Figure 6 , Figure 6 1 is a flow chart of another model distillation training provided by an embodiment of the present application. Before the second device instructs the first device to perform distillation training of the neural network model, a distillation training communication connection between the first device and the second device is established through a third device. Figure 6 In the example, the first device is an edge NWDAF, the second device is a central NWDAF, and the third device is a network repository function (NRF). The steps of distillation training of the neural network model by the edge NWDAF are as follows:

[0257] Step S601: The central NWDAF sends registration information to the NRF.

[0258] Specifically, the registration information includes a third training type ID, which is used to indicate the functional type of the neural network model supported by the central NWDAF for distillation training; a third neural network model ID, which is used to indicate a second reference neural network model corresponding to the third training type ID, and the second reference neural network model has the function indicated by the third training type ID; second storage information, which is used to indicate the size of the storage space occupied by the second reference neural network model; training response information, which is used to indicate whether the central NWDAF supports the distillation training neural network model and, if the distillation training neural network model is supported, the way to support the distillation training neural network model. The registration information also includes a second category list, which includes a set of categories preset by the central NWDAF; the ID of the central NWDAF, which is used to indicate the device number of the central NWDAF to avoid confusion caused by multiple central NWDAFs registering with the NRF. The central NWDAF sends registration information to the NRF, which is conducive to the NRF matching the central NWDAF and the edge NWDAF based on the registration information, and then establishing a distillation training communication connection between the central NWDAF and the edge NWDAF.

[0259] Step S602: The edge NWDAF sends a second training request to the network NRF.

[0260] Specifically, the second training request includes a fourth training type ID, which is used to indicate the functional type of the neural network model to be distilled and trained by the edge NWDAF; second inquiry information, which is used for the edge NWDAF to inquire the NRF whether the central NWDAF supports distillation training, and if so, the way to support distillation training; second distillation capability information, which is used to indicate the distillation training capability of the edge NWDAF, and the second distillation capability information includes one or more of the following: the design capability of the edge NWDAF neural network model, the training capability of the neural network model, and whether there is distillation training data. The second training request also includes an exclusion set, and the exclusion set includes the ID of the central NWDAF that failed to establish a distillation training communication connection with the edge NWDAF. When the ID of the central NWDAF in the exclusion set exists in the registration information contained in the NRF, the NRF no longer establishes a distillation training communication connection between the central NWDAF and the edge NWDAF.

[0261] Step S603: The NRF generates a third response based on the second training request, and sends the third response to the edge NWDAF.

[0262] Specifically, after receiving the second training request, the NRF determines whether the third training type ID is consistent with the fourth training type ID. If they are consistent, a third response is generated (this embodiment takes the consistency of the third training type ID and the fourth training type ID as an example), and the third response is sent to the edge NWDAF. The third response includes training response information, the third neural network model ID, the second storage information and the second category list, and also includes the ID of the central NWDAF.

[0263] Step S604: The edge NWDAF sends a distillation notification to the NRF.

[0264] Specifically, after receiving the third response, the edge NEDAF learns that the central NWDAF supports distillation training and the distillation training method according to the training response information in the third response (in this embodiment, the training response information indicates that the central NWDAF supports the distillation training neural network model of the edge NWDAF, and takes the method of sending indication information to the edge NWDAF to guide the edge NWDAF to distill the training neural network model as an example); and learns the information of the second reference neural network model and the size of the storage space occupied by the second reference neural network model according to the third neural network model ID and the second storage information. Based on the information of the central NWDAF obtained by the third response, the edge NWDAF establishes a communication connection for distillation training with the central NWDAF according to the ID of the central NWDAF, and after the communication connection is established, sends a distillation notification to the NRF. The distillation notification is used to inform the NRF that the edge NWDAF is successfully connected to the central NWDAF. Then, the NRF releases the communication resources between the NRF and the edge NWDAF and the central NWDAF according to the distillation notification.

[0265] For steps S605 to S611, please refer to steps S405 to S411, which will not be repeated here.

[0266] In this embodiment, the central NWDAF registers the training type ID and training response information on the NRF, and the edge NWDAF sends a training request to the NRF. The NRF can quickly and effectively establish a distillation communication connection between the edge NWDAF and the central NWDAF, thereby improving the efficiency of the distillation training, which is beneficial to the edge NWDAF to distill the training neural network model and receive the storage space of the edge NWDAF.

[0267] See also Figure 7 , Figure 7 It is a flowchart of another model distillation training provided by an embodiment of the present application. Before the first device sends the second configuration information to the second device, the second attention category list is sent to the second device, and the second indication information received by the first device is the information filtered by the second device based on the second attention category list. Figure 7In the example, taking the first device as edge NWDAF, the second device as center NWDAF, and the third device as NRF, the steps of distillation training of the neural network model by edge NWDAF are as follows:

[0268] For steps S701 to S705 , please refer to steps S601 to S605 , which will not be described in detail here.

[0269] Step S706: the edge NWDAF sends the second focus category list to the central NWDAF.

[0270] Specifically, the second list of concerned categories sent by the edge NWDAF to the central NWDAF includes a set of categories concerned by the distillation training of the edge NWDAF; wherein the set of categories concerned by the distillation training of the edge NWDAF is a subset of the set of categories in the second category list; wherein the second category list includes a set of preset categories of the second reference neural network model. By sending the second list of concerned categories to the central NWDAF, the edge NWDAF can obtain the second indication information corresponding to the category concerned by the distillation training of the edge NWDAF, which is conducive to realizing the personalized training of the second neural network model by the edge NWDAF.

[0271] Please refer to step S606 to step S607 for step S707 to step S708, which will not be repeated here.

[0272] Step S709: The central NWDAF calculates and processes the second sample data through the second reference neural network model, and filters the processed second sample data based on the second focus category list to obtain second indication information, and sends the second indication information to the edge NWDAF.

[0273] The central NWDAF inputs the second sample data in the second data information into the second reference neural network model one by one according to the Index, compares and analyzes the second sample data with the preset categories in the second reference neural network model to obtain indication information, and the central NWDAF filters out indication information related to the categories in the second focus category list from the indication information based on the categories in the second focus category list to obtain second indication information, and then, the central NWDAF sends the second indication information to the edge NWDAF.

[0274] Please refer to step S609 to step S611 for step S710 to step S712, which will not be repeated here.

[0275] In this embodiment, the edge NWDAF sends a second attention category list to the central NWDAF before the central NWDAF sends the second indication information, which is used to instruct the central NWDAF to filter the indication information obtained by passing the second sample data through the second reference neural network model based on the attention categories in the second attention category list. By sending the second attention category list to the central NWDAF, the central NWDAF filters the indication information obtained by comparing and analyzing the second sample data with the preset categories in the second reference neural network model. The obtained second indication information better meets the requirements of the distillation training of the edge NWDAF, further reduces the communication overhead without affecting the effect of the distillation training of the second neural network model, saves network resources for separate distillation training, and thus saves the storage space of the NWDAF.

[0276] The method of the embodiment of the present application is described in detail above, and the related device of the embodiment is provided below.

[0277] See also Figure 8 , Figure 8 is a structural schematic diagram of a model distillation training device provided in an embodiment of the present application. The model distillation training device 8 may be the first device in the above method embodiment. The model distillation training device 8 may include a first neural network model design unit 801, a first configuration information sending unit 802, a first data information sending unit 803, a first indication information receiving unit 804 and a first distillation training unit 805, wherein each unit is described in detail as follows:

[0278] A first neural network model design unit 801 is used to design a first neural network model if the second device supports the first device distillation training neural network model;

[0279] A first configuration information sending unit 802, configured to send first configuration information to the second device;

[0280] A first data information sending unit 803, configured to send first data information to the second device;

[0281] A first indication information receiving unit 804 is configured to receive first indication information returned by the second device;

[0282] The first distillation training unit 805 is used to train the first neural network model using the first indication information.

[0283] In a possible implementation, the model distillation training device 8 further includes:

[0284] A first training request sending unit, configured to send a first training request to the second device before the first neural network model design unit 801 designs the first neural network model;

[0285] A first response receiving unit, configured to receive a first response sent by the second device after the first training request sending unit sends the first training request to the second device if the second training type ID is consistent with the first training type ID;

[0286] A first inquiry information sending unit, configured to send first inquiry information to the second device based on the first storage information;

[0287] The second response receiving unit is configured to receive a second response from the second device after the first inquiry information sending unit sends the first inquiry information to the second device.

[0288] In a possible implementation, the model distillation training device 8 further includes:

[0289] The first attention category list sending unit is configured to send a first attention category list to the second device before the first configuration information sending unit 802 sends the first configuration information to the second device.

[0290] In a possible implementation, the model distillation training device 8 further includes:

[0291] A first network structure request sending unit, configured to send a first network structure request to the second device before the first neural network model design unit 801 designs the first neural network model;

[0292] The first structure request response receiving unit is used to receive the first structure request response sent by the second device after the first network structure request sending unit sends the first network structure request to the second device.

[0293] See also Fig. 9 , Fig. 9 is a structural schematic diagram of a model distillation training device provided in an embodiment of the present application. The model distillation training device 9 may be the second device in the above method embodiment. The model distillation training device 9 may include a first configuration information receiving unit 901, a first parameter configuration unit 902, a first data information receiving unit 903 and a first indication information sending unit 904, wherein each unit is described in detail as follows:

[0294] A first configuration information receiving unit 901 is configured to receive first configuration information sent by a first device;

[0295] A first parameter configuration unit 902, configured to configure a first reference neural network model using the first configuration information;

[0296] A first data information receiving unit 903, configured to receive first data information sent by the first device;

[0297] The first indication information sending unit 904 is configured to send first indication information to the first device.

[0298] In a possible implementation, the model distillation training device 9 further includes:

[0299] A first training request receiving unit, configured to receive a first training request sent by the first device before the first parameter configuration unit 902 configures the first reference neural network model;

[0300] a first response sending unit, configured to generate a first response based on the first training request if the second training type ID is consistent with the first training type ID, and send the first response to the first device;

[0301] A first inquiry information receiving unit, configured to receive first inquiry information sent by the first device;

[0302] The second response sending unit is configured to send the second response to the first device after the first inquiry information receiving unit receives the first inquiry information sent by the first device.

[0303] In a possible implementation, the model distillation training device 9 further includes:

[0304] The first attention category list receiving unit is configured to receive a first attention category list sent by the first device before the first configuration information receiving unit receives the first configuration information sent by the first device.

[0305] In a possible implementation, the model distillation training device 9 further includes:

[0306] A first network structure request receiving unit, configured to receive a first network structure request sent by the first device before the first configuration information receiving unit 901 receives the first configuration information sent by the first device;

[0307] The first structure request response sending unit is configured to send a first structure request response to the first device.

[0308] See also Fig.10 , Fig.101 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 10 may be the third device in the above method embodiment. The model distillation training device 10 may include a registration information receiving unit 1001, a second training request receiving unit 1002, a third response sending unit 1003 and a distillation notification receiving unit 1004, wherein each unit is described in detail as follows:

[0309] The registration information receiving unit 1001 is used to receive the registration information sent by the second device;

[0310] A second training request receiving unit 1002 is configured to receive a second training request sent by the first device;

[0311] A third response sending unit 1003 is configured to generate a third response based on the second training request if the fourth training type ID is consistent with the third training type ID, and send the third response to the first device;

[0312] The distillation notification receiving unit 1004 is configured to receive a distillation notification sent by the first device.

[0313] See also Fig.11 , Fig.11 1 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 11 may be the first device in the above method embodiment. The model distillation training device 11 may include a second training request sending unit 1101, a third response receiving unit 1102 and a distillation notification sending unit 1103, wherein each unit is described in detail as follows:

[0314] A second training request sending unit 1101 is configured to send a second training request to a third device;

[0315] A third response receiving unit 1102 is configured to receive a third response sent by the third device if the fourth training type ID is consistent with the third training type ID;

[0316] The distillation notification sending unit 1103 is configured to send a distillation notification to the third device.

[0317] In a possible implementation, the model distillation training device 11 further includes:

[0318] A second neural network model design unit, configured to design a second neural network model based on the structural information of the second reference neural network model after the distillation notification sending unit 1103 sends the distillation notification to the third device;

[0319] A second configuration information sending unit, used to send second configuration information to the second device;

[0320] A second data information sending unit, configured to send second data information to the second device;

[0321] A second indication information receiving unit, configured to receive second indication information returned by the second device;

[0322] The second distillation training unit is used to perform distillation training on the second neural network model using the second indication information.

[0323] In a possible implementation, the model distillation training device 11 further includes:

[0324] The second attention category list sending unit is configured to send a second attention category list to the second device after the second configuration information sending unit sends the second configuration information to the second device.

[0325] See also Fig.12 , Fig.12 1 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 12 may be the second device in the above method embodiment. The model distillation training device 12 may include a registration information sending unit 1201, a second configuration information receiving unit 1202, a second data information receiving unit 1203, and a second indication information sending unit 1204, wherein each unit is described in detail as follows:

[0326] The registration information sending unit 1201 is used to send registration information to a third device;

[0327] The second configuration information receiving unit 1202 is configured to receive second configuration information sent by the first device;

[0328] A second data information receiving unit 1203 is configured to receive second data information sent by the first device;

[0329] The second indication information sending unit 1204 is configured to send second indication information to the first device.

[0330] In a possible implementation, the model distillation training device 12 further includes:

[0331] The second attention category list receiving unit is configured to receive the second attention category list sent by the first device before the second configuration information receiving unit 1202 receives the second configuration information sent by the first device.

[0332] In a possible implementation, the model distillation training device 12 further includes:

[0333] A second network structure request receiving unit, configured to receive a second network structure request sent by the first device before the second configuration information receiving unit 1202 receives the second configuration information sent by the first device;

[0334] The second structure request response sending unit is used to send a second structure request response to the first device based on the second network structure request.

[0335] See also Fig.13 , Fig.13 1 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 13 may be the first device in the above method embodiment. The model distillation training device 13 may include a memory 1301, a communication module 1302 and a processor 1303, wherein each unit is described in detail as follows:

[0336] The memory 1301 is used to store program codes.

[0337] The processor 1303 is used to call the program code stored in the memory to execute the following steps:

[0338] If the second device supports the distillation training neural network model of the first device, designing a first neural network model;

[0339] Sending first configuration information to the second device through the communication module 1302;

[0340] Sending first data information to the second device through the communication module 1302;

[0341] Receiving, through the communication module 1302, first indication information returned by the second device;

[0342] The first neural network model is trained using the first indication information.

[0343] In a possible implementation, the processor 1303 is further configured to: before designing the first neural network model, send a first training request to the second device through the communication module 1302;

[0344] If the second training type ID is consistent with the first training type ID, receiving a first response sent by the second device through the communication module 1302;

[0345] Sending first inquiry information to the second device through the communication module 1302;

[0346] A second response from the second device is received through the communication module 1302 .

[0347] In a possible implementation manner, the processor 1303 is further configured to: send the first attention category list to the second device through the communication module 1302 .

[0348] In a possible implementation, the processor 1303 is further configured to: before designing the first neural network model, send a first network structure request to the second device through the communication module 1302;

[0349] The communication module 1302 receives a first structure request response sent by the second device.

[0350] See also Fig.14 , Fig.14 1 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 14 may be the second device in the above method embodiment. The model distillation training device 14 may include a memory 1401, a communication module 1402 and a processor 1403, wherein each unit is described in detail as follows:

[0351] The memory 1401 is used to store program codes.

[0352] The processor 1403 is used to call the program code stored in the memory to execute the following steps:

[0353] Receiving, through the communication module 1402, first configuration information sent by the first device;

[0354] Receiving first data information sent by the first device through the communication module 1402;

[0355] The first indication information is sent to the first device through the communication module 1402.

[0356] In a possible implementation, the processor 1403 is further configured to: receive, through the communication module 1402, a first training request sent by the first device;

[0357] Generate a first response;

[0358] Sending the first response to the first device through the communication module 1402;

[0359] Receiving, through the communication module 1402, first inquiry information sent by the first device;

[0360] generating a second response;

[0361] The second response is sent to the first device through the communication module 1402 .

[0362] In a possible implementation, the processor 1403 is further configured to: before receiving, through the communication module 1402 , the first configuration information sent by the first device, receive, through the communication module 1402 , a first attention category list sent by the first device.

[0363] In one possible implementation, the processor 1403 is further used to: before receiving the first configuration information sent by the first device through the communication module 1402, receive the first network structure request sent by the first device through the communication module 1402; and send a first structure request response to the first device through the communication module 1402.

[0364] See also Fig.15 , Fig.15 1 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 15 may be the third device in the above method embodiment. The model distillation training device 15 may include a memory 1501, a communication module 1502 and a processor 1503, wherein each unit is described in detail as follows:

[0365] The memory 1501 is used to store program codes.

[0366] The processor 1503 is used to call the program code stored in the memory to execute the following steps:

[0367] Receiving registration information sent by the second device through the communication module 1502;

[0368] Receiving, through the communication module 1502, a second training request sent by the first device;

[0369] generating a third response;

[0370] Sending the third response to the first device through the communication module 1502;

[0371] The distillation notification sent by the first device is received through the communication module 1502.

[0372] See also Fig.16 , Fig.16 1 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 16 may be the first device in the above method embodiment. The model distillation training device 16 may include a memory 1601, a communication module 1602 and a processor 1603, wherein each unit is described in detail as follows:

[0373] The memory 1601 is used to store program codes.

[0374] The processor 1603 is used to call the program code stored in the memory to execute the following steps: send a second training request to the third device through the communication module 1602; if the fourth training type ID is consistent with the third training type ID, receive a third response sent by the third device through the communication module 1602;

[0375] The distillation notification is sent to the third device through the communication module 1602.

[0376] In a possible implementation, the processor 1603 is further configured to: after sending the distillation notification to the third device through the communication module 1602, design a second neural network model;

[0377] Sending second configuration information to the second device through the communication module 1602;

[0378] Sending second data information to the second device through the communication module 1602;

[0379] receiving second indication information returned by the second device;

[0380] The second neural network model is trained using the second indication information.

[0381] In a possible implementation, the processor 1603 is further configured to: before sending the second configuration information to the second device through the communication module 1602 , send a second attention category list to the second device through the communication module 1602 .

[0382] The second neural network model is trained by distillation using the second indication information.

[0383] In a possible implementation, the processor 1603 is further configured to: before designing the second neural network model, send a second network structure request to the second device through the communication module 1602;

[0384] A second structure request response sent by the second device is received through the communication module 1602 .

[0385] See also Fig.17 , Fig.17 17 is a schematic diagram of the structure of a model distillation training device provided in an embodiment of the present application. The model distillation training device 17 may be the second device in the above method embodiment. The model distillation training device 17 may include a memory 1701, a communication module 1702 and a processor 1703, wherein each unit is described in detail as follows:

[0386] The memory 1701 is used to store program codes.

[0387] The processor 1703 is used to call the program code stored in the memory to execute the following steps:

[0388] Sending registration information to the third device through the communication module 1702;

[0389] Receiving second configuration information sent by the first device through the communication module 1702;

[0390] Receiving second data information sent by the first device through the communication module 1702;

[0391] The second indication information is sent to the first device through the communication module 1702.

[0392] In a possible implementation, the processor 1703 is further configured to: before receiving, through the communication module 1702 , the second configuration information sent by the first device, receive, through the communication module 1702 , a second attention category list sent by the first device.

[0393] In a possible implementation, the processor 1703 is further configured to: before receiving the second configuration information sent by the first device through the communication module 1702, receive a second network structure request sent by the first device through the communication module 1702;

[0394] A second structure request response is sent to the first device through the communication module 1702.

[0395] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the model distillation training method in the above embodiment and its various possible implementation methods.

[0396] An embodiment of the present application provides a computer program, which includes instructions. When the computer program is executed by a computer, a first device can execute the process executed by the first device in the above-mentioned embodiment and its various possible implementations, or a second device can execute the process executed by the second device in the above-mentioned embodiment and its various possible implementations, and a third device can execute the process executed by the third device in the above-mentioned embodiment and its various possible implementations.

[0397] An embodiment of the present application provides a chip system, which includes a processor for supporting a first device to implement the functions involved in the methods in the above-mentioned embodiments and their various possible modes, or a second device to implement the functions involved in the methods in the above-mentioned embodiments and their various possible modes, or a third device to implement the functions involved in the methods in the above-mentioned embodiments and their various possible modes.

[0398] In a possible design, the chip system further includes a memory, and the memory is used to store necessary program instructions and data of the first device, the second device, or the third device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0399] It is to be understood that those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the various embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0400] Those skilled in the art will appreciate that the functions described in the various illustrative logic blocks, modules, and algorithm steps disclosed in the various embodiments of the present application can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this way, computer-readable media can generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in the present application. A computer program product may include computer-readable media.

[0401] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0402] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0403] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0404] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0405] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0406] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for model distillation training by establishing a distillation training communication connection between a first device and a second device through a third device, characterized in that: include: The third device receives registration information sent by the second device; wherein the registration information includes a third training type ID, a third neural network model ID, second storage information, a second category list, and training response information; wherein the training response information is used to indicate whether the second device supports the distillation training neural network model and, if the distillation training neural network model is supported, the way in which the distillation training neural network model is supported; the third training type ID is used to indicate the function type of the distillation training neural network model supported by the second device; The third device receives a second training request sent by the first device; wherein the second training request includes a fourth training type ID, second distillation query information and second distillation capability information; the fourth training type ID is used to indicate the function type of the neural network model to be distilled and trained by the first device; If the fourth training type ID is consistent with the third training type ID, the third device generates a third response based on the second training request, and sends the third response to the first device; wherein the third response includes training response information, a third neural network model ID, second storage information, and a second category list; The third device receives a distillation notification sent by the first device, wherein the distillation notification is used to indicate whether the first device and the second device are matched successfully.

2. A method for model distillation training by establishing a distillation training communication connection between a first device and a second device through a third device, characterized in that: include: The first device sends a second training request to the third device; wherein the second training request includes a fourth training type ID, second distillation query information, and second distillation capability information; wherein the fourth training type ID is used to indicate the function type of the neural network model to be distilled and trained by the first device; If the fourth training type ID is consistent with the third training type ID, the first device receives a third response sent by the third device; wherein the third response includes training response information, a third neural network model ID, second storage information and a second category list; the third training type ID is used to indicate that the second device supports the function type of the distillation training neural network model; The first device sends a distillation notification to the third device; wherein the distillation notification is used to indicate whether the first device and the second device are matched successfully.

3. The method according to claim 2, characterized in that After the first device sends the distillation notification to the third device, the method further includes: The first device designs a second neural network model; The first device sends second configuration information to the second device; wherein the second configuration information is used for the second device to configure a second reference neural network model; The first device sends second data information to the second device; wherein the second data information includes second sample data used for distillation training by the second reference neural network model; The first device receives second indication information returned by the second device, and uses the second indication information to train the second neural network model; the second indication information is information obtained by inputting the second sample data into the second reference neural network model.

4. The method according to claim 3, characterized in that Before the first device sends the second configuration information to the second device, the method further includes: The first device sends a second focus category list to the second device; wherein the second focus category list includes a set of categories focused on by the distillation training of the first device; in the second focus category list, the set of categories focused on by the distillation training of the first device is a subset of the set of categories in the second category list; the second category list includes a set of preset categories of the second reference neural network model.

5. The method according to claim 4, characterized in that The second indication information is obtained by the second device calculating and processing the second sample data through the second reference neural network model, and filtering the processed second sample data based on the second focus category list.

6. The method according to any one of claims 3 to 5, characterized in that: The first device designs a second neural network model, including: The first device sends a second network structure request to the second device; wherein the second network structure request is used by the first device to obtain structural information of the second reference neural network model from the second device; The first device receives a second structure request response sent by the second device; wherein the second structure request response includes structure information of the second reference neural network model; The first device designs a second neural network model based on structural information of the second reference neural network model.

7. A method for model distillation training by establishing a distillation training communication connection between a first device and a second device through a third device, characterized in that: include: The second device sends registration information to the third device; wherein the registration information includes a third training type ID, a third neural network model ID, second storage information, a second category list, and training response information; the training response information is used to indicate whether the second device supports the distillation training neural network model and, if the distillation training neural network model is supported, the way in which the distillation training neural network model is supported; After establishing a communication connection for distillation training between the first device and the second device, the second device receives second configuration information sent by the first device; wherein the second configuration information is used by the second device to configure a second reference neural network model; The second device receives second data information sent by the first device; wherein the second data information includes second sample data used for distillation training by the second reference neural network model; The second device sends second indication information to the first device; wherein the second indication information is used by the first device to train a second neural network model; the second indication information is information obtained by inputting the second sample data into the second reference neural network model.

8. The method according to claim 7, characterized in that Before the second device receives the second configuration information sent by the first device, the method further includes: The second device receives a second focus category list sent by the first device; wherein the second focus category list includes a set of categories focused on by the first device for distillation training; in the second focus category list, the set of categories focused on by the first device for distillation training is a subset of the set of categories in the second category list; the second category list includes a set of preset categories for the second reference neural network model.

9. The method according to claim 8, characterized in that The second indication information is obtained by the second device calculating and processing the second sample data through the second reference neural network model, and filtering the processed second sample data based on the second focus category list.

10. The method according to any one of claims 7 to 9, characterized in that: Before the second device receives the second configuration information sent by the first device, the method further includes: The second device receives a second network structure request sent by the first device; wherein the second network structure request is used by the first device to obtain structural information of a second reference neural network model from the second device; The second device sends a second structure request response to the first device based on the second network structure request; wherein the second structure request response includes structural information of the second reference neural network model.

11. A model distillation training device, characterized in that: The model distillation training device is a first device, comprising a unit for executing the method as described in any one of claims 2-6.

12. A model distillation training device, characterized in that: The model distillation training device is a second device, comprising a unit for executing the method as described in any one of claims 7-10.

13. A model distillation training device, characterized in that: The model distillation training device is a third apparatus, comprising a unit for executing the method as claimed in claim 1.

14. A model distillation training device, characterized in that: The model distillation training device is a first device, comprising a processor and a memory; The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method as described in any one of claims 2-6.

15. A model distillation training device, characterized in that: The model distillation training device is a second device, including a processor and a memory; The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method as described in any one of claims 7 to 10.

16. A model distillation training device, characterized in that: The model distillation training device is a third device, including a processor and a memory; Wherein, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the model distillation training method as described in claim 1.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images