Model knowledge distillation method and related device

Abstract

The embodiment of the invention provides a model knowledge distillation method and a related device, which are at least applied to the field of artificial intelligence, and the method comprises the steps: determining a first feature channel from a plurality of channels of a first convolution layer of a first image classification model based on a training data set; based on channel parameters of the first feature channel, performing model structure enhancement on the first image classification model to obtain a second image classification model; and performing knowledge distillation on a preset student model through the first image classification model and the second image classification model to obtain a trained student model. According to the method, the key feature channel is determined, the second image classification model is determined through the key feature channel, and knowledge distillation is performed on the student model in combination with the first image classification model and the second image classification model, so that the knowledge distillation effect of the model is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and are related to but not limited to a model knowledge distillation method and related devices. Background Art

[0002] With the development of artificial intelligence (AI), image classification has become increasingly applicable. Various image classification methods have emerged, and deep learning models are particularly effective at image classification. However, due to the large number of parameters, deep learning models are difficult to deploy in real-world applications. Therefore, knowledge distillation of trained deep learning models is necessary.

[0003] In related technologies, most of them use the trained deep learning model as the teacher model, and use the teacher model to guide the student model to learn, to obtain a trained student model, and then deploy the student model to realize the application of the deep learning model in actual scenarios. Summary of the Invention

[0004] The embodiments of the present application provide a model knowledge distillation method and related devices, which can be applied at least in the field of artificial intelligence. By determining the key feature channel and determining the second image classification model through the key feature channel, the student model is combined with the first image classification model and the second image classification model to perform knowledge distillation on the student model, thereby improving the knowledge distillation effect of the model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a model knowledge distillation method, including: determining a first feature channel from multiple channels of a first convolutional layer of a first image classification model based on a training data set; performing model structure enhancement on the first image classification model based on channel parameters of the first feature channel to obtain a second image classification model; and performing knowledge distillation on a preset student model through the first image classification model and the second image classification model to obtain a trained student model.

[0007] An embodiment of the present application provides a model knowledge distillation device, including: a determination module, used to determine a first feature channel from multiple channels of a first convolutional layer of a first image classification model based on a training data set; a structure enhancement module, used to perform model structure enhancement on the first image classification model based on the channel parameters of the first feature channel to obtain a second image classification model; a knowledge distillation module, used to perform knowledge distillation on a preset student model through the first image classification model and the second image classification model to obtain a trained student model.

[0008] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned model knowledge distillation method when executing the executable instructions stored in the memory.

[0009] An embodiment of the present application provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned model knowledge distillation method is implemented.

[0010] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned model knowledge distillation method.

[0011] The above solution has the following beneficial effects:

[0012] The model knowledge distillation method in the embodiment of the present application enhances the model structure of the first image classification model according to the channel parameters of the screened first feature channel to generate a second image classification model. Since the second image classification model is constructed based on the screened first feature channel, and the first feature channel can be the key feature channel in the first convolutional layer of the first image classification model, the second image classification model can better extract image features, so that the second image classification model has higher classification accuracy and robustness. That is, when feature extraction is performed through the second image classification model, the key features of the input data can be effectively retained, and the interference of irrelevant features can be reduced, thereby improving the model quality of the teacher model in the model knowledge distillation process; and since the second image feature classification model has better feature extraction capabilities, the second feature extraction model can extract feature information of higher dimensions, and the knowledge distillation of the student model is performed simultaneously through the first image classification model and the second image classification model, so that the student model can learn the feature information of the training data from multiple dimensions, thereby improving the classification accuracy of the student model and improving the knowledge distillation effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is an optional architectural diagram of the model knowledge distillation system provided in an embodiment of the present application;

[0014] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0015] Figure 3 This is an optional flow chart of the model knowledge distillation method provided in the embodiment of the present application;

[0016] Figure 4This is another optional flow chart of the model knowledge distillation method provided in the embodiment of the present application;

[0017] Figure 5 1 is a flow chart of determining a first feature channel based on a training data set provided in an embodiment of the present application;

[0018] Figure 6 1 is a flow chart of obtaining a second image classification model based on the channel parameters of the first feature channel provided in an embodiment of the present application;

[0019] Figure 7 This is a model distillation architecture diagram provided in an embodiment of the present application;

[0020] Figure 8 This is a schematic diagram of the process of model distillation provided in the embodiments of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0023] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0025] In order to better understand the model knowledge distillation method provided in the embodiment of the present application, the model knowledge distillation method in the related art is first described below.

[0026] In related technologies, the student model is trained solely by using a trained teacher model to guide its learning. However, these model knowledge distillation methods tend to be simplistic in their feature considerations and lack a focus on key channel features, resulting in poor knowledge distillation results.

[0027] Based on the problems existing in the related art, an embodiment of the present application provides a model knowledge distillation method, which enhances the model structure of the first image classification model according to the channel parameters of the screened first feature channel to generate a second image classification model. Since the second image classification model is constructed based on the screened first feature channel, and the first feature channel can be the key feature channel in the first convolution layer of the first image classification model, the second image classification model can better extract image features, so that the second image classification model has higher classification accuracy and robustness. That is, when feature extraction is performed through the second image classification model, the key features of the input data can be effectively retained, and the interference of irrelevant features can be reduced, thereby improving the model quality of the teacher model in the model knowledge distillation process; and since the second image feature classification model has better feature extraction capabilities, the second feature extraction model can extract feature information of higher dimensions, and the knowledge distillation of the student model is performed simultaneously through the first image classification model and the second image classification model, so that the student model can learn the feature information of the training data from multiple dimensions, thereby improving the classification accuracy of the student model and improving the knowledge distillation effect of the model.

[0028] Specifically, an embodiment of the present application provides a model knowledge distillation method. First, based on a training data set, a first feature channel is determined from multiple channels of a first convolutional layer of a first image classification model; then, based on the channel parameters of the first feature channel, the model structure of the first image classification model is enhanced to obtain a second image classification model; finally, through the first image classification model and the second image classification model, knowledge distillation is performed on the preset student model to obtain a trained student model.

[0029] Before explaining the model knowledge distillation method of the embodiment of the present application, here, we first explain the exemplary application of the model knowledge distillation device of the embodiment of the present application, which is an electronic device for implementing the model knowledge distillation method. In one implementation, the model knowledge distillation device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal or a server. In one implementation, the electronic device provided in the embodiment of the present application can be implemented as any terminal with model knowledge distillation function, such as a laptop computer, a tablet computer, a desktop computer, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device; in another implementation, the model knowledge distillation device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. Next, an exemplary application of the model knowledge distillation device when implemented as a server will be described.

[0030] See also Figure 1 , Figure 1 This is an optional architectural diagram of the model knowledge distillation system provided in the embodiment of the present application. The model knowledge distillation system 10 in the embodiment of the present application includes at least a terminal 100, a network 200, and a server 300. A model knowledge distillation application is deployed on the terminal 100, and the server 300 can be a background server of the model knowledge distillation application. The server 300 can constitute the model knowledge distillation device of the embodiment of the present application, that is, the model knowledge distillation method of the embodiment of the present application is implemented through the server 300. The terminal 100 is connected to the server 300 through the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two.

[0031] See also Figure 1The terminal 100 receives the user's model knowledge distillation operation and generates a model knowledge distillation request in response to the model knowledge distillation operation. Then, the terminal 100 sends the model knowledge distillation request to the server 300 via the network 200. After receiving the model knowledge distillation request, the server 300 responds to the model knowledge distillation request and obtains a training data set. Then, based on the training data set, the server 300 determines a first feature channel from multiple channels of the first convolutional layer of the first image classification model. Then, based on the channel parameters of the first feature channel, the server 300 performs model structure enhancement on the first image classification model to obtain a second image classification model. Then, the server 300 performs knowledge distillation on the preset student model using the first image classification model and the second image classification model to obtain a trained student model, and simultaneously generates a reminder message indicating that the model knowledge distillation is complete. Finally, the server 300 sends the reminder message indicating that the model knowledge distillation is complete to the terminal 100 via the network 200, and displays the reminder message indicating that the model knowledge distillation is complete on the display interface of the terminal 100.

[0032] In some embodiments, the above-mentioned model knowledge distillation method can also be executed by the terminal, that is, after receiving the user's model knowledge distillation operation, the terminal 100 can respond to the model knowledge distillation operation and obtain a training data set by the terminal 100; then, based on the training data set, the terminal 100 determines the first feature channel from multiple channels of the first convolutional layer of the first image classification model; then, based on the channel parameters of the first feature channel, the terminal 100 performs model structure enhancement on the first image classification model to obtain a second image classification model; then, the terminal 100 performs knowledge distillation on the preset student model through the first image classification model and the second image classification model to obtain a trained student model, and generates a reminder message that the model knowledge distillation is completed; finally, the terminal 100 displays the reminder message that the model knowledge distillation is completed on the display interface.

[0033] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown can be a model knowledge distillation device, which includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the model knowledge distillation device are coupled together through a bus system 240. It can be understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 240 .

[0034] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0035] The user interface 230 includes one or more output devices 231 that enable presentation of media content, and one or more input devices 232 .

[0036] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, and the like. The memory 250 may optionally include one or more storage devices physically located away from the processor 210. The memory 250 may include volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 250 may store data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

[0037] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; a network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; an input processing module 253 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0038] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2A model knowledge distillation device 254 stored in memory 250 is shown. This model knowledge distillation device 254 can be a model knowledge distillation device in an electronic device. It can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 2541, a structure enhancement module 2542, and a knowledge distillation module 2543. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be explained below.

[0039] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the model knowledge distillation method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0040] The model knowledge distillation method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the model knowledge distillation method provided in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0041] Figure 3 This is an optional flow chart of the model knowledge distillation method provided in the embodiment of the present application. Figure 3 The steps shown are explained as Figure 3 As shown, the execution subject of the model knowledge distillation method is a server as an example for description, and the method includes the following steps S101 to S103:

[0042] Step S101: Based on a training data set, determine a first feature channel from multiple channels of a first convolutional layer of a first image classification model.

[0043] Here, a training dataset refers to a data set used by a machine learning model or a deep learning model during the training process. A training dataset usually consists of labeled data, and each item of data in the training dataset is usually associated with a target label.

[0044] In some embodiments, the training dataset in step S101 can be a public dataset, a self-collected dataset, or a commercial dataset. When the training dataset is a public dataset, users can obtain it from the public resources of major data platforms or scientific research institutions; when the training dataset is a self-collected dataset, users can collect data in a real-world environment, such as by writing a crawler program to crawl data on the Internet; when the training dataset is a commercial dataset, users can purchase it on the corresponding merchant's platform according to their needs, such as professional datasets used in fields such as financial forecasting and medical diagnosis.

[0045] The first image classification model refers to the model to be distilled. The first image classification model is typically a deep learning model used to identify objects in an image and classify the identified objects into a certain category. The first image classification model can be implemented as a convolutional neural network, a long short-term memory neural network, etc., and the specific implementation method is not limited in this application.

[0046] A convolutional layer is a network layer in the first image classification model. It performs a convolution operation on the input image using a sliding convolution kernel to generate a feature map. Each convolution kernel in the convolution layer is used to extract different local features of the image, such as edge features, texture features, and color features. The first convolutional layer refers to any convolutional layer in the first image classification model.

[0047] Channels refer to the different features output by the convolutional layer, and each channel corresponds to a feature map. The first feature channel is the channel among the multiple channels of the convolutional layer that best reflects the key features of the input data.

[0048] In some embodiments, step S101 can be implemented by the following method: first, randomly sample the training data set to obtain sample data. It should be noted that only a small amount of data needs to be sampled as sample data, such as randomly selecting 1 / 5 of the data in the training data set as sample data; then, input the selected sample data into the first image classification model for feature extraction to obtain a first output value, wherein the first output value refers to the classification probability value predicted for the input image data.

[0049] Next, obtain the first feature map output by each channel in the first convolutional layer in the first image classification model, and add perturbation data to the first feature map output by each channel in sequence, such as setting the values ​​of some pixels in the central area of ​​the first feature map to zero; then, use the first feature map after adding the perturbation to replace the original first feature map, and continue feature extraction based on the replaced feature map to obtain a second output value, which refers to the predicted classification probability value obtained by using the feature map after adding the perturbation.

[0050] Next, a sensitivity score calculation is performed based on the second output value and the first output value corresponding to each channel in the first convolutional layer to obtain the sensitivity score of each channel, wherein the sensitivity score indicates the sensitivity of the channel to the disturbance value, and the higher the sensitivity score, the more important the channel. After obtaining the sensitivity score of each channel, the importance of each channel in the first convolutional layer is sorted according to the size of the sensitivity score to obtain the sorting result; finally, the first N channels in the sorting result are selected as the first feature channel of the first convolutional layer. Here, when selecting the first feature channel, a fixed number of channels can be selected from the sorting result as the first feature channel, such as selecting the first 5 channels as the first feature channel; or a pre-set selection ratio can be selected, such as selecting the first 10% of channels as the first feature channel. The specific selection method is not limited in this application.

[0051] Step S102: Based on the channel parameters of the first feature channel, the model structure of the first image classification model is enhanced to obtain a second image classification model.

[0052] Here, channel parameters refer to the weights and biases learned by the convolution kernel during training. Channel parameters determine the type of features extracted by each feature channel.

[0053] Model structure enhancement refers to optimizing and improving the existing model structure so that the improved model can better extract features or improve classification performance.

[0054] The second image classification model refers to the optimized first image classification model. Compared with the first image feature model, the second image feature classification model has better feature extraction capability and higher classification accuracy, and can better solve image classification problems.

[0055] In some embodiments, step S102 can be implemented by the following method: first, obtain the number of channels in the first convolutional layer and the number of first channels in the first convolutional layer; then, perform a division operation based on the number of channels and the number of first channels to obtain the number of copies and the number of screenings of the first feature channel, wherein the number of copies refers to how many times the channel parameters of multiple first feature channels are copied, and the number of screenings refers to how many channel parameters of the first feature channels are screened out from the multiple first feature channels; then, copy the channel parameters of the multiple first feature channels according to the number of copies, and screen out a corresponding number of channel parameters of the first feature channels from the multiple first feature channels according to the number of screenings; then, splice the channel parameters of the multiple copied first feature channels and the channel parameters of the first feature channels obtained after screening to obtain a second convolutional layer; finally, perform the same operation on each convolutional layer in the first image classification model to obtain multiple second convolutional layers, and use the multiple second convolutional layers to replace the corresponding first convolutional layers to obtain a second image classification model.

[0056] As an example of step S102, assuming that the number of channels in the first convolutional layer is 128 and the number of first feature channels is 10, a division operation is performed according to the number of channels and the number of first feature channels to obtain the number of replications and the number of screenings, such as: 128 / 10=12.8, then the number of replications is 11, and the number of screenings is 8; then, the channel parameters of the first feature channel are replicated 11 times, and the number of channel parameters of the obtained first feature channel is 10×11+10=120; then, according to the number of screenings, among the 10 first feature channels, the channel parameters of the first 8 first feature channels are screened in descending order of sensitivity scores.

[0057] Next, the channel parameters of the 120 copied first feature channels and the channel parameters of the 8 filtered feature channels are spliced ​​to obtain 128 channel parameters of the first feature channels. These 128 channel parameters constitute the second convolutional layer. Finally, the same operation is performed on all first convolutional layers to obtain multiple second convolutional layers. Multiple second convolutional layers are used to replace the corresponding first convolutional layers to obtain the second image classification model.

[0058] It should be noted that the first convolutional layer here may include all the convolutional layers in the first image classification model, or may include a specified portion of the convolutional layers. Moreover, the number of first feature channels in the second convolutional layer is the same as the number of channels in the first convolutional layer.

[0059] Step S103: Perform knowledge distillation on the preset student model through the first image classification model and the second image classification model to obtain a trained student model.

[0060] Here, the student model refers to a model with a relatively simple structure, usually designed to be lighter and less computationally expensive than the teacher model (the first and second image classification models mentioned above). The goal of the student model is to learn the knowledge contained in the teacher model through knowledge distillation.

[0061] Knowledge distillation is a model compression technology. The purpose of knowledge distillation is to transfer the knowledge from a large model (teacher model) to a small model (student model). Through knowledge distillation, the student model can learn richer feature representations and knowledge from the classroom model.

[0062] In some embodiments, step S103 can be implemented by the following method: first, freezing the network parameters in the first image classification model and the second image classification model, inputting the training data set into the first image classification model to obtain the output of the first image classification model; inputting the input data into the second image classification model to obtain the first convolution feature output of each enhanced convolution layer in the second image classification model; then, inputting the training data set into the preset student model to obtain the output of the student model and the second convolution feature output of each convolution layer in the student model; then, performing a first loss calculation based on the output of the first image classification model and the output of the student model to obtain a first loss result, and at the same time, performing a second loss calculation based on the first convolution feature and the second convolution feature to obtain a second loss result; finally, fusing the first loss result and the second loss result to obtain a final loss result, and adjusting the network parameters of the student model based on the final loss result. When the final loss result is stable within a preset range, the training is terminated to obtain a trained student model.

[0063] The model knowledge distillation method of the embodiment of the present application enhances the model structure of the first image classification model according to the channel parameters of the screened first feature channel to generate a second image classification model. Since the second image classification model is constructed based on the screened first feature channel, and the first feature channel can be the key feature channel in the first convolutional layer of the first image classification model, the second image classification model can better extract image features, so that the second image classification model has higher classification accuracy and robustness. That is, when feature extraction is performed through the second image classification model, the key features of the input data can be effectively retained, and the interference of irrelevant features can be reduced, thereby improving the model quality of the teacher model in the model knowledge distillation process; and since the second image feature classification model has better feature extraction capabilities, the second feature extraction model can extract feature information of higher dimensions, and the knowledge distillation of the student model is performed simultaneously through the first image classification model and the second image classification model, so that the student model can learn the feature information of the training data from multiple dimensions, thereby improving the classification accuracy of the student model and improving the knowledge distillation effect of the model.

[0064] The following examples illustrate the application scenarios of the model knowledge distillation method provided in the embodiments of the present application. The embodiments of the present application can be applied to at least the following exemplary scenarios:

[0065] Scenario 1: In the smart security scenario, the smart security system needs to quickly identify abnormal situations in the scene and classify them. Therefore, it is necessary to perform knowledge distillation on the trained image classification model and deploy the distilled model into the smart security system. In this way, the model knowledge distillation method provided in the embodiment of the present application can be used for distillation. First, the terminal receives the model knowledge distillation operation of the security personnel, generates a model knowledge distillation request, and sends the model knowledge distillation request to the server; then, the server responds to the model knowledge distillation request and obtains the training data set; then, based on the training data set, the first feature channel is determined from the multiple channels of the first convolutional layer of the first image classification model; then, based on the channel parameters of the first feature channel, the model structure of the first image classification model is enhanced to obtain the second image classification model; finally, the preset student model is subjected to knowledge distillation through the first image classification model and the second image classification model to obtain the trained student model. After obtaining the trained student model, the student model can be deployed into the smart security system.

[0066] Scenario 2: In the medical image analysis scenario, it is necessary to perform high-precision classification on large-scale medical image data. Therefore, it is necessary to perform knowledge distillation on the trained image classification model and deploy the distilled model to the medical image analysis system. In this way, the model knowledge distillation method provided in the embodiment of the present application can be used for distillation. First, the terminal receives the model knowledge distillation operation of the medical staff, generates a model knowledge distillation request, and sends the model knowledge distillation request to the server; then, the server responds to the model knowledge distillation request and obtains the training data set; then, based on the training data set, the first feature channel is determined from the multiple channels of the first convolutional layer of the first image classification model; then, based on the channel parameters of the first feature channel, the model structure of the first image classification model is enhanced to obtain the second image classification model; finally, the preset student model is subjected to knowledge distillation through the first image classification model and the second image classification model to obtain the trained student model. After obtaining the trained student model, the student model can be deployed to the medical image analysis system.

[0067] Based on the above scenario one, the model knowledge distillation method of the embodiment of the present application is explained. Figure 4 This is another optional flow chart of the model knowledge distillation method provided in the embodiment of the present application, such as Figure 4 As shown, the method includes the following steps S201 to S210:

[0068] Step S201: The terminal receives a model knowledge distillation operation.

[0069] Here, a model knowledge distillation application may be running on the terminal, and the server constitutes the backend server of the model knowledge distillation application. The model knowledge distillation operation may be a selection operation or an input operation inputted by the client of the model knowledge distillation application running on the terminal. For example, the selection operation may select the storage path of the training dataset for model knowledge distillation, or the input operation may be the user inputting the storage path of the training dataset for model knowledge distillation on the client, or the user may directly input the training dataset for model knowledge distillation on the client.

[0070] In some embodiments, the model knowledge distillation application may provide an input interface or input box that allows the user to select or enter the storage path of the dataset used for model knowledge distillation. The input interface may be in the form of a form, text box, or drop-down menu, and the specific form is not limited in this application. The user may select the storage path of the training dataset used for model knowledge distillation from predetermined options, or manually enter the storage path of the training dataset used for model knowledge distillation.

[0071] Step S202: The terminal generates a model knowledge distillation request in response to the model knowledge distillation operation.

[0072] Here, the terminal may encapsulate the storage path of the training data set selected or input by the user for model knowledge distillation into the model knowledge distillation request.

[0073] In some embodiments, in order to ensure that the storage path of the training data set selected or input by the user for model knowledge distillation is accurately included in the request data, first, it is necessary to check the integrity of the storage path information. In the process of filling the storage path information into the model knowledge distillation request, ensure that the storage path information is complete and accurate. The method of checking the integrity of the storage path includes checking whether the storage path contains the file name, file extension, and necessary characters such as slash or backslash; then, select a suitable data structure (such as JSON) to ensure that the storage path information is correctly embedded in the data structure; finally, special characters are processed. For special characters such as slash, backslash, quotation marks, etc., the special characters can be converted into a specific encoding format for correct transmission and parsing.

[0074] In step S203, the terminal sends a model knowledge distillation request to the server.

[0075] In some embodiments, the terminal sends the encapsulated model knowledge distillation request to the server and requests the server to perform the model knowledge distillation operation, usually using protocols such as HTTP or Web Socket to send the model knowledge distillation request.

[0076] Step S204: The server obtains a training data set in response to the model knowledge distillation request.

[0077] Here, after receiving the model knowledge distillation request, the server will parse the model knowledge distillation request. For example, for an HTTP request, the server can parse the request header and request body. The request header is parsed to obtain relevant information about the request; the request body is parsed to obtain the main body data of the request, that is, the storage path of the training dataset used for model knowledge distillation. Specific fields or parameters in the request body are parsed. These fields or parameters contain the storage path of the training dataset used for model knowledge distillation. A specific data format, such as JSON or XML, is extracted from the request body. The data format is then parsed to obtain the storage path of the training dataset used for model knowledge distillation. After parsing and obtaining the storage path of the training dataset, the training dataset is retrieved from the preset database based on the storage path.

[0078] In step S205 , the server determines a first feature channel from multiple channels of the first convolutional layer of the first image classification model based on the training data set.

[0079] Here, the first image classification model is a trained image classification model, typically a deep convolutional neural network. The input to the first image classification model is a training dataset, which includes a set of labeled image data. The first image classification model learns from the images and their labels in the training dataset and gradually adjusts its internal parameters, thereby enabling it to identify the categories of objects in images.

[0080] The first convolutional layer is any convolutional layer in the first image classification model. Its primary function is to extract image features, such as edges, corners, and textures, from the input image. Each convolutional layer convolves the input image data with multiple convolution kernels (or filters). Each time the convolution kernel slides over the image, it extracts specific features from the image. The output of a convolutional layer typically consists of multiple channels (also called feature maps), with each channel corresponding to a feature map generated by a convolution kernel.

[0081] The first feature channel refers to the channel that exhibits strong discrimination across the entire training dataset, or the channel that plays an important role in image classification. Among the multiple output channels, at least one feature channel that best represents some key information in the image needs to be selected as the first feature channel. Typically, the first feature channel is selected based on the relevance and importance of the features extracted from the channel.

[0082] In some embodiments, see Figure 5 , Figure 5 is a flow chart of determining a first feature channel based on a training data set provided in an embodiment of the present application, Figure 5It is shown that in step S205, the server determines the first feature channel from multiple channels of the convolution layer of the first image classification model based on the training data set, which can be achieved by the following steps S2051 to S2055:

[0083] Step S2051: Sample data is obtained from the training data set.

[0084] Here, sampling refers to the process of selecting a portion of data from a training dataset. Common sampling methods include random sampling, biased sampling, and stratified sampling. Random sampling involves randomly selecting samples from a training dataset without considering the relationships between samples; biased sampling involves selecting samples based on a specific strategy or criteria; and stratified sampling ensures that each sub-dataset is proportionally represented in the sample.

[0085] The sample data refers to a data set selected from the training data set and used to determine the first feature channel.

[0086] In some embodiments, taking stratified sampling as an example, step S2051 can be implemented by the following method: first, the training data set is divided into several layers according to the category label, wherein each layer contains samples with the same characteristics; then, according to the proportion of each layer in the training data set, it is determined how many samples to extract in each layer, such as: in a binary classification problem, the positive class accounts for 60% and the negative class accounts for 40%, then in stratified sampling, the sample ratios of the positive class and the negative class should also be 60% and 40%; then, samples are randomly selected in each layer according to the set ratio to ensure that the number of samples in each layer meets the predetermined ratio; finally, the samples extracted from each layer are merged to obtain sample data.

[0087] Step S2052: Perform feature mapping on the sample data through the first image classification model to obtain a first output value.

[0088] Here, the feature map refers to the operation of convolving the input data with the filter (convolution kernel) of the convolution layer. The first output value refers to the output result after processing by the first image classification model. In classification tasks, the first output value usually refers to the score or probability value of the category predicted by the first image classification model for a certain input sample.

[0089] In some embodiments, step S2052 can be implemented by the following method: first, the sample data is input into the first image classification model; then, the sample data is subjected to feature extraction through the convolutional layer in the first image classification model, and after feature extraction is performed in sequence through multiple convolutional layers, the extracted feature map is input into the fully connected layer; finally, the feature map is converted into a first output value through the fully connected layer.

[0090] As an example of step S2052, assume that the first image classification model is a convolutional neural network, and the input sample data is an image data containing a cat; then, the convolutional neural network is used to extract the features of the input sample data to obtain a feature map; finally, the feature map is converted into a predicted category and probability value through a fully connected layer, such as "the probability of the cat category is 0.7".

[0091] Step S2053: For each of the multiple channels, a disturbance value is added to the first feature map output by each channel to obtain a second feature map.

[0092] Here, the perturbation value is usually a small random number or noise. The perturbation value is added to each pixel of the original feature map to change the value of the feature map. The purpose of adding the perturbation value is to introduce some noise to determine the sensitivity of different channels to the noise value, thereby determining the importance of the channel.

[0093] In some embodiments, step S2053 can be implemented by the following method: First, obtain the first feature map output by the first channel, assuming that the first feature map is in the form of a matrix, as shown below:

[0094]

[0095] Then, the pixel values ​​in the central area of ​​the first feature map are set to zero to obtain the second feature map, which is in the form of:

[0096]

[0097] Step S2054: perform feature mapping on the second feature map to obtain a second output value corresponding to each channel.

[0098] Here, the second output value refers to the score or probability value of the category predicted based on the second feature map.

[0099] In some embodiments, after obtaining the second feature map, the first feature map in the first channel is replaced with the second feature map, and feature extraction is continued to obtain a second output value output by the first channel.

[0100] Step S2055: Determine a first characteristic channel from the multiple channels based on the first output value and the second output value corresponding to each channel.

[0101] Importance screening is performed based on the first output value and the second output value of each channel, and at least one relatively important channel is selected from the multiple channels as the first feature channel.

[0102] In some embodiments, step S2055 can be implemented by the following method: first, based on the first output value and the second output value corresponding to each channel, determine the first score of each channel; wherein the first score is used to characterize the sensitivity of the channel to the disturbance value; then, based on the first score, determine the first feature channel from multiple channels.

[0103] Here, the first score refers to the sensitivity of the channel to the disturbance value. The higher the first score, the higher the sensitivity, indicating that the channel is more important. Sensitivity is often used to describe the degree to which a model responds to a certain change (such as noise or disturbance).

[0104] As an example of step S2055, after determining the first output value and the second output value of each channel, a calculation is performed based on the first output value and the second output value of each channel to obtain a first score for each channel. The calculation formula of the first score is as follows:

[0105] As shown in formula 1:

[0106]

[0107] Where Δy c Indicates the first score; y c Represents the second output value; y base Indicates the first output value.

[0108] Then, calculate the first score corresponding to each channel in the first convolutional layer; then, sort the first scores of all channels in the first convolutional layer, wherein the sorting is performed in descending order according to the first scores; then, select the top-n channels in the sorting result as the first feature channels, wherein n is an artificially defined hyperparameter, and n can be a fixed value or a ratio. For example: the first convolutional layer has a total of 10 channels. After sorting according to the first score corresponding to each channel, the sorting result is obtained. If n=5, the first 5 channels in the sorting result are selected as the first feature channels. If n=50%, the first 50% channels in the sorting result are selected as the first feature channels. The specific value of n can be set according to the actual situation, and this application does not limit it here.

[0109] In step S206, the server performs model structure enhancement on the first image classification model based on the channel parameters of the first feature channel to obtain a second image classification model.

[0110] Channel parameters refer to the weights and biases learned by the convolution kernel during training. Each channel is described by channel parameters, which determine how the channel responds to input features. The optimization and adjustment of channel parameters directly affect the learning ability and final performance of the model. Usually, channel parameters are optimized through the backpropagation algorithm during training.

[0111] Model structure enhancement refers to improving and optimizing the model structure to improve the performance of the model. Model structure enhancement can be achieved in many ways, such as adding new layers, optimizing the connection method of the network, adjusting the parameters of existing layers, etc.

[0112] In some embodiments, see Figure 6 , Figure 6 is a flow chart of obtaining a second image classification model based on the channel parameters of the first feature channel provided in an embodiment of the present application, Figure 6 It is shown that in step S206, the server performs model structure enhancement on the first image classification model based on the channel parameters of the first feature channel to obtain a second image classification model, which can be achieved by the following steps S2061 to S2064:

[0113] Step S2061: Obtain a first number of channels in the first convolutional layer and a second number of first feature channels in the first convolutional layer.

[0114] Here, the first number of channels refers to the total number of channels included in the first convolutional layer. The first number of channels can be determined by an existing library function.

[0115] The second number of first feature channels refers to the number of first feature channels screened out from the multiple channels of the first convolutional layer. The second number of first feature channels can be directly obtained according to the screening process of step S205.

[0116] Step S2062: Determine the number of copies and the number of screenings of the first feature channel based on the first number and the second number.

[0117] Here, the number of replications refers to how many times the parameters of the first feature channel are replicated, and the number of screenings refers to the number of first feature channels screened out from the second number of first feature channels.

[0118] In some embodiments, step S2062 can be implemented by performing a division operation on the first quantity and the second quantity to obtain a result, then determining the number of copies based on the integer portion of the result, and determining the number of screening based on the decimal portion of the result. For example, if the first quantity is 128 and the second quantity is 10, and the result of the division operation is 12.8, then the number of copies can be determined to be 11 and the number of screening can be determined to be 8.

[0119] Step S2063: Based on the channel parameters, the number of replications, and the number of screenings of the first feature channel, the convolutional layer structure is enhanced on the first convolutional layer to obtain a second convolutional layer.

[0120] Here, after determining the number of replications and the number of screenings, the convolutional layer structure in the first convolutional layer is optimized according to the channel parameters, the number of replications, and the number of screenings of the first feature channel to obtain the second convolutional layer.

[0121] In some embodiments, step S2063 can be implemented by the following method: first, the channel parameters of the first feature channel are copied according to the number of copies to obtain a first channel parameter set; then, the channel parameters of the filtered number of first feature channels are filtered out from the second number of first feature channels to obtain a second channel parameter set; finally, a second convolutional layer is constructed based on the first channel parameter set and the second channel parameter set.

[0122] As an example of step S2063, following the example of step S2062 above, when the number of replications is 11 and the number of screenings is 8, the parameters of the 10 first feature channels are replicated 11 times to obtain 10×11+10=120 channel parameters of the first feature channels; then, among the 10 first feature channels, the channel parameters of the first 8 first feature channels are screened out in descending order of the first score of each feature channel; finally, the channel parameters of the 120 replicated channels and the channel parameters of the 8 screened first feature channels are concatenated to obtain 128 channel parameters of the first feature channels, and the second convolutional layer is constructed based on the channel parameters of the 128 feature channels.

[0123] Step S2064: Replace the first convolutional layer in the first image classification model with the second convolutional layer to obtain a second image classification model.

[0124] Here, the second convolutional layer constructed using the first feature channel is used to replace the first convolutional layer corresponding to the second convolutional layer in the first image classification model, thereby obtaining a second image classification model. The first convolutional layer corresponding to the second convolutional layer means that the second convolutional layer is constructed based on the first feature channel selected in the first convolutional layer.

[0125] In some embodiments, the first convolutional layer in the first image classification model includes the following two cases:

[0126] Case 1: The first convolutional layer includes all convolutional layers in the first image classification model.

[0127] Under the conditions of case one, step S2064 can be implemented by the following method: first, obtain the second convolutional layer corresponding to each convolutional layer in the first image classification model; then, replace each convolutional layer in the first image classification model with the corresponding second convolutional layer to obtain the second image classification model.

[0128] As an example of step S2064, when the first image classification model contains a total of 20 convolutional layers, and all 20 convolutional layers are first convolutional layers, determine the second convolutional layer corresponding to each first convolutional layer in these 20 first convolutional layers; then, use the second convolutional layer corresponding to each first convolutional layer to replace the first convolutional layer of the object to obtain a second image classification model.

[0129] Case 2: The first convolutional layer includes part of the convolutional layers in the first image classification model.

[0130] Under the conditions of case 2, step S2064 can be implemented by the following method: first, obtain the second convolutional layer corresponding to each convolutional layer in the partial convolutional layer; then, replace each convolutional layer in the partial convolutional layer with the corresponding second convolutional layer to obtain a second image classification model.

[0131] As an example of step S2064, when the first image classification model includes a total of 20 convolutional layers, and only 10 of them are set as first convolutional layers, the second convolutional layer corresponding to each of the 10 first convolutional layers is determined; then, the first convolutional layer of the object is replaced by the second convolutional layer corresponding to each first convolutional layer to obtain a second image classification model.

[0132] It should be noted that steps S2061 to S2064 optimize the model structure of the first image classification model through the channel parameters of the first feature channel to obtain the second image classification model, thereby enhancing the feature expression ability of the feature extraction ability of the second image classification model, and providing a higher quality teacher model for subsequent model knowledge distillation, thereby improving the classification effect of the student model.

[0133] In step 207 , the server performs knowledge distillation on the preset student model through the first image classification model and the second image classification model to obtain a trained student model.

[0134] The student model is an untrained model with a relatively simple structure. The student model learns knowledge and feature information from the teacher model (the first image classification model and the second image classification model in the embodiment of the present application) through knowledge distillation.

[0135] In some embodiments, step S207 can be implemented by the following method: first, inputting the training data in the training data set into the first image classification model to obtain the first model output value; then, inputting the training data into the second image classification model to obtain the first convolution feature output by the second convolution layer in the second image classification model; then, inputting the training data into the preset student model to obtain the second model output value and the second convolution feature output by the convolution layer in the student model; then, determining the model loss result based on the first model output value, the second model output value, the first convolution feature and the second convolution feature; finally, based on the model loss result, adjusting the parameters of the student model to obtain the trained student model.

[0136] Here, the first model output value refers to the prediction result obtained by the first image classification model on the training data in the training dataset, which is usually a probability distribution indicating the prediction probability of each category.

[0137] The second model output value refers to the prediction result obtained by the student model for the training data in the training dataset.

[0138] The first convolution feature refers to a feature map obtained by extracting features from the training data through the second convolution layer in the second image classification model. The first convolution feature represents a local feature of the training data extracted under the second convolution layer.

[0139] The second convolutional feature refers to the feature map obtained by extracting features from the training data through the convolutional layer in the student model. The second convolutional feature represents the local features of the training data extracted under the convolutional layer.

[0140] The model loss result is a metric used to evaluate the difference between the output predicted by the student model and the outputs predicted by the first and second image classification models. Typically, the model loss result is determined using a preset loss function. The smaller the model loss result, the more similar the student model's output is to the outputs of the first and second image classification models.

[0141] Parameter adjustment refers to updating the model's parameters based on the results of a calculation using an optimization algorithm (such as gradient descent) to improve the model's performance. Specifically, parameter adjustment is achieved through optimization algorithms such as backpropagation and gradient descent. During each iterative training process, the loss result is calculated by distilling the loss function. Next, the contribution of the parameters in the student model to the loss result (i.e., the gradient) is calculated using the backpropagation algorithm. Then, the gradient descent algorithm is used to update the model parameters of the student model based on the gradient value, reducing the value of the loss function and stabilizing it within a preset range. The goal of parameter adjustment is to gradually bring the output of the student model closer to the output of the teacher model by minimizing the loss function.

[0142] In some embodiments, determining the model loss result based on the first model output value, the second model output value, the first convolution feature and the second convolution feature can be achieved by the following method: first, determining the first loss result based on the first model output value and the second model output value; then, determining the second loss result based on the first convolution feature and the second convolution feature; finally, fusing the first loss result and the second loss result to obtain the model loss result.

[0143] The first loss result is a loss function result calculated by comparing the difference between the first model output value and the second model output value. The second loss result is a loss function result calculated by comparing the difference between the first convolution feature and the second convolution feature.

[0144] The first loss result and the second loss result are fused and calculated to obtain the final model loss result. The calculation formulas for the first loss result, the second loss result and the model loss result are shown in Formula 2:

[0145]

[0146] Among them, loss represents the model loss result; loss L Represents the second loss result (the first term in Formula 2); loss KD represents the first loss result (the second and third terms in Formula 2); r is a customizable loss weight hyperparameter; k represents the number of convolutional layers in the first image sentiment classification model; i represents the number of convolutional layers; ||.|| represents the first norm; ||.||2 represents the second norm; represents the first convolution feature; Indicates that the feature map of the first convolution feature is normalized; Represents the second convolution feature; Indicates the normalization operation of the feature map of the second convolution feature; α is a hyperparameter; CE represents the cross entropy loss function; p s represents the output value of the second model; label represents the label; KL is the KL divergence, which is an indicator used to measure the difference between two probability distributions; p t Represents the output value of the first model; T is the temperature hyperparameter, which will decrease with the number of training times. The temperature hyperparameter is used to adjust the output distribution of the first image classification model to make the output of the first image classification model smoother, thereby giving the student model more information and guidance.

[0147] Step S208: The server generates a reminder message indicating that the model knowledge distillation is completed.

[0148] In step S209, the server sends a reminder message indicating that the model knowledge distillation is completed to the terminal.

[0149] In step S210, the terminal displays a reminder message on the current interface indicating that the model knowledge distillation is completed.

[0150] After confirming that the model knowledge distillation is completed, the trained student model can be obtained and deployed.

[0151] The model knowledge distillation method provided in the embodiment of the present application performs model structure enhancement on the first image classification model according to the channel parameters of the screened first feature channel to generate a second image classification model. Since the second image classification model is constructed based on the screened first feature channel, and the first feature channel can be the key feature channel in the first convolutional layer of the first image classification model, the second image classification model can better extract image features, so that the second image classification model has higher classification accuracy and robustness. That is, when feature extraction is performed through the second image classification model, the key features of the input data can be effectively retained, and the interference of irrelevant features can be reduced, thereby improving the model quality of the teacher model in the model knowledge distillation process; and since the second image feature classification model has better feature extraction capabilities, the second feature extraction model can extract feature information of higher dimensions, and the knowledge distillation of the student model is performed simultaneously through the first image classification model and the second image classification model, so that the student model can learn the feature information of the training data from multiple dimensions, thereby improving the classification accuracy of the student model and improving the knowledge distillation effect of the model.

[0152] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0153] An embodiment of the present application provides a feature-enhanced knowledge distillation framework for image sentiment classification, which can perform knowledge distillation on a trained image sentiment classification model (the above-mentioned first image classification model), distill the image sentiment classification model while considering key channel features (the above-mentioned first feature channel), and deploy the model.

[0154] An embodiment of the present application provides a feature-enhanced knowledge distillation method for image sentiment classification, which can perform knowledge distillation on a trained image sentiment classification model while considering key channel features. This can not only better maintain the performance of the image sentiment classification model, but also help the model accelerate reasoning.

[0155] See also Figure 7 , Figure 7 This is a diagram of the model distillation architecture provided by the embodiment of this application. The knowledge distillation framework provided by this application is divided into four modules. The names and functions of these four modules are introduced below:

[0156] 1) Data sampling module 702 : used to perform data sampling in the training data set 701 to obtain a sampled data set (the above-mentioned sample data), and send the sampled data set to the key feature channel screening module 703 .

[0157] In some embodiments, the process of performing data sampling by the data sampling module 702 is described in step S2051 , which will not be further described in this application.

[0158] 2) Key feature channel screening module 703: used to screen out key feature channels from the image sentiment classification model 704 to be distilled based on the sampling data set, and inform the model distillation module 705.

[0159] In some embodiments, the specific implementation process of screening key feature channels by the key feature channel screening module 703 is shown in step S205, which will not be described in detail in this application.

[0160] 3) Model distillation module 705: used to perform knowledge distillation on the image sentiment classification model 704 to be distilled using the training data set 701 while considering key feature channels, and send the trained student model to the model delivery module 706.

[0161] In some embodiments, the specific implementation process of knowledge distillation through the model distillation module 705 is shown in steps S206 to S207, and this application will not go into details here.

[0162] 4) Model distribution module 706: used to deploy the trained student model.

[0163] After the student model training is completed, the trained student model is deployed to the actual application scenario through the model distribution module.

[0164] The following is an introduction to the implementation process of each module:

[0165] 1) Data acquisition module 702:

[0166] The data acquisition module 702 collects a training data set and samples several picture samples (the above-mentioned sample data) from the training data set. The sampling process can be implemented using existing sampling methods such as simple random sampling and stratified sampling. The model distillation method provided in the embodiment of the present application does not require sampling too much data. Therefore, a smaller data sampling quantity requirement (for example, 1-2 batches of samples) can be given for sampling. The obtained sampled data set will be sent to the key feature channel screening module 703 for screening the more critical channels in the image sentiment classification model 704 to be distilled.

[0167] 2) Key feature channel screening module 703:

[0168] The key feature channel screening module 703 uses the sampled data set to analyze the convolutional layers in the image sentiment classification model 704 to be distilled, and screens out the channels that play a key role (a convolutional layer consists of multiple channels). The specific screening process is as follows:

[0169] (1) Prepare data:

[0170] Input sample: Select the sampling data set x (the above sample data);

[0171] Model: Load the image sentiment classification model h to be distilled (the first image classification model mentioned above);

[0172] Target layer: Determine the convolution layer m to be analyzed (the first convolution layer mentioned above), assuming that the convolution layer m has C channels (the multiple channels mentioned above).

[0173] (2) Baseline assessment:

[0174] Forward propagation: Use the sampled dataset x to perform forward propagation through the image sentiment classification model h to be distilled (the above feature map), and record the output y_base of the image sentiment classification model to be distilled (the above first output value).

[0175] Extract feature map: Extract the feature map F of the target layer m (the first feature map mentioned above). The shape of the feature map F is H, W, C, where H and W are the height and width of the feature map F, and C is the number of channels of the feature map F.

[0176] (3) Design disturbance:

[0177] Perturbation method: Use zero filling to set all pixel values ​​within the height and width range of 0.5H and 0.5W in the central area of ​​the feature map F to zero.

[0178] Perturbation range: The feature maps corresponding to each channel are perturbed separately.

[0179] (4) Execution perturbation:

[0180] Initialize the result storage: Create a list or array to store the sensitivity score of each channel (the first score mentioned above). Iterate over each channel and perform the following operations for each channel c (from 0 to C-1):

[0181] a. Copy feature map: Copy feature map F to get F′.

[0182] b. Perturbed feature map: Set the pixel values ​​within the height and width range of 0.5H, 0.5W in the central area of ​​the feature map of channel c to zero (adding perturbations as mentioned above), that is, F′[:,:,c]=0.

[0183] c. Restore the feature map: Re-inject the perturbed feature map F′ (the second feature map mentioned above) into the target layer m of the model h.

[0184] d. Forward propagation: Use the perturbed feature map F′ to forward propagate through the model h, and record the model's output y_c (the second output value mentioned above).

[0185] e. Calculate changes: Calculate the difference between the output y_c after the disturbance and the baseline output y_base. The relative change can be used to calculate the sensitivity score of the channel (the first score mentioned above). The calculation formula for the sensitivity score is shown in Formula 1 above, and this application will not repeat it here.

[0186] f. Store the result: store the sensitivity score Δy of channel c c Store into the created list or array.

[0187] (5) Sorting and filtering:

[0188] Sort channels: Sort channels by sensitivity scores stored in a list or array, from highest to lowest.

[0189] Output result: Output the sorted channels and their corresponding sensitivity scores. A high sensitivity score indicates that the channel is more important.

[0190] Sorting: Select the top-n important channels in each convolutional layer as the key feature channels (the first feature channel mentioned above), where n is a manually definable hyperparameter, which can be a fixed value or a ratio.

[0191] The key feature channels of each convolutional layer in the image sentiment classification model to be distilled obtained through the above analysis will be informed to the model distillation module 705 as the key focus channels during distillation.

[0192] In the embodiment of the present application, the specific implementation process of designing disturbance, executing disturbance and sorting screening is shown in steps S2051 to S2055, which will not be repeated here.

[0193] 3) Model distillation module 705:

[0194] See also Figure 8 , Figure 8 Schematic diagram of the model distillation process provided in the embodiment of the present application. The distillation process of the model distillation module can be implemented by the following method:

[0195] First, the model distillation module 705 generates an enhanced image sentiment classification model 803 (the above-mentioned second image classification model) based on the key feature channels collected from the key feature channel screening module 703. In the enhanced image sentiment classification model 803, each convolution layer (the above-mentioned second convolution layer) only contains the key feature channels that have been screened out, and the key feature channels are repeatedly superimposed and combined to form the same shape as the original convolution layer (that is, the same number of channels). For example, convolution layer A has 128 channels, and its key feature channels only contain 10. In the enhanced image sentiment classification model, convolution layer A will contain 12*10 key feature channels after key feature channel enhancement, and 8 channels with higher importance in the key feature channels, for a total of 128 channels. For neural network layers other than the convolution layer, they are consistent with the image sentiment classification model 802 to be distilled in the enhanced image sentiment classification model 803.

[0196] Then, the original image sentiment classification model 802 is efficiently distilled by re-defining the distillation loss function that takes into account the key feature channels. For the specific loss calculation, please refer to step S207, which will not be repeated in this application.

[0197] During the training process of the student image sentiment classification model 804, the network parameters of the image sentiment classification model to be distilled 802 and the enhanced image sentiment classification model 803 are frozen and used to train the student image sentiment classification model 804 (the aforementioned preset student model). The training dataset 801 is the same as the dataset used to train the image sentiment classification model to be distilled 802.

[0198] The trained student image sentiment classification model (the above-mentioned trained student model) is sent to the model sending module 706 .

[0199] 4) Model distribution module 704:

[0200] The trained student image sentiment classification model is actually deployed and distributed, so that the trained student image sentiment classification model can be applied to various scenarios that require image sentiment classification.

[0201] An embodiment of the present application proposes a feature-enhanced knowledge distillation method for image sentiment classification, including screening of key feature channels of the model and using key feature channels for more efficient knowledge distillation, so that the student model can more fully learn the image sentiment classification model, thereby improving the effect of model distillation.

[0202] It is understandable that in the embodiments of the present application, if data related to user information or corporate information is involved, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0203] The following continues to describe the exemplary structure of the model knowledge distillation device 254 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the model knowledge distillation device 254 includes: a determination module 2541, used to determine the first feature channel from multiple channels of the first convolutional layer of the first image classification model based on the training data set; a structure enhancement module 2542, used to determine the first feature channel from multiple channels of the first convolutional layer of the first image classification model based on the training data set; a knowledge distillation module 2543, used to perform knowledge distillation on the preset student model through the first image classification model and the second image classification model to obtain a trained student model.

[0204] In the above scheme, the determination module is also used to: sample sample data from the training data set; perform feature mapping on the sample data through the first image classification model to obtain a first output value; for each channel in the multiple channels, add a disturbance value to the first feature map output by each channel to obtain a second feature map; perform feature mapping on the second feature map to obtain a second output value corresponding to each channel; and determine the first feature channel from the multiple channels based on the first output value and the second output value corresponding to each channel.

[0205] In the above scheme, the determination module is also used to: determine the first score of each channel based on the first output value and the second output value corresponding to each channel; the first score is used to characterize the sensitivity of the channel to the disturbance value; based on the first score, determine the first feature channel from the multiple channels.

[0206] In the above scheme, the structure enhancement module is also used to: obtain a first number of channels in the first convolutional layer and a second number of first feature channels in the first convolutional layer; determine the number of replications and the number of screenings of the first feature channel based on the first number and the second number; perform convolution layer structure enhancement on the first convolutional layer based on the channel parameters of the first feature channel, the number of replications and the number of screenings to obtain a second convolutional layer; replace the first convolution layer in the first image classification model with the second convolution layer to obtain the second image classification model.

[0207] In the above scheme, the structure enhancement module is also used to: copy the channel parameters of the first feature channel according to the number of replications to obtain a first channel parameter set; filter out the channel parameters of the filtered number of first feature channels from the second number of the first feature channels to obtain a second channel parameter set; and construct the second convolution layer based on the first channel parameter set and the second channel parameter set.

[0208] In the above scheme, the structure enhancement module is also used to: when the first convolutional layer includes all the convolutional layers in the first image classification model, obtain the second convolutional layer corresponding to each convolutional layer in the first image classification model; replace each convolutional layer in the first image classification model with the corresponding second convolutional layer to obtain the second image classification model; when the first convolutional layer includes some convolutional layers in the first image classification model, obtain the second convolutional layer corresponding to each convolutional layer in the some convolutional layers; replace each convolutional layer in the some convolutional layers with the corresponding second convolutional layer to obtain the second image classification model.

[0209] In the above scheme, the knowledge distillation module is also used to: input the training data in the training data set into the first image classification model to obtain a first model output value; input the training data into the second image classification model to obtain the first convolution feature output by the second convolution layer in the second image classification model; input the training data into the preset student model to obtain a second model output value and the second convolution feature output by the convolution layer in the student model; determine the model loss result based on the first model output value, the second model output value, the first convolution feature and the second convolution feature; and adjust the parameters of the student model based on the model loss result to obtain the trained student model.

[0210] In the above scheme, the knowledge distillation module is also used to: determine a first loss result based on the first model output value and the second model output value; determine a second loss result based on the first convolution feature and the second convolution feature; and fuse the first loss result and the second loss result to obtain the model loss result.

[0211] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0212] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned model knowledge distillation method when executing the executable instructions stored in the memory.

[0213] An embodiment of the present application provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned model knowledge distillation method is implemented.

[0214] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, Figure 3 The method shown.

[0215] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0216] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0217] As an example, the executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed in multiple locations and interconnected by a communication network.

[0218] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A model knowledge distillation method, characterized in that: The method comprises: Determining, based on the training dataset, a first feature channel from a plurality of channels of a first convolutional layer of a first image classification model; Based on the channel parameters of the first feature channel, the first image classification model is enhanced to obtain a second image classification model; Through the first image classification model and the second image classification model, knowledge distillation is performed on the preset student model to obtain a trained student model.

2. The method according to claim 1, characterized in that The determining, based on the training data set, a first feature channel from a plurality of channels of a first convolutional layer of a first image classification model comprises: Sampling from the training data set to obtain sample data; Performing feature mapping on the sample data using the first image classification model to obtain a first output value; For each of the multiple channels, add a disturbance value to a first feature map output by each channel to obtain a second feature map; Performing feature mapping on the second feature map to obtain a second output value corresponding to each channel; The first characteristic channel is determined from the multiple channels based on the first output value and the second output value corresponding to each channel.

3. The method according to claim 2, characterized in that The determining the first characteristic channel from the plurality of channels based on the first output value and the second output value corresponding to each channel includes: Determining a first score for each channel based on the first output value and a second output value corresponding to each channel; the first score is used to characterize the sensitivity of the channel to the disturbance value; The first feature channel is determined from the plurality of channels based on the first score.

4. The method according to claim 1, wherein The step of performing model structure enhancement on the first image classification model based on the channel parameters of the first feature channel to obtain a second image classification model includes: Obtaining a first number of channels in the first convolutional layer and a second number of first feature channels in the first convolutional layer; Determining the number of replications and the number of screenings of the first feature channel based on the first number and the second number; Based on the channel parameters of the first feature channel, the number of replications, and the number of screenings, performing convolutional layer structure enhancement on the first convolutional layer to obtain a second convolutional layer; The first convolutional layer in the first image classification model is replaced by the second convolutional layer to obtain the second image classification model.

5. The method according to claim 4, characterized in that The step of performing convolutional layer structure enhancement on the first convolutional layer based on the channel parameter of the first feature channel, the number of replications, and the number of screenings to obtain a second convolutional layer includes: Copy the channel parameters of the first characteristic channel according to the number of copies to obtain a first channel parameter set; Filtering the channel parameters of the filtered number of first feature channels from the second number of first feature channels to obtain a second channel parameter set; The second convolutional layer is constructed based on the first channel parameter set and the second channel parameter set.

6. The method according to claim 4, characterized in that The step of replacing the first convolutional layer in the first image classification model with the second convolutional layer to obtain the second image classification model includes: In a case where the first convolutional layer includes all convolutional layers in the first image classification model, obtaining a second convolutional layer corresponding to each convolutional layer in the first image classification model; replacing each convolutional layer in the first image classification model with the corresponding second convolutional layer to obtain the second image classification model; In the case that the first convolutional layer includes a partial convolutional layer in the first image classification model, obtain a second convolutional layer corresponding to each convolutional layer in the partial convolutional layer; replace each convolutional layer in the partial convolutional layer with the corresponding second convolutional layer to obtain the second image classification model.

7. The method according to claim 4, characterized in that The method of performing knowledge distillation on a preset student model through the first image classification model and the second image classification model to obtain a trained student model includes: Inputting the training data in the training data set into the first image classification model to obtain a first model output value; Inputting the training data into the second image classification model to obtain a first convolution feature output by the second convolution layer in the second image classification model; Inputting the training data into the preset student model to obtain a second model output value and a second convolution feature output by the convolution layer in the student model; Determining a model loss result based on the first model output value, the second model output value, the first convolution feature, and the second convolution feature; Based on the model loss result, the parameters of the student model are adjusted to obtain the trained student model.

8. The method according to claim 7, characterized in that The determining a model loss result based on the first model output value, the second model output value, the first convolution feature, and the second convolution feature includes: Determining a first loss result based on the first model output value and the second model output value; Determining a second loss result based on the first convolution feature and the second convolution feature; The first loss result and the second loss result are fused to obtain the model loss result.

9. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor is configured to implement the model knowledge distillation method according to any one of claims 1 to 8 when executing executable instructions stored in the memory.

10. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the model knowledge distillation method according to any one of claims 1 to 8 is implemented.

Images