Model training method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence. The model training method comprises the following steps: acquiring a training sample set which comprises at least one image sample; inputting the image sample into a student model to obtain a first feature of the image sample; inputting the image sample into a teacher model to obtain a second feature of the image sample; inputting the image sample into a teaching assisting network, and performing feature extraction on the image sample by using a first feature extraction network; performing feature dimension conversion and mapping on the features extracted by the first feature extraction network by using a conversion network; performing feature extraction on the converted features by using a second feature extraction network to obtain third features; determining a first loss function according to the first feature, the second feature and the third feature; and performing parameter updating on the student model according to the first loss function to obtain a trained student model. According to the embodiment of the invention, the accuracy of cross-structure knowledge distillation can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, advanced deep neural networks have achieved higher accuracy with more parameters or training data. However, the additional costs of training, storage, and inference hinder their application on edge devices. For example, VisionTransformer (ViT)-based models have shown promising performance compared to efficient Convolutional Neural Networks (CNN) models, but they are difficult to deploy on mobile devices due to their low inference speed and high storage requirements. To balance performance and cost, a common solution is knowledge distillation, which improves a compact student model by learning knowledge from a pre-trained, cumbersome teacher model. This knowledge is typically a prediction or intermediate feature of the teacher model's final output.

[0003] Most knowledge distillation methods focus on isomorphic distillation, where the teacher and student models have similar architectures. In this case, the choice of teacher models is limited to those structurally similar to the student models, leading to suboptimal performance from the available teacher models. Furthermore, when proposing new models, it is difficult to find teacher models with similar architectures, hindering the application of homogeneous distillation. Therefore, researching how to distill cross-architecture teacher knowledge into student models has significant practical implications. Further improving the accuracy of cross-architecture knowledge distillation remains a pressing issue. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, and storage medium that can help improve the accuracy of cross-structure knowledge distillation.

[0005] In a first aspect, embodiments of this application provide a model training method, including:

[0006] Obtain a training sample set, which includes at least one image sample;

[0007] The image sample is input into the student model to obtain the first feature of the image sample;

[0008] The image sample is input into the teacher model to obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models.

[0009] The image sample is input into the teaching assistant network. A first feature extraction network is used to extract features from the image sample to obtain features with the same dimension as the first feature. A transformation network is used to transform and map the feature dimension of the features extracted by the first feature extraction network. A second feature extraction network is used to extract features from the transformed features to obtain a third feature. The teaching assistant network includes the first feature extraction network, the transformation network, and the second feature extraction network. The first feature extraction network is obtained based on the student model, and the second feature extraction network is obtained based on the teacher model. The third feature has the same feature dimension and feature space as the second feature.

[0010] Based on the first feature, the second feature, and the third feature, a first loss function is determined;

[0011] The parameters of the student model are updated according to the first loss function to obtain the trained student model.

[0012] Secondly, embodiments of this application provide a model training apparatus, comprising:

[0013] An acquisition unit is used to acquire a training sample set, the training sample set including at least one image sample;

[0014] A student model is used to input the image sample and obtain the first feature of the image sample;

[0015] The teacher model is used to input the image sample and obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models.

[0016] A teaching assistant network is used to input the image sample, extract features from the image sample using a first feature extraction network to obtain features with the same dimension as the first feature, transform and map the feature extracted by the first feature extraction network using a transformation network, and extract a third feature from the transformed feature using a second feature extraction network. The teaching assistant network includes the first feature extraction network, the transformation network, and the second feature extraction network. The first feature extraction network is obtained based on the student model, and the second feature extraction network is obtained based on the teacher model. The third feature has the same feature dimension and feature space as the second feature.

[0017] The determining unit determines a first loss function based on the first feature, the second feature, and the third feature;

[0018] The parameter update unit updates the parameters of the student model according to the first loss function to obtain the trained student model.

[0019] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0021] Fifthly, embodiments of this application provide a computer program product including computer program instructions that cause a computer to perform the method as described in the first aspect.

[0022] In a sixth aspect, embodiments of this application provide a computer program that causes a computer to perform the method as described in the first aspect.

[0023] This application embodiment designs a teaching assistant network based on a student model and a teacher model. The teaching assistant network connects a first feature extraction network from the student model and a second feature extraction network from the teacher model through a transformation network. The features output by the first feature extraction network are transformed in feature dimension and then input into the second feature extraction network, thereby obtaining a third feature with the same dimension and feature space as the second feature output by the teacher model. Compared with the related technology, which projects the intermediate features of the student model into the prediction space and damages the complete information of the student model features, the third feature in this application embodiment can directly align with the second feature, thus helping to preserve the complete information of the student model features. This can help reduce the representational differences between the cross-structure teacher model and the student model, thereby improving the accuracy of cross-structure knowledge distillation. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of the solution in an embodiment of this application;

[0025] Figure 2 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of the feature extraction process of a CNN model;

[0027] Figure 4 A schematic diagram of a three-layer distillation paradigm provided in an embodiment of this application;

[0028] Figure 5 A schematic diagram of the feature extraction process for multi-head self-attention;

[0029] Figure 6A schematic flowchart illustrating another model training method provided in an embodiment of this application;

[0030] Figure 7 This is a schematic block diagram of a model training apparatus according to an embodiment of this application;

[0031] Figure 8 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0034] In the description of this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. Additionally, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0035] It should also be understood that the descriptions of "first", "second", etc. appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and there is no order to them. They do not indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0036] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0037] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0038] The embodiments of this application are applied to the field of artificial intelligence technology.

[0039] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0040] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0042] This application's embodiments may relate to Computer Vision (CV) technology within artificial intelligence. Computer vision is a science that studies how to enable machines to "see," or more specifically, machine vision that uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0043] This application's embodiments may also relate to Machine Learning (ML) in artificial intelligence technology. ML is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating the above techniques.

[0044] Deep learning (DL) is a branch of machine learning that attempts to perform high-level abstractions of data using multiple processing layers containing complex structures or multiple nonlinear transformations. Deep learning learns the inherent patterns and hierarchical representations of training sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm, and its performance in speech and image recognition far surpasses previous related technologies.

[0045] Figure 1This is a schematic diagram of a system architecture involved in an embodiment of this application. For example... Figure 1 As shown, the system architecture may include user equipment 101, data acquisition equipment 102, training equipment 103, execution equipment 104, database 105, and content library 106.

[0046] The data acquisition device 102 is used to read training data from the content library 106 and store the read training data in the database 105. The training data involved in this embodiment includes image samples and target classes (or annotations, labels, etc.) of the image samples.

[0047] Training device 103 trains a machine learning model based on training data maintained in database 105. The machine learning model obtained by training device 103 can effectively recognize images, such as performing image classification, object detection, and semantic segmentation. For example, the machine learning module can be a discriminative deep learning model such as an image classification model, object detection model, or semantic segmentation model; this application does not limit this. Optionally, the machine learning model obtained by training device 103 can be applied to different systems or devices.

[0048] Additionally, refer to Figure 1 The execution device 104 is equipped with an I / O interface 107 for data interaction with external devices. For example, it receives images sent by the user device 101 through the I / O interface. The computing module 109 in the execution device 104 processes the input image using a trained machine learning model, outputs prediction results, and sends the corresponding prediction results to the user device 101 through the I / O interface.

[0049] User equipment 101 may include mobile phones, computers, smart voice interaction devices, wearable devices, smart home appliances, in-vehicle equipment, aircraft, or other terminal devices.

[0050] The execution device 104 can be a server. For example, the server can be a rack server, blade server, tower server, or cabinet server, etc. The server can be a standalone server or a server cluster composed of multiple servers.

[0051] In this embodiment, the execution device 104 is connected to the user equipment 101 via a network. The network can be an intranet, the Internet, the Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, voice communication network, or other wireless or wired networks.

[0052] Optionally, in other embodiments, the machine learning model can also be deployed on the user device 101. After acquiring an image, the user device 101 processes the image using the trained machine learning model and outputs a prediction result.

[0053] It should be noted that, Figure 1 This is merely a schematic diagram of a system architecture provided in this application embodiment, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. In some embodiments, the data acquisition device 102, user device 101, training device 103, and execution device 104 may be the same device. The database 105 may be distributed across one server or multiple servers, and the content library 106 may be distributed across one server or multiple servers.

[0054] In practical applications, the choice of teacher models in isomorphic distillation is limited to models structurally similar to student models, resulting in suboptimal performance of the available teacher models. For example, when using appropriate methods to extract knowledge from ViT into ResNet50, the student model outperforms using ResNet152 as the teacher model. Furthermore, when proposing new models, it is difficult to find teacher models with similar architectures, hindering the application of isomorphic distillation. Therefore, researching how to distill knowledge from cross-architecture teachers into student models has significant practical implications.

[0055] One of the most challenging aspects of cross-structure knowledge distillation is the significant representational gap between cross-structure models, stemming from the inherent inductive biases and model functions of different structures. Inductive bias refers to the set of assumptions a model bases on when predicting unknown data, while the model function represents how the model reads, encodes, decodes, and processes data. For example, CNN-based models have hard inductive biases, namely locality and translation invariance. Therefore, CNN model features are generated using local receptive fields produced from local pixels. Multi-headed self-attention (MSA) and multi-layer perceptron (MLP) models have softer inductive biases, namely long-range dependencies. Therefore, most MSA and MLP-based models generate features by exchanging information from global patches. Furthermore, different model functions can produce different features at different stages. For instance, shallow and deep features in a ViT model exhibit higher similarity than features in a hierarchical CNN. Therefore, compared with isomorphic knowledge distillation, cross-structure models have greater differences in inductive bias and model function, resulting in significant representation gaps between heterogeneous models and hindering knowledge transfer.

[0056] Cross-architecture knowledge distillation refers to situations where the teacher and student models are heterogeneous networks. For example, the student model might be based on a CNN, while the teacher model might be based on a Transformer architecture, such as ViT. In related solutions, for this heterogeneous situation, the intermediate features of the student model are projected into the prediction space, and the predictions from the teacher model are used to supervise the intermediate features of the student model. However, mapping the student model's features to the prediction space impairs the integrity of the student model's features, resulting in low accuracy in cross-architecture knowledge distillation.

[0057] In view of this, embodiments of this application provide a model training method, apparatus, electronic device, and storage medium, which are beneficial to improving the accuracy of cross-structure knowledge distillation.

[0058] Specifically, the process involves: acquiring a training sample set, which includes at least one image sample; inputting the image sample into a student model to obtain a first feature of the image sample; inputting the image sample into a teacher model to obtain a second feature of the image sample; the student model and the teacher model are heterogeneous models; inputting the image sample into a teaching assistant network, where a first feature extraction network is used to extract features from the image sample to obtain features with the same dimension as the first feature; a transformation network is used to transform and map the feature dimensions of the features extracted by the first feature extraction network; and a second feature extraction network is used to extract features from the transformed features to obtain a third feature; a first loss function is determined based on the first, second, and third features; and the parameters of the student model are updated based on the first loss function to obtain a trained student model. The teaching assistant network includes a first feature extraction network, a transformation network, and a second feature extraction network. The first feature extraction network is obtained from the student model, and the second feature extraction network is obtained from the teacher model. The third feature has the same feature dimension and feature space as the second feature.

[0059] This application embodiment designs a teaching assistant network based on a student model and a teacher model. The teaching assistant network connects a first feature extraction network from the student model and a second feature extraction network from the teacher model through a transformation network. The features output by the first feature extraction network are transformed in feature dimension and then input into the second feature extraction network, thereby obtaining a third feature with the same dimension and feature space as the second feature output by the teacher model. Compared with the related technology, which projects the intermediate features of the student model into the prediction space and damages the complete information of the student model features, the third feature in this application embodiment can directly align with the second feature, thus helping to preserve the complete information of the student model features. This can help reduce the representational differences between the cross-structure teacher model and the student model, thereby improving the accuracy of cross-structure knowledge distillation.

[0060] This application can be applied to discriminative deep learning models such as image classification, object detection, and semantic segmentation. Discriminative models directly model conditional probabilities, representing a method for modeling the relationship between unobserved data y and observed data x. For example, a pre-trained Transformer-based model, such as the ViT model, can be used as the teacher model, leveraging its features or predicted distributions as knowledge to improve the accuracy of the student model. The student model can be a CNN-based model. For example, the student model can be an image classification model, object detection model, semantic segmentation model, etc., without limitation. In the cross-architecture knowledge distillation process, this application's embodiments design a teaching assistant network based on the student and teacher models. This network transforms the feature dimensions of features from the student model before inputting them into the teacher model, resulting in features with the same dimensions and feature space as the teacher model's output features. This aligns the student model's output features with the teacher model's features, reducing representational differences between the cross-architecture teacher and student models and thus improving the accuracy of cross-architecture knowledge distillation.

[0061] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0062] Figure 2 This is a schematic flowchart illustrating a model training method 200 provided in an embodiment of this application. Method 200 can be executed by any electronic device with data processing capabilities; for example, the electronic device can be implemented as a server or a terminal device. Figure 1 The training device 103 in this application is not limited thereto. Figure 2 As shown, method 200 includes steps 210 to 260.

[0063] 210. Obtain the training sample set, which includes at least one image sample.

[0064] For example, image samples of the corresponding image type can be obtained according to the application scenario of the model, such as face image samples, object image samples, scene image samples, etc., without limitation.

[0065] For example, existing public datasets such as imageNet and COCO can be used to obtain image samples, and this application does not limit this to that.

[0066] Optionally, image samples can be labeled and classified. For example, object bounding boxes and key points can be labeled, or image samples can be classified according to application requirements to determine the target class of the image samples.

[0067] In some embodiments, a test sample set may also be obtained, which includes at least one image sample. The test sample set is used to evaluate the model's performance.

[0068] 220. Input the image sample into the student model to obtain the first feature of the image sample.

[0069] Specifically, image samples can be input into a student model, which can then be used to extract features from the image samples to obtain the first feature of the image samples.

[0070] In some embodiments, the student model includes a convolutional neural network (CNN). For example, the student model can be a CNN-based model.

[0071] Figure 3 This diagram illustrates the feature extraction process of a CNN model. Figure 3 As shown, the convolution kernel performs convolution on local pixel values ​​of the input image and outputs the convolution result (i.e., output features). The convolution kernel only performs convolution operations on local pixels of the input image; these local pixels are the local receptive fields of the convolution kernel. The convolution kernel continuously traverses and convolves on the input image with a specific stride to obtain the output features of the entire image. Therefore, the first feature obtained by the CNN model from feature extraction of image samples is the local receptive field generated using local pixels, which can be called a local feature.

[0072] Figure 4 The diagram illustrates a three-layer distillation paradigm provided in this application embodiment. The three-layer distillation paradigm consists of a teacher model, a teaching assistant network, and a student model. The teaching assistant network can integrate the inductive bias and model function of the teacher model and the student model, thereby reducing the feature representation differences between the cross-structure teacher model and the student model and improving the accuracy of cross-structure distillation.

[0073] For example, see Figure 4 Image samples x can be input into a CNN model, represented as Here, 'i' represents the i-th stage of the CNN model. For example, the CNN model in the diagram is divided into 4 stages according to its layer structure, such as every 3 convolutional layers forming one stage. Thus, the input image is sequentially processed in... The convolution process is performed, with the output of the previous stage serving as the input of the next stage. After average pooling, the feature f is obtained. s (An example of the first feature). As an example, in a CNN model, a feature can be represented as (B, C, H, W), where B is the number of data in the batch, C is the dimension of the feature, and H and W are the height and width of the feature.

[0074] 230. Input the image sample into the teacher model to obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models.

[0075] Specifically, image samples can be input into the teacher model, and the teacher model can be used to extract features from the image samples to obtain the second features of the image samples.

[0076] In some embodiments, the teacher model includes a multi-head self-attention model (MSA) or a multilayer perceptron (MLP). For example, the teacher model can be a Transformer-based model.

[0077] Figure 5 A schematic diagram of the feature extraction process of multi-head self-attention is shown. Figure 5 As shown, the input image is segmented into image patches, linearly mapped using embedding, and then processed using three matrices W. Q W K and W V The query (Q), key (K), and value (V) are obtained by multiplying them separately. The similarity between the Q vector and all K vectors is calculated (usually using dot product or cosine similarity), and then normalized using a softmax function to obtain the attention weight for each K vector. Next, the attention weights are weighted and summed over the V vector to obtain the output feature. This weighted summation process combines local attention weights with the global V vector, thereby achieving the transformation from local features to global features and capturing more complex feature relationships. Therefore, multi-head self-attention-based models have a softer inductive bias, i.e., long-distance dependency, and their output features are generated by exchanging information from global patches, and can be called global features.

[0078] For example, see [link to previous article] Figure 4 Image samples x can be input into the MSA / MLP model, represented as Where i represents the i-th stage of the MSA / MLP model. For example, the MSA / MLP model has the same number of stages as the CNN model, i.e., 4 stages, such as a stacked structure of 4 MSA / MLP layers constituting one stage. Thus, the input image is sequentially processed in... The process involves processing the data, with the output of the previous stage serving as the input to the next stage. After pooling, the feature f is obtained. t (An example of a second feature). As an example, in an MSA / MLP model, a feature can be represented as (B, L, D), where B is the number of data in a batch, L is the number of feature slices, and D is the feature dimension.

[0079] 240. Input the image sample into the teaching assistant network, use the first feature extraction network to extract features from the image sample to obtain features with the same dimension as the first feature; use the transformation network to transform and map the feature dimension of the features extracted by the first feature extraction network; and use the second feature extraction network to extract features from the transformed features of the transformation network to obtain the third feature.

[0080] The teaching assistant network includes a first feature extraction network, a transformation network, and a second feature extraction network. The first feature extraction network is obtained based on the student model, and the second feature extraction network is obtained based on the teacher model. The third feature has the same feature dimension and feature space as the second feature.

[0081] In other words, the first feature extraction network can have the same or similar network structure as the student model, and the second feature extraction network can have the same or similar network structure as the teacher model. For example, the first feature extraction network may include a CNN, and the second feature extraction network may include MSA or MLP; this application does not limit this.

[0082] Optionally, in some embodiments, the first feature extraction network is identical to at least one preceding feature extraction network layer of the student model and shares parameters, and the second feature extraction network has the same structure as at least one subsequent feature extraction network layer of the teacher model and shares parameters.

[0083] For example, the first feature extraction network may include the network structure corresponding to at least one of the preceding stages of the student model. For instance, when the student model is divided into four stages, the first feature extraction network may be the network structure corresponding to the first three stages of the student model. In other words, the first feature extraction network has the same network structure as the first three stages of the student model and shares parameters. Similarly, the second feature extraction network may include the network structure corresponding to at least one of the following stages of the teacher model. For instance, when the teacher model is divided into four stages, the second feature extraction network may be the network structure corresponding to the last stage of the teacher model. In other words, the second feature extraction network has the same network structure as the last stage of the teacher model and shares parameters.

[0084] A transformation network is positioned between the first and second feature extraction networks to connect them. Since the first and second feature extraction networks have different network structures, their corresponding features do not belong to the same feature dimension or feature space. The transformation network can transform the feature dimension of the features output by the first feature extraction network, converting them to the same feature dimension as the second feature extraction network. Thus, the features output by the transformation network can be input into the second feature extraction network, making the third feature output by the second feature extraction network have the same feature dimension and feature space as the second feature. Furthermore, since the first feature extraction network is identical to at least one preceding feature extraction network layer of the student model and shares parameters, and the second feature extraction network is structurally identical to at least one subsequent feature extraction network layer of the teacher model and shares parameters, the teaching assistant network (or transformation network) can achieve the fusion of the inductive biases and model functions of the teacher and student models.

[0085] In some embodiments, the transformation network includes a patch embedding (PE) layer. For example, PE can be implemented using convolutional layers, and this application is not limited to this. Figure 6 As shown, the feature dimension transformation and mapping of the features extracted by the first feature extraction network are performed using a transformation network, which can be specifically implemented as follows: 241.

[0086] 241. The feature input block embedded layer PE extracted by the first feature extraction network is sliced, and each feature block after slicing is linearly mapped to a one-dimensional vector to obtain the fourth feature. The fourth feature has the same feature dimension as the second feature.

[0087] For example, the features extracted by the first feature extraction network are completely equivalent to the features extracted by the learning model, such as (B, C, H, W). This feature input PE is sliced, for example, into L feature blocks. Further, each feature block is linearly mapped to obtain a one-dimensional vector with dimension D. In this way, a fourth feature can be obtained, which can be represented as (B, L, D).

[0088] Specifically, the first feature extraction network, including CNNs, extracts features based on image data, with a feature format of (B, C, H, W). The second feature extraction network, including MSA / MLP, is a Transformer-based model, whose feature format is clearly not what the second feature extraction network desires. Therefore, a block embedding layer (such as embedding) is needed to transform the data. First, the image features can be divided into patches of a given size. For example, an input image (224×224) can be divided into 196 patches of 16×16. Then, each patch is mapped to a one-dimensional vector using a linear mapping. For example, each patch has a shape of (16×16×3), so the mapping yields a vector of length 768.

[0089] Optionally, in some embodiments, the transformation network further includes a self-attention module. See also: Figure 6 The step 241 may be followed by the following step 242:

[0090] 242. Input the fourth feature into the self-attention module to obtain the transformed feature.

[0091] Specifically, although the fourth feature is obtained by transforming the feature dimension of the output of the first feature extraction network, since the first feature extraction network and the student network have similar feature extraction methods, when the student model includes a CNN, the features extracted by the first feature extraction network are local features and cannot represent the global information of the input image. Therefore, the fourth feature can be input into the attention module to process the local fourth feature so that the output feature contains global information, i.e., the global feature.

[0092] Specifically, the attention module multiplies the input features by matrix W. Q W K and W V We obtain Q, K, and V vectors. The similarity between the Q vector and all K vectors is calculated (usually using dot product or cosine similarity), and then normalized using a softmax function to obtain the attention weight for each K vector. Next, the attention weights are weighted and summed over the V vector to obtain the output feature. This weighted summation process combines the local attention weights with the global V vector, thereby achieving the transformation from local features to global features, that is, projecting local features onto the global receptive field.

[0093] Optionally, the self-attention module can include a multi-head self-attention module. Specifically, each head corresponds to a set of matrices W. Q W K and W VThis yields a set of Q, K, and V vectors, which are then used to output the corresponding global features. Concatenating these global features from multiple heads, such as by linking the vectors end-to-end, yields the final output features.

[0094] It should be understood that the implementation of this method does not limit the network structure of the transformation network in the teaching assistant network. For example, it may include different MSAs, such as different MSA structures, feature dimensions, lengths, etc.

[0095] For example, see [link to previous article] Figure 4 Image samples can be input into the teaching assistant network, including CNN models. Local to Global (L2G) and MSA / MLP CNN model L2G is an example of the first feature extraction network mentioned above, and MSA / MLP is an example of the transformation network mentioned above. This is an example of the second feature extraction network mentioned above. The CNN model can include... It can be used in the student model Shared parameter (Weight-Sharing) acquisition, MSA / MLP can include It can be used in the teacher model Shared parameter acquisition. L2G can include PE and MSA.

[0096] like Figure 4 As shown, image sample x passes through The resulting features are completely equivalent to those of the CNN model and can be represented as (B, C, H, W). L2G then slices the features (B, C, H, W) using PE and performs a linear mapping to obtain the features (B, L, D). At this point, the features (B, L, D) are still local CNN features. Next, L2G projects the local CNN features onto the global receptive field using MSA, thus transforming the local features into global features. The L2G output features are then further input into... The obtained features will be able to be directly aligned with the output features of the MSA / MLP model. Then, The output features are pooled to obtain feature f. a (An example of the third feature). For instance, a teaching assistant network can satisfy the following formula (1):

[0097]

[0098] It should be noted that this embodiment uses the L2G module connecting the first three stages of the student model and the fourth stage of the teacher model as an example for illustration, but this embodiment is not limited to this. For example, the L2G module, i.e., the teaching assistant network, can achieve diversification of the teaching assistant network by connecting different stages of the student model and the teacher model.

[0099] 250. Determine the first loss function based on the first feature, the second feature, and the third feature.

[0100] Specifically, the knowledge K of the teacher model, teaching assistant network, and student model t K a K s The final features output by the teacher model, teaching assistant network, and student model, respectively, namely the second feature, the first feature, and the third feature (e.g., ...). Figure 4 f in t f s f a This is represented by the first feature, second feature, and third feature. Based on this, a loss function, such as the first loss function, can be determined to update the model parameters.

[0101] In some embodiments, a first feature loss may be determined based on a first feature and a third feature; a second feature loss may be determined based on a second feature and a third feature; a third feature loss may be determined based on a first feature and a second feature; and the aforementioned first loss function may be determined based on the first feature loss, the second feature loss, and the third feature loss.

[0102] For example, the first loss function can be the sum of the first feature loss, the second feature loss, and the third feature loss.

[0103] For example, see [link to previous article] Figure 4 It can be based on feature f s and feature f a Determine the feature loss between the student model output features and the teaching assistant network output features, i.e., the first feature loss; this can be determined based on feature f. t and feature f a Determine the feature loss between the teacher model output features and the teaching assistant network output features, i.e., the second feature loss; this can be determined based on feature f. t and feature f s The feature loss between the output features of the teacher model and the output features of the teaching assistant network is determined, namely the third feature loss.

[0104] For example, let f be the output feature of the teacher model. t The student model outputs features f s For example, feature f t and feature f sThe feature loss function L between InfoNCE (f s ,f t ) satisfies the following formula (2):

[0105]

[0106] in, F represents positive sample features, i.e., features obtained from the same image for both the teacher and student models; t τ represents the set of all sample features; τ2 is a learnable parameter used to soften the probability output.

[0107] Similarly, feature f t and feature f a The feature loss function L between InfoNCE (f a ,f t ), feature f s and feature f a The feature loss function L between InfoNCE (f a ,f s ) and L InfoNCE (f s ,f t Similar to formula (2), it can be obtained by simple adaptation.

[0108] In some embodiments, a first predicted probability distribution of the sample image may be obtained based on a first feature; a second predicted probability distribution of the sample image may be obtained based on a second feature; and a third predicted probability distribution of the sample image may be obtained based on a third feature.

[0109] Specifically, each feature can be input into the corresponding fully connected layer to obtain the corresponding predicted probability distribution.

[0110] Optionally, the third feature can be input into the fully connected layer to obtain the third predicted probability distribution; wherein, the fully connected layer shares parameters with the fully connected layer of the teacher model.

[0111] For details, please refer to [link / reference]. Figure 4 The feature f can be s The fully connected (FC) layer of the input student model is denoted as fc. s The first predicted probability distribution of the sample image is obtained, which can be represented as p. s (x), feature f t The full-joint (FC) of the input teacher model is denoted as fc. t The second predicted probability distribution of the sample image is obtained, which can be represented as p. t (x), feature f aInput the FC of the teacher model, i.e., fc t The third predicted probability distribution of the sample image is obtained, which can be represented as p. a (x).

[0112] Continuing with the example above, for the feature f of the teaching assistant network output shown in formula (1) above... a The corresponding probability distribution p a (x), can be expressed as the following formula (3):

[0113]

[0114] Among them, fc m The term FC, representing the teacher network, originates from the teacher model itself.

[0115] In some embodiments, a second loss function may be determined based on a first predicted probability distribution, a second predicted probability distribution, a third predicted probability distribution, and the target class of the sample image.

[0116] Specifically, the knowledge K of the teacher model, teaching assistant network, and student model t K a K s The final predicted probability distributions output by the teacher model, teaching assistant network, and student model, respectively, namely the second predicted probability distribution, the first predicted probability distribution, and the third predicted probability distribution (as shown in p above). t (x), p s (x), p a (x) represents this. Based on this, the loss function, such as the second loss function, can be determined to update the model parameters according to the target class of the sample images from the first, second, and third predicted probability distributions.

[0117] In some embodiments, a first probability loss may be determined based on a first predicted probability distribution, a third predicted probability distribution, and a target class; a second probability loss may be determined based on a second predicted probability distribution, a third predicted probability distribution, and a target class; a third probability loss may be determined based on the first predicted probability distribution, the second predicted probability distribution, and the target class; and a second loss function may be determined based on the first probability loss, the second probability loss, and the third probability loss.

[0118] For example, the second loss function can be the sum of the first probability loss, the second prediction probability loss, and the third prediction probability loss.

[0119] For example, it can be based on a first predicted probability distribution p s (x), Third prediction probability distribution p aGiven (x) and the target class of the image sample, determine the probability loss between the predicted probability distribution output by the student model and the predicted probability distribution output by the teaching assistant network, i.e., the first probability loss; and then determine the second predicted probability distribution p. t (x), Third prediction probability distribution p a Given (x) and the target class of the image sample, determine the probability loss between the predicted probability distribution output by the teacher model and the predicted probability distribution output by the teaching assistant network, i.e., the second probability loss; this can be determined based on the first predicted probability distribution p. s (x), second prediction probability distribution p t (x) and the target class of the image sample, determine the probability loss between the predicted probability distribution output by the student model and the predicted probability distribution output by the teacher model, i.e., the third probability loss.

[0120] For example, let p be the predicted probability distribution output by the teacher model. t The student model outputs a predicted probability distribution of p. s For example, predict the probability distribution p t and p s The feature loss function L between OFA (p t ,p s It satisfies the following formula (4):

[0121]

[0122] in, c and c represent the target class and predicted class of the input image, respectively; C represents all classes in the dataset. Represents the target class output by the teacher model. The predicted probability distribution; Represents the target class output by the student model. The predicted probability distribution; This represents the probability distribution of the predicted class c output by the teacher model; γ represents the predicted probability distribution of the predicted class c output by the student model; γ represents an adjustable parameter used to enhance the information of the target class when the teacher model is not confident in its prediction.

[0123] Similarly, predict the probability distribution p t and p a The feature loss function L between OFA (p t ,p a ), predict probability distribution p s and p a The feature loss function L between OFA (p s ,p s ) and L OFA (pt ,p s Similar to this, formula (4) can be easily adapted to obtain the result.

[0124] 260. Update the parameters of the student model according to the first loss function to obtain the trained student model.

[0125] For example, the parameters of the student model, or the parameters of the student model and the teaching assistant network, can be updated according to a loss function, such as a first loss function, or a first loss function and a second loss function, to obtain a trained student model. For example, the parameters of the first feature extraction network and the transformation network in the teaching assistant network can be updated.

[0126] Optionally, the sum of the first loss function and the second loss function can be used as the final loss function to update the parameters of the student model or teaching assistant network. This application embodiment does not limit this.

[0127] For example, see [link to previous article] Figure 4 The feature loss L between the student model and the teaching assistant network InfoNCE (f a ,f s ) and probability loss L OFA (p s ,p a The sum of these can be considered as the total loss between the student model and the teaching assistant network, i.e. Right now Feature loss L between teacher model and teaching assistant network InfoNCE (f a ,f t ) and probability loss L OFA (p t ,p a The sum of these values ​​can be considered the total loss between the teacher model and the teaching assistant network, i.e. Right now Feature loss L between teacher model and student network InfoNCE (f s ,f t ) and probability loss L OFA (p t ,p s The sum of these values ​​can be considered the total loss between the teacher model and the student model, i.e. Right now The final total loss L of the system can be expressed as the following formula (5):

[0128]

[0129] For example, the parameter update module in the system can update the parameters of each module in the student model, or the parameters of each module in the student model and the teaching assistant network, based on the final total loss L of the system, using the gradient descent algorithm, until the training stop condition is met, and output the student model determined when the training stop condition is met as the trained model.

[0130] In some embodiments, for the case where the conversion network includes a block embedding layer and a self-attention module (such as a multi-head self-attention module), since only one convolutional layer and a self-attention layer are used, the embodiments of this application introduce fewer training parameters compared to related schemes, which is beneficial to achieve knowledge distillation with lower training costs.

[0131] Therefore, this embodiment designs a teaching assistant network based on a student model and a teacher model. The teaching assistant network connects the first feature extraction network from the student model and the second feature extraction network from the teacher model through a transformation network. The feature output by the first feature extraction network is transformed in feature dimension and then input into the second feature extraction network, thereby obtaining a third feature with the same dimension and feature space as the second feature output by the teacher model. Compared with the related technology, which projects the intermediate features of the student model into the prediction space and damages the complete information of the student model features, the third feature in this embodiment can directly align with the second feature, thus helping to preserve the complete information of the student model features. This can help reduce the representational differences between the cross-structure teacher model and the student model, thereby improving the accuracy of cross-structure knowledge distillation.

[0132] The specific embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0133] It should also be understood that, in the various method embodiments of this application, the sequence numbers of the above processes do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that these sequence numbers can be interchanged where appropriate so that the embodiments of this application described can be implemented in a sequence other than those illustrated or described.

[0134] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figures 7 to 8 The following describes in detail the device embodiments of this application.

[0135] Figure 7 This is a schematic block diagram of the model training device 10 according to an embodiment of this application. Figure 7 As shown, the device 10 may include an acquisition unit 11, a student model 12, a teacher model 13, a teaching assistant network 14, a determination unit 15, and a parameter update unit 16.

[0136] Acquisition unit 11 is used to acquire a training sample set, the training sample set including at least one image sample;

[0137] Student model 12 is used to input the image sample and obtain the first feature of the image sample;

[0138] Teacher model 13 is used to input the image sample and obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models;

[0139] The teaching assistant network 14 is used to input the image sample, extract features from the image sample using a first feature extraction network to obtain features with the same dimension as the first feature, transform and map the feature extracted by the first feature extraction network using a transformation network, and extract features from the transformed features using a second feature extraction network to obtain a third feature; wherein, the teaching assistant network includes the first feature extraction network, the transformation network, and the second feature extraction network, the first feature extraction network is obtained based on the student model, the second feature extraction network is obtained based on the teacher model; the third feature has the same feature dimension and feature space as the second feature;

[0140] The determining unit 15 determines a first loss function based on the first feature, the second feature, and the third feature;

[0141] The parameter update unit 16 updates the parameters of the student model according to the first loss function to obtain the trained student model.

[0142] Optionally, in some embodiments, the transformation network 14 includes a block embedding layer, and the teaching assistant network 14 is specifically used for:

[0143] The features extracted by the first feature extraction network are input into the block embedding layer for slicing, and each feature block after slicing is linearly mapped to a one-dimensional vector to obtain the fourth feature; the fourth feature has the same feature dimension as the second feature.

[0144] Optionally, in some embodiments, the transformation network 14 further includes a self-attention module, and the teaching assistant network 14 is specifically used for:

[0145] The fourth feature is input into the self-attention module to obtain the transformed feature.

[0146] Optionally, in some embodiments, the self-attention module includes a multi-head self-attention module.

[0147] Optionally, in some embodiments, the determining unit 15 is further configured to:

[0148] Based on the first feature, obtain the first predicted probability distribution of the sample image;

[0149] Based on the second feature, obtain the second predicted probability distribution of the sample image;

[0150] Based on the third feature, obtain the third predicted probability distribution of the sample image;

[0151] A second loss function is determined based on the first predicted probability distribution, the second predicted probability distribution, the third predicted probability distribution, and the target class of the sample image.

[0152] Optionally, in some embodiments, the parameter update unit 16 is specifically used for:

[0153] The parameters of the student model are updated according to the first loss function and the second loss function to obtain the trained student model.

[0154] Optionally, in some embodiments, the determining unit 15 is specifically used for:

[0155] The third feature is input into the fully connected layer to obtain the third predicted probability distribution; wherein the fully connected layer shares parameters with the fully connected layer of the teacher model.

[0156] Optionally, in some embodiments, the determining unit 15 is specifically used for:

[0157] Based on the first feature and the third feature, determine the first feature loss;

[0158] The second feature loss is determined based on the second feature and the third feature;

[0159] Based on the first feature and the second feature, determine the third feature loss;

[0160] The first loss function is determined based on the first feature loss, the second feature loss, and the third feature loss.

[0161] Optionally, in some embodiments, the determining unit 15 is specifically used for:

[0162] A first probability loss is determined based on the first predicted probability distribution, the third predicted probability distribution, and the target class;

[0163] The second probability loss is determined based on the second predicted probability distribution, the third predicted probability distribution, and the target class;

[0164] A third probability loss is determined based on the first predicted probability distribution, the second predicted probability distribution, and the target class;

[0165] The second loss function is determined based on the first probability loss, the second probability loss, and the third probability loss.

[0166] Optionally, in some embodiments, the first feature extraction network is identical to at least one preceding feature extraction network layer of the student model and shares parameters, and the second feature extraction network is identical in structure to at least one subsequent feature extraction network layer of the teacher model and shares parameters.

[0167] Optionally, in some embodiments, the student model includes a convolutional neural network.

[0168] Optionally, in some embodiments, the teacher model includes a multi-head self-attention model or a multilayer perceptron.

[0169] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 7 The apparatus 10 shown can perform the above-described method embodiments, and the foregoing and other operations and / or functions of each module in the apparatus 10 are respectively for implementing the above-described methods. Figure 2 The corresponding process in method 200 will not be elaborated here for the sake of brevity.

[0170] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0171] Figure 8 This is a schematic block diagram of the electronic device 30 provided in the embodiments of this application.

[0172] like Figure 8 As shown, the electronic device 30 may include:

[0173] The system includes a memory 31 and a processor 32. The memory 31 stores computer programs and transfers the program code to the processor 32. In other words, the processor 32 can retrieve and run the computer programs from the memory 31 to implement the methods described in the embodiments of this application.

[0174] For example, the processor 32 can be used to execute the above-described model training method according to instructions in the computer program, including:

[0175] Obtain a training sample set, which includes at least one image sample;

[0176] The image sample is input into the student model to obtain the first feature of the image sample;

[0177] The image sample is input into the teacher model to obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models.

[0178] The image sample is input into the teaching assistant network. A first feature extraction network is used to extract features from the image sample to obtain features with the same dimension as the first feature. A transformation network is used to transform and map the feature dimension of the features extracted by the first feature extraction network. A second feature extraction network is used to extract features from the transformed features to obtain a third feature. The teaching assistant network includes the first feature extraction network, the transformation network, and the second feature extraction network. The first feature extraction network is obtained based on the student model, and the second feature extraction network is obtained based on the teacher model. The third feature has the same feature dimension and feature space as the second feature.

[0179] Based on the first feature, the second feature, and the third feature, a first loss function is determined;

[0180] The parameters of the student model are updated according to the first loss function to obtain the trained student model.

[0181] In some embodiments of this application, the processor 32 may include, but is not limited to:

[0182] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0183] In some embodiments of this application, the memory 31 includes, but is not limited to:

[0184] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0185] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0186] like Figure 8 As shown, the electronic device 30 may further include:

[0187] Transceiver 33, which can be connected to processor 32 or memory 31.

[0188] The processor 32 can control the transceiver 33 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 33 may further include antennas, and the number of antennas may be one or more.

[0189] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0190] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0191] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0192] It is understood that in the specific implementation of this application, when the above embodiments of this application are applied to specific products or technologies and involve user information and other related data, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0193] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0195] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0196] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, include: Obtain a training sample set, which includes at least one image sample; The image sample is input into the student model to obtain the first feature of the image sample; The image sample is input into the teacher model to obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models. The image sample is input into the teaching assistant network. A first feature extraction network is used to extract features from the image sample to obtain features with the same dimension as the first feature. A transformation network is used to transform and map the feature dimension of the features extracted by the first feature extraction network. A second feature extraction network is used to extract features from the transformed features to obtain a third feature. The teaching assistant network includes the first feature extraction network, the transformation network, and the second feature extraction network. The first feature extraction network is obtained based on the student model, and the second feature extraction network is obtained based on the teacher model. The third feature has the same feature dimension and feature space as the second feature. Based on the first feature, the second feature, and the third feature, a first loss function is determined; The parameters of the student model are updated according to the first loss function to obtain the trained student model.

2. The method according to claim 1, characterized in that, The transformation network includes a block embedding layer. The step of using the transformation network to perform feature dimension transformation and mapping on the features extracted by the first feature extraction network includes: The features extracted by the first feature extraction network are input into the block embedding layer for slicing, and each feature block after slicing is linearly mapped to a one-dimensional vector to obtain the fourth feature; the fourth feature has the same feature dimension as the second feature.

3. The method according to claim 2, characterized in that, The transformation network further includes a self-attention module, and the method further includes: The fourth feature is input into the self-attention module to obtain the transformed feature.

4. The method according to claim 3, characterized in that, The self-attention module includes a multi-head self-attention module.

5. The method according to claim 1, characterized in that, Also includes: Based on the first feature, obtain the first predicted probability distribution of the sample image; Based on the second feature, obtain the second predicted probability distribution of the sample image; Based on the third feature, obtain the third predicted probability distribution of the sample image; A second loss function is determined based on the first predicted probability distribution, the second predicted probability distribution, the third predicted probability distribution, and the target class of the sample image; The step of updating the parameters of the student model according to the first loss function to obtain the trained student model includes: The parameters of the student model are updated according to the first loss function and the second loss function to obtain the trained student model.

6. The method according to claim 5, characterized in that, The step of obtaining the third predicted probability distribution of the sample image based on the third feature includes: The third feature is input into the fully connected layer to obtain the third predicted probability distribution; wherein the fully connected layer shares parameters with the fully connected layer of the teacher model.

7. The method according to claim 1, characterized in that, Determining the first loss function based on the first feature, the second feature, and the third feature includes: Based on the first feature and the third feature, determine the first feature loss; The second feature loss is determined based on the second feature and the third feature; Based on the first feature and the second feature, determine the third feature loss; The first loss function is determined based on the first feature loss, the second feature loss, and the third feature loss.

8. The method according to claim 5, characterized in that, The step of determining the second loss function based on the first predicted probability distribution, the second predicted probability distribution, the third predicted probability distribution, and the target class of the sample image includes: A first probability loss is determined based on the first predicted probability distribution, the third predicted probability distribution, and the target class; The second probability loss is determined based on the second predicted probability distribution, the third predicted probability distribution, and the target class; A third probability loss is determined based on the first predicted probability distribution, the second predicted probability distribution, and the target class; The second loss function is determined based on the first probability loss, the second probability loss, and the third probability loss.

9. The method according to claim 1, characterized in that, The first feature extraction network is identical to at least one preceding feature extraction network layer of the student model and shares parameters, while the second feature extraction network is identical in structure to at least one subsequent feature extraction network layer of the teacher model and shares parameters.

10. The method according to claim 1, characterized in that, The student model includes a convolutional neural network.

11. The method according to claim 1, characterized in that, The teacher model includes a multi-head self-attention model or a multilayer perceptron.

12. A model training device, characterized in that, include: An acquisition unit is used to acquire a training sample set, the training sample set including at least one image sample; A student model is used to input the image sample and obtain the first feature of the image sample; The teacher model is used to input the image sample and obtain the second feature of the image sample; the student model and the teacher model are heterogeneous models. A teaching assistant network is used to input the image sample, extract features from the image sample using a first feature extraction network to obtain features with the same dimension as the first feature, transform and map the feature extracted by the first feature extraction network using a transformation network, and extract a third feature from the transformed feature using a second feature extraction network. The teaching assistant network includes the first feature extraction network, the transformation network, and the second feature extraction network. The first feature extraction network is obtained based on the student model, and the second feature extraction network is obtained based on the teacher model. The third feature has the same feature dimension and feature space as the second feature. The determining unit determines a first loss function based on the first feature, the second feature, and the third feature; The parameter update unit updates the parameters of the student model according to the first loss function to obtain the trained student model.

13. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-11.

14. A computer storage medium, characterized in that, Used to store a computer program, the computer program including a method for performing any one of claims 1-11.

15. A computer program product, characterized in that, It includes computer program code that, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 1-11.

Images