Distillation method and device for visual large model and image recognition method, device, apparatus and program

Abstract

The application provides a distillation method for a visual large model, comprising the following steps: obtaining a teacher model, wherein the teacher model is a visual large model with a Transformer structure; performing structural pruning on the teacher model to generate multiple auxiliary models; optimizing and training the teacher model and the multiple auxiliary models by parameter sharing through an optimization loss function Loss; selecting an auxiliary model with the highest comprehensive score from the multiple optimized auxiliary models as a selected auxiliary model; training the selected auxiliary model and a student model by using the optimization loss function to obtain a final student model. According to the technical scheme of the application, model knowledge can be transmitted from a full-scale teacher model to a student model as much as possible, so that the student model reaches the best scale and performance, and the scale of the student model is small enough to be conveniently embedded into an actual application device.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of image recognition, and in particular to a distillation method for a visual large model and an image recognition method, device, apparatus and program. BACKGROUND

[0002] Large models based on the Transformer structure have been widely applied to various visual tasks. Although many deep learning models have good performance after training, their computational complexity and storage requirements are often very high, which limits their application on resource-constrained devices.

[0003] Optimization methods for deep learning models can improve performance, speed up training, and reduce computational resource consumption. In existing model optimization methods, knowledge distillation technology is widely used. Knowledge distillation technology uses a teacher model and a student model to enable the student model to maintain a small size and low computational cost while achieving high performance.

[0004] Although existing distillation technology can reduce model complexity to some extent, it also has some shortcomings, mainly manifested as: 1. Performance loss. During the compression process, some information is inevitably lost, resulting in a decrease in the accuracy of the student model; 2. High demand for training resources. In the distillation process, the student model needs a large amount of data to learn the output distribution of the teacher model, resulting in high computational overhead; 3. Difficulty in knowledge transfer. When the difference between the teacher model and the student model is too large, the knowledge transfer effect may not be good, resulting in the performance of the student model not meeting expectations. SUMMARY

[0005] To address the performance loss, high resource demand, and difficulty in knowledge transfer in the model distillation process of existing technology, the present application proposes an improved distillation method that optimizes the distillation process of the teacher model output and improves the knowledge transfer method between the teacher model and the student model, thereby significantly improving the performance of the student model without increasing additional computational costs.

[0006] To this end, one aspect of the present application provides a distillation method for a visual large model, comprising:

[0007] obtaining a teacher model, the teacher model being a visual large model with a Transformer structure;

[0008] performing structural pruning on the teacher model to generate a plurality of auxiliary models;

[0009] optimizing the training of the teacher model and the plurality of auxiliary models through parameter sharing using an optimization loss function Loss;

[0010] select an auxiliary model with the highest comprehensive score from the multiple auxiliary models after optimization as a selected auxiliary model;

[0011] train the selected auxiliary model and a student model by using the optimization loss function Loss, and obtain a final student model.

[0012] In another aspect of the present application, an image recognition method is provided, comprising:

[0013] obtaining an image to be recognized;

[0014] inputting the image to be recognized into a recognition model to recognize a target in the image, wherein the recognition model is a student model obtained by the aforementioned distillation method.

[0015] In another aspect of the present application, a distillation device for a visual large model is provided, comprising:

[0016] an obtaining module configured to obtain a teacher model, wherein the teacher model is a visual large model with a Transformer structure;

[0017] a pruning module configured to perform structural pruning on the teacher model to generate multiple auxiliary models;

[0018] an optimization module configured to optimize and train the teacher model and the multiple auxiliary models by parameter sharing through an optimization loss function Loss;

[0019] a selection module configured to select an auxiliary model with the highest comprehensive score from the multiple auxiliary models after optimization as a selected auxiliary model;

[0020] a training module configured to train the selected auxiliary model and a student model by using the optimization loss function, and obtain a final student model.

[0021] In another aspect of the present application, an image recognition device is also provided, comprising:

[0022] an image obtaining module configured to obtain an image to be recognized;

[0023] an image recognition module configured to input the image to be recognized into a recognition model to recognize a target in the image, wherein the recognition model is a student model obtained by the aforementioned distillation method.

[0024] In another aspect of the present application, an intelligent ultrasound machine device is also provided, comprising:

[0025] an image generating unit configured to generate an ultrasound image;

[0026] an image recognition unit configured to recognize a lesion target in the ultrasound image, wherein the image recognition unit is embedded with a student model obtained by the aforementioned distillation method.

[0027] Another aspect of the present application also provides a computer readable storage medium having computer readable instructions stored thereon, which, when executed by a processor, cause the processor to implement the various methods described above.

[0028] Another aspect of the present application also provides a computer program, which, when executed by a processor, causes the processor to implement the various methods described above.

[0029] The technical solution of the present application has the following beneficial effects:

[0030] 1) A dynamic knowledge transfer mechanism is adopted, and by introducing an adaptive knowledge transfer mechanism, the distillation process is more suitable for different model structures, and the applicability and accuracy of the student model are improved.

[0031] 2) Multi-level feature distillation is adopted, and not only the output result is focused on in the distillation process, but also the feature distribution of the intermediate layer is included in the learning goal, further improving the performance of the student model.

[0032] 3) Resource efficiency is improved, and by designing an improved distillation target, the computational cost in the training process is reduced, and the efficiency and scope of application of the distillation process are improved.

[0033] By using the technical solution of the present application, model knowledge can be transferred from a full-scale teacher model to a student model as much as possible, so that the student model reaches the best in size and performance. The size of the student model is small enough to be easily embedded in practical application devices, such as intelligent ultrasonic device, so that the application device has light and high-performance image recognition capability. BRIEF DESCRIPTION OF DRAWINGS

[0034] Other features, objects and advantages of the present application will become more apparent from the following detailed description of the non-limiting embodiments, combined with the attached drawings.

[0035] Figure 1 A schematic diagram of a system architecture for applying the method of the embodiment of the present disclosure is shown in the figure.

[0036] Figure 2 A flowchart of the distillation method for a visual large model provided by the embodiment of the present disclosure is shown in the figure.

[0037] Figure 3 A schematic diagram of the distillation process in the embodiment of the present disclosure is shown in the figure.

[0038] Figure 4 A structural block diagram of the distillation device for a visual large model provided by the embodiment of the present disclosure is shown in the figure.

[0039] Figure 5 A flowchart of the image recognition method provided by the embodiment of the present disclosure is shown in the figure.

[0040] Figure 6 A structural block diagram of an image recognition device provided by an embodiment of the present disclosure is shown.

[0041] Figure 7 A structural block diagram of an intelligent ultrasound machine device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0042] In the specification of the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there are features, numbers, steps, actions, components or parts that are disclosed in the specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, components, parts or a combination thereof exist or are added.

[0043] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so as to make technicians in the art more easily carry them out. In addition, parts irrelevant to the description of the exemplary embodiments are omitted in the drawings for the sake of clarity. It is noted that the embodiments in the present disclosure and features in the embodiments can be combined with each other without conflict.

[0044] Figure 1 A schematic diagram of a system architecture to which the method of the embodiment of the present disclosure is applied is shown.

[0045] As shown in Figure 1 , in the system architecture, terminal devices 101, 102, 103, a network 104 and a server 105 can be included. The network 104 is a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0046] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various client applications can be installed on the terminal devices 101, 102, 103. For example, a special application program with functions such as ultrasound image processing and display of uploaded diagnosis results.

[0047] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various special or general electronic devices, including but not limited to ultrasound machines, smart phones, tablet computers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or as a single software or software module.

[0048] The server 105 can be a server providing various intelligent services, for example, a backend server providing services for a client application installed on the terminal device 101, 102, 103. For example, the server can train and deploy an ultrasound image recognition model, implement support for an intelligent diagnosis system, so as to display a visualization result on the terminal device 101, 102, 103.

[0049] The server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or as a single software or software module.

[0050] The various methods provided by the embodiments of the present application can be executed by the server 105 or the terminal device 101, 102, 103. Alternatively, the various methods of the embodiments of the present application can be partially executed by the terminal device 101, 102, 103 and partially executed by the server 105.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in

[0052] The following describes an embodiment of the method for implementing the method of the present disclosure. Figure 2

[0053] Figure 2 The flowchart of the method provided by the embodiment of the present disclosure. From the program point of view, the execution subject of the flowchart can be a program loaded on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capability.

[0054] As Figure 2 shown, the distillation method for the visual large model includes operations S210-S250.

[0055] S210: Obtain a teacher model, which is a visual large model with a Transformer structure.

[0056] ​In the optimization method of deep learning model, distillation is a widely used method to simplify or compress complex models, aiming to reduce computational complexity and resource consumption while maintaining main functions. The teacher model refers to the model used to guide the training of the student model during the distillation process, which usually has high performance but high complexity. The student model refers to the model with simplified structure that learns from the teacher model output, and the learning goal is to approach the performance of the teacher model.

[0057] In this step, a visual large model is initialized as a teacher model, which is pre-trained to achieve the best performance on image recognition tasks. Then the model is distilled into a more compact student model.

[0058] S220: Structure pruning of the teacher model to generate multiple auxiliary models.

[0059] This step specifically includes: calculating the importance score of each self-attention layer of the teacher model; based on the ranking result of the importance score, the teacher model is pruned greedily to generate multiple auxiliary models.

[0060] Based on the importance score ranking scheme, a set of auxiliary models (auxiliary) is generated, a total of n, n can be selected as 10. By pruning, the key parameters in each network layer of the teacher model are selectively retained, while the less important parameters are pruned to reduce the complexity and computational cost of the model.

[0061] The importance score is calculated as follows: for a SelfAttention layer with x as input, the learnable mask is initialized to 1,

[0062]

[0063] The importance score is defined as the expected sensitivity of the model to the mask variable, that is,

[0064]

[0065] where Loss p is the loss function used in the pruning process, D χ is the distribution of training data, and E represents the expected value. The importance score of each self-attention layer reflects its contribution to the model performance, and the higher the score, the greater the impact on the model.

[0066] S230: Optimize the teacher model and the plurality of auxiliary models by parameter sharing using the optimization loss function Loss.

[0067] S240: Select the auxiliary model with the highest comprehensive score from the plurality of optimized auxiliary models as the selected auxiliary model.

[0068] When a set of auxiliary models is generated, the auxiliary model with the best performance and the smallest size, i.e., the auxiliary model balancing the model performance and the size, needs to be selected.

[0069] The optimization loss function includes three terms, i.e., cross-entropy loss, log loss, and feature-based loss, and is calculated as follows:

[0070]

[0071] The first term represents the cross-entropy loss of the predicted labels of the teacher model and the auxiliary model; the second term represents the KL divergence of the logits output by the teacher model and the auxiliary model, and γ represents the temperature; and the third term represents the mean square error of the last hidden layer of the teacher model and the auxiliary model.

[0072] After the above steps, the performance of each auxiliary model is maximized, and then the optimal one is selected from the auxiliary models. In order to balance the model size and the model performance at the same time, the following formula is used to calculate the comprehensive score of each auxiliary model:

[0073]

[0074] Wherein, Acc represents the performance of the corresponding model, and if it is a segmentation or detection task, it can be replaced by the corresponding most concerned index for calculation; and Scale represents the size of the corresponding model. Intuitively, the auxiliary model with higher performance and smaller size will get a higher comprehensive score.

[0075] According to the comprehensive score calculation result, the auxiliary model with the highest comprehensive score Score2 is selected from all auxiliary models as the selected auxiliary model.

[0076] S250: Train the selected auxiliary model and the student model using the optimization loss function to obtain the final student model.

[0077] Additional distillation is performed between the selected auxiliary model and the student model, and the same training objective as in the optimization training in step S230 is followed to obtain the final student model.

[0078] According to the distillation method provided in the above embodiments, Figure 3 A schematic diagram of a complete distillation process is given.

[0079] As Figure 3As shown, the teacher model is pruned to generate multiple auxiliary models, the auxiliary models are optimized, the auxiliary model with the highest comprehensive score is selected as the selected auxiliary model, and then the selected auxiliary model and the student model are distilled to obtain the final student model.

[0080] The above describes the distillation method for the visual large model. Correspondingly, the present disclosure also provides a distillation device 400 for the visual large model, which will be described below with reference to Figure 4 .

[0081] Figure 4 The distillation device 400 provided by the embodiments of the present disclosure is shown in the schematic diagram.

[0082] As Figure 4 shown, the distillation device 400 includes an acquisition module 410, a pruning module 420, an optimization module 430, a selection module 440, and a training module 450. The distillation device 400 can be implemented by software, hardware, or a combination of both. The distillation device 400 can perform the distillation method for the visual large model described above.

[0083] The acquisition module 410 is configured to acquire a teacher model, which is a visual large model with a Transformer structure.

[0084] The pruning module 420 is configured to prune the structure of the teacher model to generate multiple auxiliary models.

[0085] The optimization module 430 is configured to optimize and train the teacher model and the multiple auxiliary models by parameter sharing using an optimization loss function Loss.

[0086] The selection module 440 is configured to select the auxiliary model with the highest comprehensive score from the multiple auxiliary models as a selected auxiliary model.

[0087] The training module 450 is configured to train the selected auxiliary model and a student model using the optimization loss function to obtain a final student model.

[0088] The distillation method and distillation device for a visual large model provided by the embodiments of the present disclosure adopt a dynamic knowledge migration mechanism, and by introducing an adaptive knowledge transfer mechanism, the distillation process is more suitable for different model structures, and the applicability and precision of the student model are improved. Multi-level feature distillation is adopted, and in the distillation process, not only the output result is focused on, but also the feature distribution of the intermediate layer is included in the learning goal, so that the performance of the student model is further improved. The resource efficiency is improved, and by designing an improved distillation target, the calculation cost in the training process is reduced, and good training targets can be achieved without using a large amount of training data, thereby improving the efficiency and application range of the distillation process.

[0089] The specification of the present disclosure also provides an application method of the student model obtained by the above-mentioned distillation method, that is, an image recognition method, which comprises operations S510-S520, as shown in Figure 5 .

[0090] S510: obtaining an image to be recognized.

[0091] S520: inputting the image to be recognized into a recognition model to recognize a target in the image, wherein the recognition model is a student model obtained by the above-mentioned distillation method.

[0092] Correspondingly, the specification of the present disclosure also provides an image recognition device 600. Figure 6 The schematic diagram of the image recognition device 600 provided by the embodiments of the present disclosure is shown in

[0093] As shown in Figure 6 , the image recognition device 600 comprises an image acquisition module 610 and an image recognition module 620. The image recognition device 600 can be realized by software, hardware or a combination of both.

[0094] The image acquisition module 610 is configured to obtain an image to be recognized.

[0095] The image recognition module 620 is configured to input the image to be recognized into a recognition model to recognize a target in the image, wherein the recognition model is a student model obtained by the above-mentioned distillation method.

[0096] The image to be recognized can be a medical image, a monitoring image, etc., and the target to be recognized can be a lesion, an object, etc. in the image. By using the technical solution of the present disclosure, model knowledge can be transferred from a full-scale teacher model to a student model as much as possible, so that the student model reaches the best scale and performance, and the scale of the student model is small enough to be easily embedded into an actual application device, so that the application device has light and high-performance image recognition capability.

[0097] The specification of the present disclosure also provides an application scenario of the student model obtained by the distillation method, i.e., an intelligent ultrasound machine device 700, which comprises an image generation unit 710 configured to generate an ultrasound image, and an image recognition unit 720 configured to recognize a lesion target in the ultrasound image, wherein the image recognition unit is embedded with the student model obtained by the distillation method.

[0098] The specification of the present disclosure also provides a computer readable storage medium, which can be a computer readable storage medium included in the electronic device or the computer system in the above-mentioned embodiments, or can exist independently and not be assembled into the device. The computer readable storage medium stores one or more programs, which are executed by one or more processors to implement the method of the embodiments of the present disclosure.

[0099] Another aspect of the present disclosure also provides a computer program, which, when executed by a processor, causes the processor to implement various methods as described above.

[0100] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps in a claim can be performed in an order other than the order in which the acts or steps are recited, and still accomplish the desired result. In addition, the process depicted in the accompanying drawings does not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0101] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the device, equipment, computer storage medium and computer program embodiments are basically similar to the method embodiments, and thus are described more simply, and the relevant parts can be referred to the part of the method embodiment.

[0102] The device, equipment, computer storage medium and computer program provided by the embodiments of the present disclosure correspond to the method, and thus have similar beneficial technical effects. Since the beneficial technical effects of the method have been described in detail above, they will not be described here.

[0103] The units or modules described in the embodiments of the present disclosure can be implemented in the form of software or programmable hardware. The described units or modules can also be arranged in a processor. In some cases, the name of the unit or module does not constitute a limitation on the unit or module itself.

[0104] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the inventive scope involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present disclosure (but not limited to) without departing from the inventive concept.

Claims

1. A distillation method for a visual large model, characterized in that, The method comprises the following steps: obtaining a teacher model, the teacher model being a visual large model with a Transformer structure, the teacher model reaching optimal performance on an image recognition task after pre-training; performing structural pruning on the teacher model to generate a plurality of auxiliary models; optimizing and training the teacher model and the plurality of auxiliary models by parameter sharing through an optimization loss function Loss; selecting an auxiliary model with the highest comprehensive score from the plurality of optimized auxiliary models, the highest comprehensive score representing the best performance and the smallest size; training the selected auxiliary model and a student model through the optimization loss function to obtain a final student model, the student model being deployed on a terminal device with limited resources for image recognition; wherein the optimization loss function Loss is calculated as follows: wherein the first term is a cross-entropy loss of predicted labels of the teacher model and the auxiliary models; the second term is a KL divergence of logits output by the teacher model and the auxiliary models, and γ represents a temperature; the third term is a mean square error of the last hidden layer of the teacher model and the auxiliary models; and M represents the number of the plurality of auxiliary models.

2. The distillation method according to claim 1, characterized by The comprehensive score of the auxiliary model is: wherein Acc represents the performance of the corresponding model, and Scale represents the size of the corresponding model.

3. An image recognition method characterized by, The method comprises the following steps: obtaining an image to be recognized; inputting the image to be recognized into a recognition model to recognize a target in the image, the recognition model being a student model obtained through the distillation method of claim 1 or 2.

4. An intelligent ultrasound machine device, characterized by, The method comprises the following steps: an image generation unit for generating an ultrasound image; an image recognition unit for recognizing a lesion target in the ultrasound image, the image recognition unit being embedded with a student model obtained through the distillation method of claim 1 or 2.

5. A distillation apparatus for a visual large model, comprising: The method comprises the following steps: an obtaining module for obtaining a teacher model, the teacher model being a visual large model with a Transformer structure, the teacher model reaching optimal performance on an image recognition task after pre-training; a pruning module for performing structural pruning on the teacher model to generate a plurality of auxiliary models; an optimization module for optimizing and training the teacher model and the plurality of auxiliary models by parameter sharing through an optimization loss function Loss; a selecting module for selecting an auxiliary model with the highest comprehensive score from the plurality of optimized auxiliary models, the highest comprehensive score representing the best performance and the smallest size; a training module for training the selected auxiliary model and a student model through the optimization loss function to obtain a final student model, the student model being deployed on a terminal device with limited resources for image recognition; wherein the optimization loss function Loss is calculated as follows: wherein the first term is a cross-entropy loss of predicted labels of the teacher model and the auxiliary models; the second term is a KL divergence of logits output by the teacher model and the auxiliary models, and γ represents a temperature; the third term is a mean square error of the last hidden layer of the teacher model and the auxiliary models; and M represents the number of the plurality of auxiliary models.

6. An image recognition apparatus characterized by comprising: The method comprises the following steps: an image obtaining module for obtaining an image to be recognized; An image recognition module is configured to input the image to be recognized into a recognition model to recognize a target in the image, wherein the recognition model is a student model obtained by the distillation method of claim 1 or 2.

7. A computer program product, which, when executed by a processor, causes the processor to implement the method of any one of claims 1-3.

Images