Distillation and image recognition method, apparatus, device and program for visual large model

Abstract

The invention provides a distillation method for a visual large model, and the method comprises the steps: obtaining a teacher model which is a visual large model with a Transform structure; performing structure pruning on the teacher model to generate a plurality of auxiliary models; optimizing and training the teacher model and the plurality of auxiliary models in a parameter sharing mode by utilizing an optimization loss function Loss number; selecting the auxiliary model with the highest comprehensive score from the plurality of optimized auxiliary models as a selected auxiliary model; and training the selected auxiliary model and the student model by using an optimized loss function to obtain a final student model. By utilizing the technical scheme of the invention, the model knowledge can be transmitted from the full-scale teacher model to the student model as much as possible, so that the student model achieves the best scale and performance, and the scale of the student model is small enough, so that the student model can be conveniently embedded into actual application equipment.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, apparatus and program for distillation and image recognition of large visual models. Background Art

[0002] Large models based on the Transformer structure have been widely used in various visual tasks. Although many deep learning models have good performance after training, their computational and storage requirements are often very high, which limits their application on resource-constrained devices.

[0003] Optimization methods for deep learning models can improve performance, speed up training, and reduce computing resource consumption. Among the existing model optimization methods, knowledge distillation technology is widely used. Knowledge distillation technology uses the cooperation of teacher models and student models to enable the student model to achieve higher performance while maintaining a smaller scale and lower computing cost.

[0004] Although the existing distillation technology can reduce the complexity of the model to a certain extent, it also has some shortcomings, mainly manifested as follows: 1. Performance loss. During the compression process of the model, some information is inevitably lost, which leads to a decrease in the accuracy of the student model; 2. Large training resource requirements. During the distillation process, in order to achieve performance goals, the student model requires a large amount of data to learn the output distribution of the teacher model, which has high computational overhead; 3. Knowledge transfer is difficult. When the difference between the teacher model and the student model is too large, the knowledge transfer effect may not be good, resulting in the student model's performance failing to meet expectations. Summary of the invention

[0005] In view of the problems of performance loss, high resource requirements and difficulty in knowledge transfer in the model distillation process of the prior art, the present invention proposes an improved distillation method, which can significantly improve the performance of the student model without increasing additional computing costs by optimizing the distillation process of the teacher model output and improving the knowledge transfer method between the teacher model and the student model.

[0006] To this end, one aspect of the present invention provides a distillation method for a large visual model, comprising:

[0007] Obtain a teacher model, where the teacher model is a large visual model with a Transformer structure;

[0008] Performing structural pruning on the teacher model to generate multiple auxiliary models;

[0009] Utilizing the optimized loss function Loss, the teacher model and the plurality of auxiliary models are optimized and trained by parameter sharing;

[0010] The auxiliary model with the highest comprehensive score is selected from the optimized multiple auxiliary models as the selected auxiliary model;

[0011] The selected auxiliary model and the student model are trained using the optimized loss function Loss to obtain a final student model.

[0012] Another aspect of the present invention provides an image recognition method, comprising:

[0013] Obtain an image to be recognized;

[0014] The image to be recognized is input into a recognition model to recognize the target in the image, and the recognition model is a student model obtained by the aforementioned distillation method.

[0015] Another aspect of the present invention provides a distillation device for a large visual model, comprising:

[0016] An acquisition module is used to acquire a teacher model, where the teacher model is a large visual model with a Transformer structure;

[0017] A pruning module, used for performing structural pruning on the teacher model to generate multiple auxiliary models;

[0018] An optimization module, used for optimizing the teacher model and the plurality of auxiliary models by means of parameter sharing using an optimization loss function Loss;

[0019] A selection module is used to select the auxiliary model with the highest comprehensive score from the optimized multiple auxiliary models as the selected auxiliary model;

[0020] The training module is used to train the selected auxiliary model and the student model by optimizing the loss function to obtain the final student model.

[0021] Another aspect of the present invention further provides an image recognition device, comprising:

[0022] An image acquisition module, used for acquiring an image to be identified;

[0023] The image recognition module is used to input the image to be recognized into a recognition model to recognize the target in the image, and the recognition model is a student model obtained by the aforementioned distillation method.

[0024] Another aspect of the present invention further provides an intelligent ultrasound machine device, comprising:

[0025] An image generating unit, used for generating an ultrasound image;

[0026] An image recognition unit is used to recognize the lesion target in the ultrasound image, and the image recognition unit is embedded with a student model obtained by the aforementioned distillation method.

[0027] Another aspect of the present invention provides a computer-readable storage medium on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor implements the various methods described above.

[0028] Another aspect of the present invention further provides a computer program, which, when executed by a processor, enables the processor to implement the various methods described above.

[0029] The technical solution of the present invention has the following beneficial effects:

[0030] 1) A dynamic knowledge transfer mechanism is adopted. By introducing an adaptive knowledge transfer mechanism, the distillation process is made more adaptable to different model structures, thereby improving the applicability and accuracy of the student model.

[0031] 2) Multi-level feature distillation is adopted. In the distillation process, not only the output results are focused on, but also the feature distribution of the intermediate layers is incorporated into the learning objectives to further improve the performance of the student model.

[0032] 3) Improved resource efficiency. By designing an improved distillation target, the computational cost during training is reduced, and the efficiency and applicability of the distillation process are improved.

[0033] By utilizing the technical solution of the present invention, model knowledge can be transferred from a full-scale teacher model to a student model as much as possible, so that the student model achieves the best scale and performance. The scale of the student model is small enough to be easily embedded in actual application equipment, such as an intelligent ultrasound machine, so that the application equipment has lightweight and high-performance image recognition capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objectives and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings.

[0035] Figure 1 A schematic diagram of a system architecture for applying the method of the embodiment of the present disclosure;

[0036] Figure 2 A flow chart of a distillation method for a large visual model provided in an embodiment of the present disclosure;

[0037] Figure 3 is a schematic diagram of a distillation process in an embodiment of the present disclosure;

[0038] Figure 4 A structural block diagram of a distillation device for a large visual model provided by an embodiment of the present disclosure;

[0039] Figure 5 A flowchart of an image recognition method provided by an embodiment of the present disclosure;

[0040] Figure 6 A structural block diagram of an image recognition device provided by an embodiment of the present disclosure;

[0041] Figure 7 This is a structural block diagram of the intelligent ultrasound machine device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In the specification of the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts or a combination thereof disclosed in the specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts or a combination thereof exist or are added.

[0043] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can more easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0044] Figure 1 A schematic diagram of a system architecture for applying the method of an embodiment of the present disclosure is schematically shown.

[0045] like Figure 1 As shown, in the system architecture, terminal devices 101, 102, 103, a network 104 and a server 105 may be included. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0046] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various client applications may be installed on the terminal devices 101, 102, 103, such as dedicated applications with functions such as ultrasound image processing and uploading diagnosis results for display.

[0047] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various special or general electronic devices, including but not limited to ultrasound machines, smart phones, tablet computers, laptop portable computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or they may be implemented as a single software or software module.

[0048] The server 105 may be a server that provides various intelligent services, such as a backend server that provides services for client applications installed on the terminal devices 101, 102, and 103. For example, the server may train and deploy an ultrasound image recognition model to support an intelligent diagnosis system so that visualization results are displayed on the terminal devices 101, 102, and 103.

[0049] The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 105 is software, it may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or as a single software or software module.

[0050] The various methods provided in the embodiments of the present invention may be executed by the server 105 or by the terminal devices 101, 102, 103. Alternatively, the various methods provided in the embodiments of the present invention may be partially executed by the terminal devices 101, 102, 103 and the rest executed by the server 105.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0052] Combine the following Figure 2 An embodiment for implementing the method disclosed herein is described.

[0053] Figure 2 A flowchart of a method provided by an embodiment of the present disclosure. From a program perspective, the execution subject of the process can be a program installed on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities.

[0054] like Figure 2 As shown, the distillation method for the large visual model includes operations S210 to S250.

[0055] S210: Obtain a teacher model, which is a large visual model with a Transformer structure.

[0056] Among the optimization methods of deep learning models, distillation is a widely used method to simplify or compress complex models. The purpose is to reduce computational complexity and resource consumption while maintaining the main functions. The teacher model refers to the model used to guide the training of the student model during the distillation process. It usually has higher performance but higher complexity. The student model refers to a model with a simplified structure output by learning the teacher model. The learning goal is to approach the performance of the teacher model.

[0057] In this step, a large visual model needs to be initialized as a teacher model, which has been pre-trained to achieve the best performance on image recognition tasks. Next, the model is distilled into a student model with a more compact structure.

[0058] S220: Perform structural pruning on the teacher model to generate multiple auxiliary models.

[0059] This step specifically includes: calculating the importance score of each self-attention layer of the teacher model; based on the ranking results of the importance scores, greedily pruning the teacher model to generate multiple auxiliary models.

[0060] Greedy pruning is performed based on the importance score sorting scheme to generate a set of auxiliary models, a total of n, where n can be selected as 10. Through pruning, the key parameters in each network layer of the teacher model are selectively retained, while the less important parameters are pruned to reduce the complexity and computational cost of the model.

[0061] The importance score is calculated as follows: for a SelfAttention layer with x as input, the learnable mask mask is Initialized to 1,

[0062]

[0063] The importance score is defined as the expected sensitivity of the model to the masked variable, i.e.

[0064]

[0065] Among them, Loss p is the loss function used in the pruning process, D χ is the distribution of training data, and E represents the expected value. The importance score of each self-attention layer reflects its contribution to the model performance, and the higher the score, the greater the impact on the model.

[0066] S230: Utilize the optimized loss function Loss to optimize the training of the teacher model and multiple auxiliary models through parameter sharing.

[0067] S240: Select the auxiliary model with the highest comprehensive score from among the optimized multiple auxiliary models as the selected auxiliary model.

[0068] After a set of auxiliary models are generated, it is necessary to select the auxiliary model with the best performance and the smallest scale, that is, the one that balances the model performance and scale.

[0069] The optimization loss function includes cross entropy loss, logarithmic loss and feature-based loss, which are calculated as follows:

[0070]

[0071] The first term represents the cross entropy loss of the teacher model and the auxiliary model’s predicted labels; the second term represents the KL divergence of the teacher model and the auxiliary model’s output logits, and γ represents the temperature; the third term represents the mean square error of the last hidden layer of the teacher model and the auxiliary model.

[0072] After the above steps, the performance of each auxiliary model is maximized, and then the best one is selected from these auxiliary models. In order to balance the model scale and model performance at the same time, the following formula is used to calculate the comprehensive score of each auxiliary model:

[0073]

[0074] Among them, Acc represents the corresponding model performance. If it is a segmentation or detection task, it can also be replaced with the corresponding most concerned indicator for calculation. Scale represents the size of the corresponding model. Intuitively, a higher performance and smaller scale auxiliary model will get a higher comprehensive score.

[0075] According to the comprehensive score calculation results, the auxiliary model with the highest comprehensive score Score2 is selected from all auxiliary models as the selected auxiliary model.

[0076] S250: Train the selected auxiliary model and the student model using the optimized loss function to obtain the final student model.

[0077] Additional distillation is performed between the selected auxiliary model and the student model, following the same training objective as the optimization training in step S230, to obtain the final student model.

[0078] According to the distillation method provided in the above embodiment, Figure 3 A schematic diagram of the complete distillation process is given.

[0079] like Figure 3As shown in the figure, after structural pruning, the teacher model generates multiple auxiliary models. After optimizing these auxiliary models, the one with the highest comprehensive score is selected as the selected auxiliary model, and then distillation is performed between the selected auxiliary model and the student model to obtain the final student model.

[0080] The above content describes in detail the distillation method for the visual large model. Correspondingly, the specification of the present disclosure also provides a distillation device 400 for the visual large model. Figure 4 Provide explanation.

[0081] Figure 4 A schematic diagram of a distillation apparatus 400 provided in accordance with an embodiment of the present disclosure.

[0082] like Figure 4 As shown, the distillation device 400 includes an acquisition module 410, a pruning module 420, an optimization module 430, a selection module 440 and a training module 450. The distillation device 400 can be implemented by software, hardware or a combination of both. The distillation device 400 can execute the distillation method for the visual large model described above.

[0083] The acquisition module 410 is used to acquire a teacher model, which is a large visual model with a Transformer structure.

[0084] The pruning module 420 is used to perform structural pruning on the teacher model to generate multiple auxiliary models.

[0085] The optimization module 430 is used to optimize the training teacher model and multiple auxiliary models by using the optimization loss function Loss through parameter sharing.

[0086] The selection module 440 is used to select the auxiliary model with the highest comprehensive score from multiple auxiliary models as the selected auxiliary model.

[0087] The training module 450 is used to train the selected auxiliary model and the student model using the optimized loss function to obtain the final student model.

[0088] The distillation method and distillation device for large visual models of the disclosed embodiments adopt a dynamic knowledge transfer mechanism. By introducing an adaptive knowledge transfer mechanism, the distillation process is more adaptable to different model structures, thereby improving the applicability and accuracy of the student model. Multi-level feature distillation is adopted. In the distillation process, not only the output results are paid attention to, but also the feature distribution of the middle layer is incorporated into the learning objectives, further improving the performance of the student model. Resource efficiency is improved. By designing improved distillation targets, the computational cost in the training process is reduced. Good training targets can be achieved without using a large amount of training data, thereby improving the efficiency and applicability of the distillation process.

[0089] The specification of the present disclosure also provides an application method of the student model obtained according to the above distillation method, that is, an image recognition method, which includes operations S510 to S520, such as Figure 5 shown.

[0090] S510: Acquire an image to be recognized.

[0091] S520: Input the image to be recognized into a recognition model to recognize the target in the image. The recognition model is a student model obtained by the aforementioned distillation method.

[0092] Correspondingly, the specification of the present disclosure also provides an image recognition device 600 . Figure 6 A schematic diagram of an image recognition device 600 provided in an embodiment of the present disclosure.

[0093] like Figure 6 As shown, the image recognition device 600 includes an image acquisition module 610 and an image recognition module 620. The image recognition device 600 can be implemented by software, hardware or a combination of both.

[0094] The image acquisition module 610 is used to acquire an image to be identified.

[0095] The image recognition module 620 is used to input the image to be recognized into a recognition model to recognize the target in the image. The recognition model is a student model obtained by the aforementioned distillation method.

[0096] The image to be identified may be a medical image, a monitoring image, etc., and the corresponding target to be identified may be a lesion, an object, etc. in the image. By using the technical solution of the present invention, the model knowledge can be transferred from the full-scale teacher model to the student model as much as possible, so that the student model is optimized in scale and performance. The scale of the student model is small enough to be easily embedded in the actual application device, so that the application device has a lightweight and high-performance image recognition capability.

[0097] The specification of the present disclosure also provides an application scenario of the student model obtained according to the above-mentioned distillation method, namely, an intelligent ultrasound machine device 700, which includes an image generation unit 710 for generating ultrasound images, and an image recognition unit 720 for identifying lesion targets in ultrasound images, and this image recognition unit is embedded with the student model obtained by the above-mentioned distillation method.

[0098] The specification of the present disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method of the embodiment of the present invention.

[0099] Another aspect of the present disclosure also provides a computer program, which, when executed by a processor, enables the processor to implement the various methods described above.

[0100] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, computer storage medium and computer program embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0102] The apparatus, device, computer storage medium and computer program provided in the embodiments of the present disclosure correspond to the method. Therefore, the apparatus, device, computer storage medium and computer program also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, they will not be repeated here.

[0103] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or programmable hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0104] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A distillation method for a large visual model, characterized in that: include: Obtain a teacher model, where the teacher model is a large visual model with a Transformer structure; Performing structural pruning on the teacher model to generate multiple auxiliary models; Utilizing the optimized loss function Loss, the teacher model and the plurality of auxiliary models are optimized and trained by parameter sharing; The auxiliary model with the highest comprehensive score is selected from the optimized multiple auxiliary models as the selected auxiliary model; The selected auxiliary model and the student model are trained using the optimized loss function Loss to obtain a final student model.

2. The distillation method according to claim 1, characterized in that The step of performing structural pruning on the teacher model to generate multiple auxiliary models includes: Calculate the importance score of each self-attention layer of the teacher model; Based on the ranking results of the importance scores, greedy pruning is performed on the teacher model to generate multiple auxiliary models.

3. The distillation method according to claim 2, characterized in that The importance score is calculated as follows: For the self-attention layer with x as input, the learnable mask Initialized to 1, The importance scores are: Among them, Loss p is the loss function used in the pruning process, D x is the distribution of training data, and E represents the expected value.

4. The distillation method according to claim 1, characterized in that The optimization loss function Loss is calculated as follows: Among them, the first item is the cross entropy loss of the predicted labels of the teacher model and the auxiliary model; the second item is the KL divergence of the logits output by the teacher model and the auxiliary model, and γ represents the temperature; the third item is the mean square error of the last hidden layer of the teacher model and the auxiliary model.

5. The distillation method according to claim 1, characterized in that The comprehensive score of the auxiliary model is: Among them, Acc represents the corresponding model performance, and Scale represents the corresponding model scale.

6. An image recognition method, characterized in that: include: Obtain an image to be recognized; The image to be recognized is input into a recognition model to recognize the target in the image, and the recognition model is a student model obtained by the distillation method according to any one of claims 1 to 5.

7. An intelligent ultrasound machine, characterized in that: include: An image generating unit, used for generating an ultrasound image; An image recognition unit is used to recognize the lesion target in the ultrasound image, and the image recognition unit is embedded with a student model obtained by the distillation method according to any one of claims 1 to 5.

8. A distillation device for a large visual model, characterized in that: include: An acquisition module is used to acquire a teacher model, where the teacher model is a large visual model with a Transformer structure; A pruning module, used for performing structural pruning on the teacher model to generate multiple auxiliary models; An optimization module, used for optimizing the teacher model and the plurality of auxiliary models by means of parameter sharing using an optimization loss function Loss; A selection module is used to select the auxiliary model with the highest comprehensive score from the optimized multiple auxiliary models as the selected auxiliary model; The training module is used to train the selected auxiliary model and the student model by optimizing the loss function to obtain the final student model.

9. An image recognition device, characterized in that: include: An image acquisition module, used for acquiring an image to be identified; An image recognition module is used to input the image to be recognized into a recognition model to recognize a target in the image, wherein the recognition model is a student model obtained by the distillation method according to any one of claims 1 to 5.

10. A computer program, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 6.

Images