Knowledge distillation method and device for instance segmentation, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a knowledge distillation method and device for instance segmentation, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps that U first prediction examples output by an example segmentation teacher model for a sample image are acquired, V second prediction examples output by an example segmentation student model for the sample image are acquired, and U and V are not equal; aligning the U first prediction instances and the V second prediction instances to L real instances of the sample image to obtain M first prediction instances aligned with each real instance and N second prediction instances aligned with each real instance; calculating instance distillation loss based on the difference between the M first prediction instances and the N second prediction instances; calculating a result distillation loss based on L instance distillation losses corresponding to the L real instances; model parameters of the instance segmentation student model are adjusted based on the resulting distillation loss. According to the application, the applicability of knowledge distillation can be improved.

Description

Technical Field

[0001] This application relates to instance segmentation technology in the field of artificial intelligence, and more particularly to a knowledge distillation method, apparatus, electronic device, computer-readable storage medium, and computer program product for instance segmentation. Background Technology

[0002] Knowledge distillation, also known as model distillation or distillation in general, is a model compression and acceleration technique used to transfer knowledge from a teacher model to a student model. In instance segmentation, to achieve knowledge distillation, the outcome distillation loss can be calculated based on the difference between the predicted instances of the teacher model and the predicted instances of the student model, thus enabling knowledge distillation based on this outcome distillation loss. However, when there is a difference in the number of predicted instances between the teacher and student models, the feasibility of knowledge distillation based on the outcome distillation loss is affected, thus impacting the applicability of knowledge distillation. Summary of the Invention

[0003] This application provides a knowledge distillation method, apparatus, electronic device, computer-readable storage medium, and computer program product for instance segmentation, which can improve the applicability of knowledge distillation.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a knowledge distillation method for instance segmentation, the method comprising: Obtain U first predicted instances output by the instance segmentation teacher model for the sample image, and obtain V second predicted instances output by the instance segmentation student model for the sample image, where U and V are unequal positive integers; Align the U first predicted instances and the V second predicted instances to the L real instances of the sample image to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance, where L, M, and N are positive integers; Instance distillation loss is calculated based on the differences between the M first prediction instances and the N second prediction instances; The distillation loss is calculated based on the distillation loss of the L instances corresponding to the L real instances; The model parameters of the instance segmentation student model are adjusted based on the distillation loss results.

[0005] This application provides a knowledge distillation apparatus for instance segmentation, the knowledge distillation apparatus comprising: The output acquisition module is used to acquire U first predicted instances output by the instance segmentation teacher model for the sample image, and to acquire V second predicted instances output by the instance segmentation student model for the sample image, where U and V are unequal positive integers. The instance alignment module is used to align the U first predicted instances and the V second predicted instances to L real instances of the sample image, so as to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance, where L, M and N are positive integers. The loss acquisition module is used to calculate the instance distillation loss based on the difference between the M first prediction instances and the N second prediction instances; The loss acquisition module is also used to calculate the result distillation loss based on the distillation losses of the L instances corresponding to the L real instances; The parameter adjustment module is used to adjust the model parameters of the instance segmentation student model based on the distillation loss of the results.

[0006] In some embodiments, the instance alignment module is further configured to perform the following processing for each of the L real instances: selecting K first predicted instances initially aligned with the real instance from the U first predicted instances, and selecting K second predicted instances initially aligned with the real instance from the V second predicted instances, where K is a positive integer; aligning and deduplicating the first predicted instances based on the K first predicted instances initially aligned with each of the L real instances to obtain the M first predicted instances aligned with each of the real instances; aligning and deduplicating the second predicted instances based on the K second predicted instances initially aligned with each of the L real instances to obtain the N second predicted instances aligned with each of the real instances.

[0007] In some embodiments, the instance alignment module is further configured to: select, from the U first predicted instances, a first predicted instance whose center point is within the real detection box of the real instance; calculate the intersection-union ratio (IU) of the predicted detection box and the real detection box for each selected first predicted instance; integrate the IU IU and the classification score of the first predicted instance to obtain an alignment index value; and, based on the alignment index value, select, from the selected plurality of first predicted instances, the K first predicted instances that are initially aligned with the real instance.

[0008] In some embodiments, the instance alignment module is further configured to: determine first predicted instances initially aligned with a plurality of real instances based on the K first predicted instances initially aligned to each of the L real instances; calculate the complete intersection-union ratio (CIU) of the first predicted instances initially aligned with the plurality of real instances; determine the real instance aligned to the first predicted instance from the plurality of real instances initially aligned to the first predicted instance based on the CIU; and determine the M first predicted instances aligned to each of the real instances based on the real instances aligned to the first predicted instance.

[0009] In some embodiments, the loss acquisition module is further configured to perform the following processing on each of the M first prediction instances and each of the N second prediction instances: calculating a classification distillation loss based on the difference between the predicted category of the first prediction instance and the predicted category of the second prediction instance; calculating a detection distillation loss based on the difference between the predicted detection box of the first prediction instance and the predicted detection box of the second prediction instance; calculating a segmentation distillation loss based on the difference between the predicted segmentation mask of the first prediction instance and the predicted segmentation mask of the second prediction instance; and integrating the M*N classification distillation losses, the M*N detection distillation losses, and the M*N segmentation distillation losses to obtain the instance distillation loss.

[0010] In some embodiments, the parameter adjustment module is further configured to: obtain first features at I scales output by the instance segmentation student model for the sample image; obtain second features at J scales output by the instance segmentation teacher model for the sample image, where I and J are unequal positive integers; align the second features at J scales to the first features at I scales to obtain third features at I scales; calculate feature distillation loss based on the difference between the first features at I scales and the third features at I scales; and adjust the model parameters of the instance segmentation student model based on the feature distillation loss and the result distillation loss.

[0011] In some embodiments, the parameter adjustment module is further configured to: upsample the second feature starting from the lowest scale among the J scales to obtain the p-th upsampled feature, where p is a variable representing the number of upsampling times; integrate the second feature with the same scale as the p-th upsampled feature among the second features of the J scales and the p-th upsampled feature into a fourth feature; when upsampling to the highest scale among the I scales, obtain the fourth feature of the I scale; downsample starting from the fused feature of the second feature at least one scale above the highest scale of the I scale among the J scales to obtain the q-th downsampled feature, where q is a variable representing the number of downsampling times; integrate the fourth feature with the same scale as the q-th downsampled feature among the fourth features of the I scales and the q-th downsampled feature into a third feature; when downsampling to the lowest scale among the I scales, obtain the third feature of the I scale.

[0012] In some embodiments, the parameter adjustment module is further configured to: determine multiple first features at multiple I scales and multiple third features at multiple I scales corresponding to multiple sample images based on the first features at multiple I scales and the third features at multiple I scales corresponding to the sample images; calculate a first feature similarity among the multiple first features at multiple I scales corresponding to multiple sample images; calculate a second feature similarity among the multiple third features at multiple I scales corresponding to multiple sample images; calculate a feature relationship distillation loss based on the difference between the first feature similarity and the first feature similarity; and adjust the model parameters of the instance segmentation student model based on the feature relationship distillation loss, the feature distillation loss, and the result distillation loss.

[0013] This application provides an electronic device for knowledge distillation, the electronic device comprising: Memory is used to store executable instructions or computer programs. When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the knowledge distillation method for instance segmentation provided in the embodiments of this application.

[0014] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the knowledge distillation method for instance segmentation provided in this application.

[0015] This application provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the knowledge distillation method for instance segmentation provided in this application.

[0016] The embodiments of this application have at least the following beneficial effects: After obtaining U first predicted instances of the instance segmentation teacher model for the sample image and V second predicted instances of the instance segmentation student model for the sample image, by aligning the U first predicted instances and V second predicted instances to L real instances of the sample image, and using real instances as intermediate information to align the first predicted instances and second predicted instances, M first predicted instances aligned with N second predicted instances are obtained; thus, when the number of predicted instances of the instance segmentation teacher model and the instance segmentation student model are different, the alignment of the predicted instances of the instance segmentation teacher model and the instance segmentation student model is achieved, enabling the calculation of the result distillation loss based on the difference between the N second predicted instances and the M first predicted instances to achieve knowledge distillation; therefore, the feasibility of knowledge distillation based on the result distillation loss is improved, and the applicability of knowledge distillation is enhanced. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the architecture of the instance segmentation system provided in the embodiments of this application; Figure 2 This is one of the embodiments provided in this application. Figure 1 A schematic diagram of the server structure in the diagram; Figure 3 This is a flowchart illustrating the knowledge distillation method for instance segmentation provided in the embodiments of this application. Figure 1 ; Figure 4 This is a flowchart illustrating the knowledge distillation method for instance segmentation provided in the embodiments of this application. Figure 2 ; Figure 5 This is a flowchart illustrating the knowledge distillation method for instance segmentation provided in the embodiments of this application. Figure 3 ; Figure 6 This is a schematic diagram of the model parameter adjustment process provided in the embodiments of this application; Figure 7 This is a schematic diagram of an exemplary knowledge distillation process provided in an embodiment of this application; Figure 8 This is a schematic diagram illustrating an exemplary feature fusion method provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0020] In the following description, the terms “first, second, third, etc.” are used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, third, etc.” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0021] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0022] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0023] In practice, the data collection and processing in this application embodiment should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0024] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0025] 1) Instance Segmentation is a computer vision task used to detect each instance (i.e., object) in an image, determine the category for each object, and generate a pixel-level mask.

[0026] 2) A teacher model refers to an artificial intelligence model that outputs knowledge during knowledge distillation. The model size (model layers and parameters) is larger than a specified size, the inference speed is lower than a specified speed, and the accuracy is higher than a specified accuracy. In this application's embodiments, the instance segmentation teacher model is a teacher model used for instance segmentation.

[0027] 3) The student model refers to the AI ​​model that receives knowledge during knowledge distillation; it is the final AI model to be deployed. Compared to the teacher model, the student model is smaller in scale and faster in computation, making it suitable for practical applications where computing resources and inference latency are limited. During knowledge distillation, the training objective of the student model is not only to fit the original training data but also to mimic the behavior of the teacher model. The instance segmentation student model in this embodiment is a student model used for instance segmentation.

[0028] 4) Distillation loss is a loss function used in the knowledge distillation process to guide the student model to imitate the behavior of the teacher model. It measures the difference between the output of the student model and the output of the teacher model. It includes feature-level distillation loss (referred to as feature distillation loss), feature-relation level distillation loss (referred to as feature-relation distillation loss), and outcome-level distillation loss (referred to as outcome distillation loss).

[0029] It should be noted that when performing knowledge distillation between isomorphic or structurally similar AI models, knowledge distillation can be based on features, results, or feature relationships, which can improve the performance of the student model to some extent. However, when the teacher and student models undergoing knowledge distillation are heterogeneous, feature-based, result-based, or feature relationship-based knowledge distillation all negatively impact the performance of the student model. Even when using adapters or mapping networks to map the features of the teacher and student models to the same latent space, the performance of the student model is still affected. Furthermore, in instance segmentation, when there is a difference in the number of predicted instances between the teacher model and the student model, the feasibility of knowledge distillation based on result distillation loss is affected, thus impacting the applicability of knowledge distillation.

[0030] Based on this, embodiments of this application provide a knowledge distillation method, apparatus, electronic device, computer-readable storage medium, and computer program product for instance segmentation, which can improve the applicability of knowledge distillation and enhance the performance of instance segmentation models. The following describes exemplary applications of the electronic device for instance segmentation model distillation (hereinafter referred to as the knowledge distillation device) provided in embodiments of this application. The knowledge distillation device provided in embodiments of this application can be implemented as various types of terminals such as robots, smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart in-vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices, and smart speakers. It can also be implemented as a server, and can be implemented as both a terminal and a server. The following will describe exemplary applications when the knowledge distillation device is implemented as a server.

[0031] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the instance segmentation system provided in the embodiments of this application; as shown Figure 1 As shown, to support an instance partitioning application, in the instance partitioning system 100, terminal 200 (terminal 200-1 and terminal 200-2 are shown as examples) connects to server 400 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the instance partitioning system 100 includes a database 500 for providing data support to server 400; and... Figure 1 The example shown illustrates a scenario where the database 500 is independent of the server 400. However, the database 500 can also be integrated into the server 400, and this embodiment does not limit this to any particular case.

[0032] Terminal 200 is used to receive a trained instance segmentation student model sent by server 400 through network 300; it is also used to receive an image to be segmented, perform instance segmentation on the image to be segmented based on the trained instance segmentation student model, obtain the instance segmentation result, and present the instance segmentation result (graphical interface 200-11 and graphical interface 200-21 are shown for example).

[0033] Server 400 is used to perform knowledge distillation on the instance segmentation student model based on the instance segmentation teacher model to obtain a trained instance segmentation model, and then send the trained instance segmentation student model to terminal 200 through network 300.

[0034] It should be noted that the terminal 200 can also send the image to be segmented to the server 400 through the network 300, so that the server 400 can perform instance segmentation on the image to be segmented based on the trained instance segmentation student model, obtain the instance segmentation result, and send the instance segmentation result to the terminal 200 through the network 300.

[0035] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0036] See Figure 2 , Figure 2 This is one of the embodiments provided in this application. Figure 1 A schematic diagram of the server structure in the diagram; such as Figure 2 As shown, server 400 includes at least one processor 410, memory 450, and at least one network interface 420. The various components in server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0037] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0038] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0039] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0040] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0041] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0042] In some embodiments, the knowledge distillation apparatus for instance segmentation provided in this application can be implemented in software. Figure 2 A knowledge distillation apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: an output acquisition module 4551, an instance alignment module 4552, a loss acquisition module 4553, and a parameter adjustment module 4554. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0043] In some embodiments, the terminal or server can implement the knowledge distillation method for instance segmentation provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as instance segmentation APPs or image processing APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0044] The following will describe the knowledge distillation method for instance segmentation provided in this application embodiment, using exemplary applications and implementations of the knowledge distillation equipment provided in the embodiments of this application. Furthermore, the knowledge distillation method for instance segmentation provided in the embodiments of this application is applicable to various knowledge distillation scenarios for instance segmentation, such as cloud technology, artificial intelligence, smart transportation, medicine, security detection, and vehicle-mounted systems.

[0045] See Figure 3 , Figure 3 This is a flowchart illustrating the knowledge distillation method for instance segmentation provided in the embodiments of this application. Figure 1 In this process, the knowledge distillation equipment is the main entity executing each step; the following will combine... Figure 3 The steps shown are explained.

[0046] Step 101: Obtain the U first predicted instances output by the instance segmentation teacher model for the sample image, and obtain the V second predicted instances output by the instance segmentation student model for the sample image.

[0047] In this embodiment of the application, the knowledge distillation device is used to perform knowledge distillation on the trained instance segmentation teacher model and the instance segmentation student model to be trained; thus, when the knowledge distillation device obtains sample images for training the instance segmentation student model, the sample images are input into the instance segmentation teacher model and the instance segmentation student model respectively, so that U first predicted instances output by the instance segmentation teacher model for the sample images can be obtained, and V second predicted instances output by the instance segmentation student model for the sample images can be obtained.

[0048] It should be noted that the instance segmentation teacher model is a pre-trained teacher model used for instance segmentation of images, while the instance segmentation student model is a student model to be trained for instance segmentation of images. The sample image is the image to be segmented and used to train the instance segmentation student model; that is, the sample image is the training sample for the instance segmentation student model and contains instances to be segmented. The U first predicted instances are the prediction results obtained by the instance segmentation teacher model when segmenting the sample image. A first predicted instance refers to the predicted instance of the sample image by the instance segmentation teacher model, and the predicted instance includes the predicted category, predicted detection box, and predicted segmentation mask. The V second predicted instances are the prediction results obtained by the instance segmentation student model when segmenting the sample image. The second predicted instance refers to the predicted instance of the sample image by the instance segmentation student model. Here, U and V are unequal positive integers, indicating that the instance segmentation teacher model and the instance segmentation student model are AI models with different numbers of predicted instances, i.e., the instance segmentation teacher model and the instance segmentation student model are heterogeneous models.

[0049] It should also be noted that the process of the knowledge distillation device acquiring the U first predicted instances output by the instance segmentation teacher model for the sample image and the process of acquiring the V second predicted instances output by the instance segmentation student model for the sample image are not sequential in execution. They can be executed simultaneously, sequentially, or alternately, etc. This application embodiment does not limit this.

[0050] Step 102: Align U first predicted instances and V second predicted instances to L real instances of the sample image to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance.

[0051] In this embodiment, since the sample images are training samples for instance segmentation, the sample images have corresponding labeled data, and this labeled data consists of L real instances, each real instance including a real category, a real bounding box, and a real segmentation mask. Here, the knowledge distillation device aligns U first predicted instances and V second predicted instances to the L real instances. Thus, for each real instance, it can determine M first predicted instances aligned with the real instance from the U first predicted instances, and it can determine N second predicted instances aligned with the real instance from the V second predicted instances.

[0052] It should be noted that alignment refers to the prediction instance and the ground truth instance corresponding to the same instance in the sample image. L, M, and N are positive integers. It is easy to see that U is greater than or equal to M, and V is greater than or equal to N. Here, M can be an integer greater than 1, and N can also be an integer greater than 1. When both M and N are greater than 1, it means that multiple first prediction instances correspond to multiple second prediction instances based on a single ground truth instance. In addition, M and N can be equal or unequal.

[0053] See Figure 4 , Figure 4 This is a flowchart illustrating the knowledge distillation method for instance segmentation provided in the embodiments of this application. Figure 2 In this process, the entity executing each step is the knowledge distillation device; such as... Figure 4 As shown, step 102 can be implemented through steps 1021 to 1024; that is, in this embodiment of the application, the knowledge distillation device aligns U first predicted instances and V second predicted instances to L real instances of the sample image to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance, including steps 1021 to 1024. Each step is described below.

[0054] In this embodiment of the application, the knowledge distillation device performs the following processes (i.e., steps 1021 and 1022) for each of the L real instances.

[0055] Step 1021: From the U first predicted instances, select K first predicted instances that are initially aligned with the real instances.

[0056] In this embodiment of the application, the knowledge distillation device performs initial alignment of a first predicted instance and a real instance based on at least one of classification score and intersection-union ratio (IU). The classification score refers to the probability that the predicted category of the first predicted instance is the real category of the real instance, and the IU is the ratio of the intersection area of ​​the predicted detection box of the first predicted instance and the real detection box of the real instance to the union area.

[0057] It should be noted that K is a positive integer, and it is easy to see that K is less than or equal to U.

[0058] In step 1021 of this embodiment, the knowledge distillation device selects K first predicted instances that are initially aligned with the real instance from U first predicted instances. This includes: the knowledge distillation device first selects first predicted instances from U first predicted instances whose center point of the predicted detection box is within the real detection box of the real instance; then calculates the intersection-union ratio (IU) between the predicted detection box and the real detection box of each selected first predicted instance; next, it integrates the IU and the classification score of the first predicted instance to obtain an alignment index value; finally, based on the alignment index value, it selects K first predicted instances that are initially aligned with the real instance from the selected multiple first predicted instances.

[0059] It should be noted that since the first predicted instance includes a predicted detection box, and the predicted detection box includes a center point, such as a two-dimensional coordinate, the knowledge distillation device can determine whether the center point of the predicted detection box of the first predicted instance is located within the real detection box of the real instance. Therefore, it can select from U first predicted instances where the center point of the predicted detection box is within the real detection box of the real instance. Here, the knowledge distillation device can select one or more first predicted instances from U first predicted instances where the center point of the predicted detection box is within the real detection box of the real instance. For each first predicted instance whose center point is within the real detection box, the knowledge distillation device determines whether the first predicted instance is initially aligned with the real instance based on the classification score of the predicted detection box and the intersection-union ratio (IU) between the predicted detection box and the real detection box. The alignment index value is an index value determined by the knowledge distillation device based on the intersection-union ratio (IU) and classification score, used to characterize the probability of alignment between the first predicted instance and the real instance. The knowledge distillation device can use the weighted sum of the IU and classification score as the alignment index value, or the product of the IU and classification score, or the sum of the IU and classification score, or a combination thereof, etc. This application embodiment does not limit this. The knowledge distillation device sequentially selects the K first predicted instances with the largest alignment index values ​​from among multiple first predicted instances whose center point is within the real detection box, as the K first predicted instances initially aligned with the real instance.

[0060] Understandably, determining the K first predicted instances that are initially aligned with each real instance from the U first predicted instances based on whether the center point of the predicted detection box is within the real detection box, the intersection-union ratio, and the classification score is a process of initial alignment from multiple dimensions; therefore, it can improve the accuracy of the K first predicted instances that are initially aligned with the real instances.

[0061] Step 1022: From the V second predicted instances, select K second predicted instances that are initially aligned with the real instances.

[0062] In this embodiment, the knowledge distillation device performs initial alignment between the second predicted instance and the real instance based on at least one of the following: the probability that the predicted category of the second predicted instance is the real category in the real instance, and the ratio of the intersection area to the union area of ​​the predicted detection box of the second predicted instance and the real detection box of the real instance. Wherein, K is less than or equal to U.

[0063] In step 1021 of this embodiment, the knowledge distillation device selects K second predicted instances that are initially aligned with the real instance from V second predicted instances. This includes: the knowledge distillation device first selects second predicted instances from V first predicted instances whose center point of the predicted detection box is within the real detection box of the real instance; then calculates the intersection-union ratio (IU) between the predicted detection box and the real detection box of each selected second predicted instance; next, it integrates the IU and the classification scores of the second predicted instances to obtain an alignment index value; finally, based on the alignment index value, it selects K second predicted instances that are initially aligned with the real instance from the selected multiple second predicted instances.

[0064] It should be noted that since the second predicted instance includes a predicted detection box, and the predicted detection box includes a center point, such as a two-dimensional coordinate, the knowledge distillation device can determine whether the center point of the predicted detection box of the second predicted instance is located within the real detection box of the real instance. Therefore, it can select from V second predicted instances whose center point is within the real detection box of the real instance. Here, the knowledge distillation device can select one or more second predicted instances from V second predicted instances whose center point is within the real detection box of the real instance. For each selected second predicted instance whose center point is within the real detection box, the knowledge distillation device determines whether the second predicted instance is initially aligned with the real instance based on the classification score of the predicted detection box and the intersection-union ratio (IU) between the predicted detection box and the real detection box. The alignment index value here is an index value determined by the knowledge distillation device based on the IU and classification score to characterize the probability of the second predicted instance aligning with the real instance. From the multiple selected second predicted instances whose center points are within the real detection boxes, the knowledge distillation device sequentially selects the K second predicted instances with the largest alignment index values ​​as the K second predicted instances initially aligned with the real instance.

[0065] Understandably, determining the K second predicted instances that are initially aligned with each real instance from the V second predicted instances based on whether the center point of the predicted detection box is within the real detection box, the intersection-union ratio, and the classification score is a process of initial alignment from multiple dimensions; therefore, it can improve the accuracy of the K second predicted instances that are initially aligned with the real instances.

[0066] Step 1023: Based on the K first predicted instances initially aligned to each of the L real instances, align and deduplicate the first predicted instances to obtain M first predicted instances aligned with each real instance.

[0067] In this embodiment, when L is greater than 1, since each of the L real instances is initially aligned with K first predicted instances, L*K first predicted instances can be obtained for the L real instances. Among the L*K first predicted instances, there are first predicted instances that are initially aligned with multiple real instances. Thus, the knowledge distillation device selects one real instance from the multiple real instances that are initially aligned with the first predicted instances to achieve alignment and deduplication of the first predicted instances. Through alignment and deduplication, the knowledge distillation device redetermines the aligned real instances for the first predicted instances that are also initially aligned with other real instances among the K first predicted instances. Thus, each real instance is aligned with M first predicted instances among the K first predicted instances.

[0068] It should be noted that alignment deduplication refers to the process of determining an aligned real instance from among the multiple initially aligned real instances, given a first predicted instance that is initially aligned with multiple real instances.

[0069] In step 1023 of this application embodiment, the knowledge distillation device aligns and removes duplicates from the first predicted instances based on the K first predicted instances initially aligned to each of the L real instances, to obtain M first predicted instances aligned with each real instance. This includes: the knowledge distillation device first determines the first predicted instances initially aligned with multiple real instances based on the K first predicted instances initially aligned to each of the L real instances; then calculates the complete intersection over union (CIoU) of the first predicted instances initially aligned with multiple real instances; next, based on the CIoU, determines the real instances aligned with the first predicted instances from the multiple real instances initially aligned with the first predicted instances; finally, based on the real instances aligned with the first predicted instances, determines the M first predicted instances aligned with each real instance.

[0070] It should be noted that the knowledge distillation device, based on L real instances and K initially aligned first predicted instances for each real instance, determines the first predicted instance initially aligned with multiple real instances from the L*K first predicted instances corresponding to the L real instances; for example, first predicted instance 1 is initially aligned with real instances 3, 5, and 6. Since each first predicted instance is aligned with a first real instance, the knowledge distillation device achieves deduplication of the alignment of the first predicted instances by determining the real instance aligned with the first predicted instance from the multiple real instances initially aligned with the first predicted instance; thus, alignment deduplication refers to the process of aligning a first predicted instance with a real instance, or the process of selecting an aligned real instance from the multiple real instances initially aligned with the first predicted instance. Here, the knowledge distillation device performs alignment deduplication based on the complete intersection-union ratio to select the most aligned real instance from the multiple real instances initially aligned with the first predicted instance. The complete intersection-union ratio (CIU) is calculated based on the overlap, center point distance, and aspect ratio of the predicted detection box of the first predicted instance and the real detection box of the real instance. When calculating the CIU, for example, the difference between a specified threshold (such as 1) and the CIU is calculated first, then the ratio of the square of the distance between the center points of the predicted detection box and the real detection box to the square of the distance between the diagonals of the smallest closure box is calculated, and the aspect ratio consistency penalty term is calculated. Finally, the sum of the three is taken as the CIU.

[0071] It is understandable that alignment and deduplication through full intersection-union ratio (CIU) aligns a first predicted instance with a real instance. CIU is a composite index that integrates overlap, center distance, and aspect ratio, which can improve the accuracy of alignment between the first predicted instance and the real instance.

[0072] Step 1024: Based on the K second predicted instances initially aligned to each of the L real instances, align and deduplicate the second predicted instances to obtain N second predicted instances aligned to each real instance.

[0073] In this embodiment, when L is greater than 1, since each of the L real instances is initially aligned with K second predicted instances, L*K second predicted instances can be obtained for the L real instances. Among the L*K second predicted instances, there are second predicted instances that are initially aligned with multiple real instances. Thus, the knowledge distillation device selects one real instance from the multiple real instances that are initially aligned with the second predicted instances to achieve alignment and deduplication of the second predicted instances. Through alignment and deduplication, the knowledge distillation device redetermines the aligned real instances for the second predicted instances that are also initially aligned with other real instances among the K second predicted instances. Thus, each real instance is aligned with N second predicted instances among the K second predicted instances.

[0074] It should be noted that alignment deduplication refers to the process of determining an aligned real instance from among the multiple initially aligned real instances for a second predicted instance that is initially aligned with multiple real instances.

[0075] In step 1024 of this application embodiment, the knowledge distillation device aligns and removes duplicates from the K second predicted instances initially aligned to each of the L real instances, obtaining M second predicted instances aligned with each real instance. This includes: the knowledge distillation device first determines the second predicted instances initially aligned with multiple real instances based on the K second predicted instances initially aligned to each of the L real instances; then calculates the complete intersection-union ratio (CIU) of the second predicted instances initially aligned with multiple real instances; next, based on the CIU, determines the real instances aligned with the second predicted instances from the multiple real instances initially aligned with the second predicted instances; finally, based on the real instances aligned with the second predicted instances, determines N second predicted instances aligned with each real instance.

[0076] It should be noted that the knowledge distillation device, based on L real instances and K initially aligned second predicted instances for each real instance, determines second predicted instances that are initially aligned with multiple real instances from the L*K second predicted instances corresponding to the L real instances; for example, second predicted instance 1 is initially aligned with real instances 1, 3, and 4. Since each second predicted instance is aligned with a first real instance, the knowledge distillation device achieves alignment deduplication by determining the real instance aligned with the second predicted instance from the multiple initially aligned real instances. Alignment deduplication refers to the process of aligning a second predicted instance with a real instance, or the process of selecting an aligned real instance from the multiple initially aligned real instances of the second predicted instance. Here, the knowledge distillation device performs alignment deduplication based on the complete intersection-union ratio (CIU) to select the most aligned real instance from the multiple initially aligned real instances of the second predicted instance. The CIU is calculated based on the overlap, center-point distance, and aspect ratio of the predicted detection boxes of the second predicted instance and the real detection boxes of the real instances.

[0077] It is understandable that alignment and deduplication through full intersection-union ratio (CIU) can align a second predicted instance with a real instance. CIU is a composite index that integrates overlap, center point distance, and aspect ratio, which can improve the accuracy of alignment between the second predicted instance and the real instance.

[0078] It should be noted that steps 1021 and 1022 are not in any particular order of execution, as are steps 1023 and 1024. Figure 4 An example execution order is given in the document.

[0079] Step 103: Calculate the instance distillation loss based on the differences between the M first prediction instances and the N second prediction instances.

[0080] In this embodiment, the knowledge distillation device calculates the distillation loss based on the difference between each of the M first predicted instances and each of the N second predicted instances, thus obtaining the instance distillation loss. Furthermore, the difference between each first predicted instance and each second predicted instance refers to the difference corresponding to at least one of the following: detection box, category, and segmentation mask.

[0081] It should be noted that instance distillation loss refers to the distillation loss between the M first predicted instances and N second predicted instances corresponding to each real instance. Distillation loss refers to the loss function value of knowledge distillation.

[0082] See also Figure 4 In this embodiment of the application, step 103 can be implemented by steps 1031 to 1034; that is, the knowledge distillation device calculates the instance distillation loss based on the difference between M first prediction instances and N second prediction instances, including steps 1031 to 1034, and each step is described below.

[0083] In this embodiment of the application, the knowledge distillation device performs the following processing (i.e., steps 1031 to 1033) on each of the M first prediction instances and each of the N second prediction instances.

[0084] Step 1031: Calculate the classification distillation loss based on the difference between the predicted category of the first prediction instance and the predicted category of the second prediction instance.

[0085] It should be noted that since both the first and second predicted instances include their respective predicted categories, and both correspond to the same real instance, the knowledge distillation device can calculate the distillation loss based on the difference between the predicted categories of the first and second predicted instances, thus obtaining the classification distillation loss. Therefore, the classification distillation loss refers to the loss function value between the predicted categories of the first and second predicted instances. Here, the knowledge distillation device can obtain the classification distillation loss by calculating the divergence (e.g., KL divergence) between the predicted categories of the first and second predicted instances.

[0086] Step 1032: Calculate the detection distillation loss based on the difference between the predicted detection box of the first prediction instance and the predicted detection box of the second prediction instance.

[0087] It should be noted that since both the first and second predicted instances include their own predicted detection boxes, and both correspond to the same real instance, the knowledge distillation device can calculate the distillation loss based on the difference between the predicted detection boxes of the first and second predicted instances, thus obtaining the detection distillation loss. Therefore, the detection distillation loss refers to the loss function value between the predicted detection boxes of the first and second predicted instances. Here, the knowledge distillation device can obtain the detection distillation loss by calculating the KL divergence between the predicted detection boxes of the first and second predicted instances.

[0088] Step 1033: Calculate the segmentation distillation loss based on the difference between the predicted segmentation mask of the first prediction instance and the predicted segmentation mask of the second prediction instance.

[0089] It should be noted that since both the first and second predicted instances include their own predicted segmentation masks, and the first and second predicted instances correspond to the same real instance, the knowledge distillation device can calculate the distillation loss based on the difference between the predicted segmentation masks of the first and second predicted instances, thus obtaining the segmentation distillation loss. Therefore, the segmentation distillation loss refers to the loss function value between the predicted segmentation masks of the first and second predicted instances. Here, the knowledge distillation device can obtain the segmentation distillation loss by calculating the smoothed average absolute error (L1) loss between the predicted segmentation masks of the first and second predicted instances.

[0090] In the embodiments of this application, steps 1031 to 1033 are not in any particular order of execution.

[0091] Step 1034: Integrate the M*N classification distillation losses, M*N detection distillation losses, and M*N segmentation distillation losses to obtain the instance distillation loss.

[0092] It should be noted that after the knowledge distillation device obtains the classification distillation loss for the predicted category of each first prediction instance and the predicted category of each second prediction instance, it can obtain M*N classification distillation losses for M first prediction instances and N second prediction instances. Similarly, after the knowledge distillation device obtains the detection distillation loss for the predicted detection box of each first prediction instance and the predicted detection box of each second prediction instance, it can obtain M*N detection distillation losses for M first prediction instances and N second prediction instances. Finally, after the knowledge distillation device obtains the segmentation distillation loss for the predicted segmentation mask of each first prediction instance and the predicted segmentation mask of each second prediction instance, it can obtain M*N segmentation distillation losses for M first prediction instances and N second prediction instances. Here, the knowledge distillation device can integrate M*N classification distillation losses, M*N detection distillation losses, and M*N segmentation distillation losses in at least one of the following ways: weighted summation, mean calculation, maximum value acquisition, first calculating the mean values ​​corresponding to the M*N classification distillation losses, M*N detection distillation losses, and M*N segmentation distillation losses respectively, and then weighted summation, combinations of the above, etc.

[0093] Understandably, by calculating the loss function value for each first predicted instance and each second predicted instance corresponding to the same real instance from three dimensions—classification, detection, and segmentation—the processing of obtaining instance distillation loss from multiple dimensions is achieved; therefore, the accuracy of instance distillation loss can be improved.

[0094] Step 104: Calculate the distillation loss based on the distillation losses of the L instances corresponding to the L real instances.

[0095] In this embodiment, since the knowledge distillation device obtains an instance distillation loss for each of the L real instances, it can obtain L instance distillation losses for the L real instances; the L instance distillation losses correspond one-to-one with the L real instances. Here, the knowledge distillation device can process the L instance distillation losses using at least one of the following methods to calculate the final distillation loss: weighted summation, averaging, obtaining the maximum value, or a combination of the above.

[0096] It should be noted that the result distillation loss refers to the distillation loss of the prediction result dimension of the instance segmentation teacher model and the instance segmentation student model for the sample image.

[0097] Step 105: Adjust the model parameters of the instance segmentation student model based on the result distillation loss.

[0098] In this embodiment, with the model parameters of the instance segmentation teacher model frozen, the knowledge distillation device can directly perform backpropagation in the instance segmentation student model based on the result distillation loss. Alternatively, it can perform backpropagation in the instance segmentation student model using a model that combines the result distillation loss function with other distillation losses to adjust the model parameters and train the instance segmentation student model. These other distillation losses include at least one of the following: distillation loss based on the feature dimension of both the instance segmentation teacher and student models, distillation loss based on the feature relationship dimension, and the loss between the prediction result of the instance segmentation student model and the labeled data of the sample images.

[0099] It should be noted that the adjustment of the model parameters of the instance segmentation student model can be iterative. When the iteration termination condition is met, the adjustment of the model parameters ends, and the instance segmentation student model adjusted in the last iteration is determined as the trained instance segmentation student model. The iteration termination condition can be greater than the accuracy index threshold, equal to the iteration number threshold, greater than the iteration duration threshold, or a combination of the above, etc. This application embodiment does not limit this.

[0100] Understandably, after obtaining U first predicted instances of the instance segmentation teacher model and V second predicted instances of the instance segmentation student model for the sample image, the U first predicted instances and V second predicted instances are aligned to L real instances of the sample image. Using real instances as intermediate information, the alignment of the first and second predicted instances is performed to obtain M first predicted instances aligned with N second predicted instances. Thus, even when the number of predicted instances in the instance segmentation teacher model and the instance segmentation student model differs, the alignment of the predicted instances in the instance segmentation teacher model and the instance segmentation student model is achieved. This allows the result distillation loss to be calculated based on the difference between the N second predicted instances and the M first predicted instances, thereby achieving knowledge distillation. Therefore, the feasibility and applicability of knowledge distillation based on result distillation loss are improved.

[0101] See Figure 5 , Figure 5 This is a flowchart illustrating the knowledge distillation method for instance segmentation provided in the embodiments of this application. Figure 3 In this process, the entity executing each step is the knowledge distillation device; such as... Figure 5 As shown, step 105 can be implemented through steps 1051 to 1054; that is, in this embodiment of the application, the knowledge distillation device adjusts the model parameters of the instance segmentation student model based on the result distillation loss, including steps 1051 to 1054, and each step is described below.

[0102] 1051. Obtain the first features at scale I output by the instance segmentation student model for the sample image, and obtain the second features at scale J output by the instance segmentation teacher model for the sample image.

[0103] In this embodiment of the application, after the knowledge distillation device inputs the sample image into the instance segmentation student model and the instance segmentation teacher model respectively, it obtains the multi-scale features extracted by the instance segmentation student model for the sample image and used as input to the segmentation head, thus obtaining the first feature of I scales output by the instance segmentation student model for the sample image; it also obtains the multi-scale features extracted by the instance segmentation teacher model for the sample image and used as input to the segmentation head, thus obtaining the second feature of J scales output by the instance segmentation teacher model for the sample image.

[0104] It should be noted that the first feature at scale I consists of multiple first features with different scales, and the second feature at scale J consists of multiple second features with different scales. Since I and J are not equal positive integers, this indicates that the instance segmentation teacher model and the instance segmentation student model output features with different scales. Of course, I and J can also be equal positive integers. Furthermore, since I and J are not equal positive integers, this indicates that the scales of the features output by the instance segmentation teacher model and the instance segmentation student model are also different, meaning that at least one of the resolution and the number of channels is different.

[0105] 1052. Align the second features at J scales to the first features at I scales to obtain the third features at I scales.

[0106] It should be noted that the knowledge distillation device aligns the second features at J scales to the first features at I scales by fusing features from the second features at J scales; and the knowledge distillation device uses the I scale as the reference scale after feature fusion to perform feature fusion on the second features at J scales. The third feature at the I scale is the feature fusion result of the second features at J scales. Here, the knowledge distillation device can use a bidirectional feature pyramid network (BiFPN) to align the second features at J scales to the first features at I scale. In this case, when adjusting the model parameters of the instance segmentation model based on losses (e.g., various distillation losses, supervised losses, etc.), the model parameters of the bidirectional feature pyramid network are also adjusted based on the losses.

[0107] In embodiment 1052 of this application, the knowledge distillation device aligns a second feature of J scales to a first feature of I scale to obtain a third feature of I scale. This includes: the knowledge distillation device upsampling the second feature starting from the lowest scale among the J scales to obtain the p-th upsampled feature; integrating the second feature of the J scales that has the same scale as the p-th upsampled feature and the p-th upsampled feature into a fourth feature; when upsampling to the highest scale among the I scales, a fourth feature of I scale is obtained; downsampling the fourth feature starting from the highest scale among the I scales to obtain the q-th downsampled feature; integrating the fourth feature of the I scales that has the same scale as the q-th downsampled feature and the q-th downsampled feature into a third feature; and when downsampling to the lowest scale among the I scales, a third feature of I scale is obtained.

[0108] It should be noted that since J scales are multiple different scales, the knowledge distillation device determines the lowest scale from the J scales and upsamples the second feature from the lowest scale. Each upsampling yields an upsampled feature. Here, the variable p represents the number of upsampling operations, so the p-th upsampling operation yields the p-th upsampled feature. The fourth feature refers to the integration result of the upsampled feature and the second feature of the same scale. Integration may involve concatenation followed by convolution. Convolution is used to obtain features with the same number of channels as the I scales. Therefore, convolution is performed on the concatenated features to align the number of channels. When the knowledge distillation device upsamples to the highest scale among the I scales, it stops upsampling at the fourth feature obtained at the corresponding scale, at which point the fourth feature of the I scales is obtained. Here, the first feature to be upsampled is the second feature of the lowest scale, and subsequent upsampled features are the fourth features of the corresponding scale of the current output.

[0109] The knowledge distillation device determines the first feature to be downsampled based on the second feature of at least one scale higher than the highest scale among J scales. Downsampling begins from this first feature. When the second feature of at least one scale is a second feature of one scale, this second feature of that scale is the feature to be downsampled. When the second feature of at least one scale is a second feature of multiple scales, the fusion result of the second features of multiple scales is the feature to be downsampled. The feature to be downsampled is also called the fusion feature of the second features of at least one scale. The knowledge distillation device starts downsampling from the fusion feature of the second features of at least one scale, and obtains a downsampled feature after each downsampling. Here, the sampling variable q represents the number of downsampling attempts, so the qth downsampling attempt obtains the qth downsampled feature. The third feature refers to the integration result of the downsampled feature and the fourth feature of the same scale. When the knowledge distillation device downsamples to the lowest scale among the I scales, it stops downsampling when it obtains the third feature of the corresponding scale, at which point it obtains the third features of I scales. Here, the first feature to be downsampled is a fused feature of the second feature at at least one scale, and the subsequent downsampled features are the third feature of the current output at the corresponding scale.

[0110] It is understandable that by using the I scales of the first feature as a reference, the second feature at J scales is fused to obtain the third feature at I scale. This achieves alignment between the second feature at J scales and the first feature at I scales in terms of scale number and scale, which can improve the accuracy of feature alignment.

[0111] 1053. Calculate the feature distillation loss based on the difference between the first feature at I scales and the third feature at I scales.

[0112] It should be noted that the knowledge distillation device calculates the loss function value between the first feature and the second feature of each of the I scales, and integrates the obtained I loss function values ​​(e.g., weighted summation, mean calculation, maximum value acquisition, etc.) into the feature distillation loss; thus, the feature distillation loss represents the distillation loss of the feature dimension between the second feature of the J scales and the first feature of the I scales.

[0113] 1054. Adjust the model parameters of the instance segmentation student model based on feature distillation loss and result distillation loss.

[0114] It should be noted that, with the model parameters of the instance segmentation teacher model frozen, the knowledge distillation device can directly perform backpropagation in the instance segmentation student model based on the feature distillation loss and the outcome distillation loss. Alternatively, it can perform backpropagation in the instance segmentation student model using a model that combines the feature distillation loss, the outcome distillation loss, and an additional distillation loss to adjust the model parameters and train the instance segmentation student model. The additional distillation loss includes at least one of the following: distillation loss based on the feature relationship dimension of the instance segmentation teacher model and the instance segmentation student model, and the loss between the prediction result of the instance segmentation student model and the labeled data of the sample images. Here, when the knowledge distillation device directly adjusts the model parameters of the instance segmentation student model based on the feature distillation loss and the outcome distillation loss, it can adjust the model parameters based on the weighted sum of the feature distillation loss and the outcome distillation loss.

[0115] It is understandable that by adjusting the model parameters of the instance segmentation student model through feature distillation loss and result distillation loss, knowledge distillation from two dimensions, namely features and prediction results, can be achieved, thereby improving the accuracy of knowledge distillation and enhancing the instance segmentation performance of the instance segmentation student model.

[0116] See Figure 6 , Figure 6 This is a schematic diagram of the model parameter adjustment process provided in the embodiments of this application; as follows: Figure 6 As shown in the embodiment of this application, step 1054 can be implemented through steps 10541 to 10545; that is, the knowledge distillation device adjusts the model parameters of the instance segmentation student model based on feature distillation loss and result distillation loss, including steps 10541 to 10545, and each step is described below.

[0117] Step 10541: Based on the first feature and the third feature at the I scale corresponding to the sample image, determine the multiple first features at the I scale and the multiple third features at the I scale corresponding to the multiple sample images.

[0118] It should be noted that, since the instance segmentation teacher model can obtain J scales of second features for each sample image, and I scales of third features can be obtained from the J scales of second features, the knowledge distillation device can obtain multiple I scales of third features for multiple sample images; since the instance segmentation student model can obtain I scales of first features for each sample image, the knowledge distillation device can obtain multiple I scales of first features for multiple sample images.

[0119] Step 10542: Calculate the first feature similarity between the first features of multiple I scales corresponding to multiple sample images.

[0120] It should be noted that the knowledge distillation device performs the following processing for each of the I scales: It obtains the first similarity matrix of multiple first features at the corresponding scale, resulting in I first similarity matrices, which are referred to as the first feature similarity. Here, the knowledge distillation device may use the matrix dot product of multiple first features at the corresponding scale with the transpose of those first features as the first similarity matrix.

[0121] Step 10543: Calculate the second feature similarity between the third features of multiple I scales corresponding to multiple sample images.

[0122] It should be noted that the knowledge distillation device performs the following processing for each of the I scales: It obtains the second similarity matrix of multiple third features at the corresponding scale, resulting in I second similarity matrices, which are referred to as the second feature similarity. Here, the knowledge distillation device can use the matrix dot product of multiple third features at the corresponding scale with their transposes as the second similarity matrix.

[0123] In this embodiment of the application, steps 10542 and 10543 are not in any particular order of execution. Figure 6 The given example execution order is as follows.

[0124] Step 10544: Calculate the feature relationship distillation loss based on the difference between the first feature similarity and the first feature similarity.

[0125] It should be noted that the knowledge distillation device performs the following processing for each of the I scales: It calculates the difference between the first and second similarity matrices at the corresponding scale, thus obtaining the sub-feature relation distillation loss for that scale; from the sub-feature relation distillation losses at each scale, it obtains I sub-feature relation distillation losses for all I scales; and it integrates the I sub-feature relation distillation losses as the feature relation distillation loss. Here, the knowledge distillation device can use a specified loss function to determine the sub-feature relation distillation loss between the first and second similarity matrices at the corresponding scale. For example, the ratio of the square of the norm (L2 norm, etc.) to the square of the number of sample images can be used as the sub-feature relation distillation loss.

[0126] Step 10545: Adjust the model parameters of the instance segmentation student model based on feature relation distillation loss, feature distillation loss, and result distillation loss.

[0127] It should be noted that the knowledge distillation device integrates the feature relationship distillation loss, feature distillation loss, and result distillation loss (e.g., weighted summation, mean calculation, maximum value acquisition, etc.) into the final distillation loss, and adjusts the model parameters of the instance segmentation student model based on the obtained final distillation loss.

[0128] It is understandable that by combining feature relation distillation loss, feature distillation loss, and result distillation loss to adjust the model parameters of the instance segmentation student model, the knowledge distillation of the instance segmentation student model is carried out from three different dimensions: features, feature relations, and results. This improves the comprehensiveness of knowledge distillation, thereby enhancing the effectiveness of knowledge distillation and improving the instance segmentation accuracy of the instance segmentation student model.

[0129] In this embodiment of the application, the knowledge distillation device can also obtain the supervision loss of V second predicted instances and L real instances, so as to integrate the supervision loss, feature relationship distillation loss, feature distillation loss and result distillation loss (e.g., weighted summation, mean calculation, maximum value acquisition, etc.) as the final loss of the instance segmentation student model.

[0130] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. This exemplary application describes the process in an instance segmentation task.

[0131] In the embodiments of this application, the artificial intelligence models used for instance segmentation include two categories. One category is convolutional neural network models (e.g., the YOLO model). Convolutional neural network models utilize multi-scale feature pyramids and convolutional structures, enabling real-time inference on computing-limited devices. For example, they can be applied in fields such as autonomous driving, industrial inspection, and intelligent security. The other category is vision transformation (ViT) models (e.g., the DINOv3 model, the EVA-02 model, etc.). Vision transformation models employ a self-attention mechanism, including global modeling capabilities. Compared to convolutional neural network models, they have stronger generalization, higher accuracy, and higher robustness. However, due to the number of parameters and inference speed, they cannot be directly deployed in actual edge testing environments.

[0132] To balance the accuracy and deployment requirements of artificial intelligence models, a knowledge distillation method can be employed. This method uses a visual transformation model as the teacher model and a convolutional neural network model as the student model, transferring the feature representation and generalization capabilities of the teacher model to the student model. Based on this, this application proposes a knowledge distillation method that simultaneously combines feature-level alignment, feature relationship-level alignment, and result-level alignment for the distillation of heterogeneous instance segmentation models.

[0133] See Figure 7 , Figure 7This is a schematic diagram of an exemplary knowledge distillation process provided in an embodiment of this application; as shown below. Figure 7 As shown, knowledge distillation begins. With the model parameters of teacher model 7-11 (referred to as instance segmentation teacher model) frozen, teacher model 7-11 and student model 7-12 (referred to as instance segmentation student model) are used to perform forward propagation on the same sample image 7-2 respectively. The multi-scale features 7-31 (referred to as J scale second features) of teacher model 7-11 used as input to the segmentation head and the multi-scale features 7-32 (referred to as I scale first features) of student model 7-12 used as input to the segmentation head are obtained. The prediction results 7-41 (referred to as U first predicted instances) of teacher model 7-11 and the prediction results 7-42 (referred to as V second predicted instances) of student model 7-12 are also obtained.

[0134] Next, the multi-scale feature 7-31 is input into the bidirectional feature pyramid network 7-5 to output a multi-scale feature 7-33 (called the third feature of I scales) with the same number of scales, channels and resolution as the multi-scale feature 7-32.

[0135] See Figure 8 , Figure 8 This is a schematic diagram illustrating an exemplary feature fusion method provided in an embodiment of this application; as shown... Figure 8As shown, the number of scales in multi-scale feature 7-31 is 5, namely 1 / 64, 1 / 32, 1 / 16, 1 / 8, and 1 / 4. When the number of scales in multi-scale feature 7-32 is 3, namely 1 / 32, 1 / 16, and 1 / 8, the feature 7-311 with a scale of 1 / 64 (called the lowest scale among the J scales) in multi-scale feature 7-31 is subjected to stepwise interpolation upsampling (upSample, called upsampling). The resulting feature with a scale of 1 / 32 (called the upsampled feature) is concatenated (Concat) with the feature 7-312 with a scale of 1 / 32 in multi-scale feature 7-31, and the concatenated features are fused (ConvBlock) to obtain feature 8-1 with a scale of 1 / 32 (called the fourth feature). Feature 8-1 is progressively upsampled using interpolation. The resulting feature at scale 1 / 16 (called the upsampled feature) is concatenated with feature 7-313 at scale 1 / 16 from multi-scale feature 7-31. The concatenated features are then fused to obtain feature 8-2 at scale 1 / 16 (called the fourth feature). Feature 8-2 is then progressively upsampled using interpolation. The resulting feature at scale 1 / 8 (called the upsampled feature) is concatenated with feature 7-314 at scale 1 / 8 from multi-scale feature 7-31. The concatenated features are then fused to obtain feature 8-3 at scale 1 / 8 (called the fourth feature). A convolutional block with stride 2 is used to progressively downsample feature 7-315 at scale 1 / 4 from multi-scale feature 7-31 (called the second feature at at least one scale). The resulting feature at scale 1 / 8 (called the downsampled feature) is concatenated with feature 8-3. The concatenated features are then fused to obtain feature 8-4 at the target scale 1 / 8 (called the third feature). Feature 8-4 is downsampled stepwise using a convolutional block with a stride of 2. The resulting feature at a scale of 1 / 16 (called the downsampled feature) is concatenated with feature 8-2, and the concatenated features are then fused to obtain feature 8-5 (called the third feature) at a target scale of 1 / 16. Feature 8-5 is downsampled stepwise using a convolutional block with a stride of 2. The resulting feature at a scale of 1 / 32 (called the downsampled feature) is concatenated with feature 8-1, and the concatenated features are then fused to obtain feature 8-6 (called the third feature) at a target scale of 1 / 32. It is easy to see that multi-scale feature 7-31 includes features 7-311, 7-312, 7-313, 7-314, and 7-315, and multi-scale feature 7-33 includes features 8-4, 8-5, and 8-6.

[0136] See also Figure 7 The smoothed average absolute error is calculated based on multi-scale feature 7-32 and multi-scale feature 7-33 aligned with multi-scale feature 7-32. )loss (referred to as the distillation loss corresponding to each scale in the characteristic distillation loss), through This allows students to mimic the feature values ​​of the same spatial locations (e.g., the same pixels) in the teacher's model, thereby transferring the knowledge of the teacher's model to the student model in the form of a representation distribution. As shown in formula (1).

[0137] (1); in, The first teacher model The first sample image Features at each scale (called the third feature) The student model represents the first The first sample image Features at each scale (called the first feature). This represents a threshold parameter, for example, 1; Indicates the number of sample images. Indicates the number of scales.

[0138] See also Figure 7 Furthermore, the consistency loss of feature relationships between samples can be calculated based on multi-scale feature 7-32 and multi-scale feature 7-33 aligned with multi-scale feature 7-32. (referred to as characteristic relationship distillation loss), as shown in formula (2).

[0139] (2); in, It can be a function that measures any relation. The feature matrix represents the teacher model or student model. Represents the similarity matrix. Representing the teacher model The first sample image Feature matrices at various scales Representing the student model The first sample image Feature matrices at various scales Indicates spatial dimension, This represents the square of the norm.

[0140] Understandably, using the feature relationship consistency loss between samples as an auxiliary alignment objective, by calculating the similarity matrix of features between samples to minimize the difference between the teacher model and the student model, enables the student model to learn the teacher model's knowledge of feature relationships between samples during knowledge distillation. Since feature relationships are independent of feature representation forms, the feature relationship consistency loss can provide a stable and reliable distillation signal for the student model, thereby accelerating the convergence speed of the student model.

[0141] Ultimately, it can be based on and Calculation of multi-scale characteristic distillation loss 7-61 ( As shown in formula (3).

[0142] (3); See also Figure 7 In the distillation of prediction results, this embodiment performs one-to-many matching between the prediction results 7-41 of the teacher model and 7-42 of the prediction results 7-12 of the student model and the real instances, respectively, so that multiple prediction results are matched for each real instance. For example, for each real instance, a task-aligned assigner is used to filter out prediction results whose center points are within the real detection boxes. Then, an alignment matrix for all real instances is calculated based on IoU and classification scores. Finally, the top K prediction instances (referred to as K first prediction instances and K second prediction instances) are selected for each real instance according to the alignment matrix, and the repeatedly assigned prediction instances are removed using CIoU (referred to as first prediction instances initially aligned with multiple real instances and second prediction instances initially aligned with multiple real instances). The teacher model and the student model use the same assigner. Based on the correspondence between the two and the real annotations, a many-to-many matching relationship between the student model and the teacher model can be obtained (referred to as M first prediction instances aligned with each real instance and N second prediction instances aligned with each real instance). Finally, the KL divergence loss (for category and segmentation mask) or smoothing L1 loss (for detection box) is calculated between each predicted instance of the student model and multiple predicted instances of the teacher model corresponding to the same real instance, in order to transfer the prediction information of the teacher model to the student model to the greatest extent.

[0143] It should be noted that the KL divergence loss for predicting categories... (Referred to as the loss of distillation) is shown in Equation (4).

[0144] (4); in, The first teacher model Each prediction category The student model represents the first Each prediction category This represents the number of predicted instances from the teacher model that are aligned with the same real instance. This represents the number of predicted instances of the student model aligned with the same real instance. The loss function represents the KL divergence.

[0145] KL divergence loss for predicting segmentation masks (Referred to as the fractional distillation loss) is shown in Equation (5).

[0146] (5); in, The student model represents the first A predicted segmentation mask, The first teacher model A predictive segmentation mask.

[0147] Smooth L1 loss for predicting detection boxes (Referred to as the detection of distillation loss) as shown in formula (6).

[0148] (6); in, The student model represents the first One predicted detection box, The first teacher model One predicted detection box, Indicates smoothness Loss function.

[0149] Based on the KL divergence loss of the predicted category, the KL divergence of the predicted segmentation mask, and the smoothing L1 loss of the predicted detection box, the distillation loss of the many-to-many prediction result is obtained (7-62).

[0150] Furthermore, for the prediction result 7-42 of the student model 7-12, the task loss 7-63 (referred to as the supervision loss) between the prediction result 7-42 and the true label is also included. The multi-scale feature distillation loss 7-61, the prediction result distillation loss 7-62, and the task loss 7-63 are weighted and summed to obtain the final target loss 7-7. Backpropagation is then performed on the student model 7-12 and the bidirectional feature pyramid network 7-5 based on the target loss 7-6 to update the model parameters of both models until training is complete. This significantly improves accuracy and generalization ability while maintaining lightweight design and efficiency.

[0151] Understandably, BiFPN can fuse semantic and spatial detail information. Based on BiFPN's multi-scale heterogeneous feature alignment mechanism, it can fuse and remap the multi-scale features of the teacher model, making its output fully aligned with the student model in terms of scale quantity, channel quantity, spatial resolution, and feature representation. This achieves knowledge distillation across architectures (such as attention mechanisms (Vision Transformer, ViT) and CNN (Convolutional Neural Network)). Furthermore, after achieving alignment in scale quantity, channel quantity, and spatial resolution, the consistency of feature relationships between samples is used as the alignment target to overcome differences in feature representation forms across different architectures and reduce the impact of BiFPN learning, thereby improving model convergence speed. Additionally, through a many-to-many matching heterogeneous soft label distillation method, a many-to-many matching relationship is constructed between the prediction results of the teacher and student models using real instances as intermediate information. This achieves knowledge transfer between heterogeneous instance segmentation heads, enhancing the accuracy of instance segmentation in the student model. Therefore, the knowledge distillation method for instance segmentation provided in this application, on the one hand, maps teacher features to scale, channels, and resolution using BiFPN, and combines the dual constraints of feature value consistency and sample relationship consistency, reducing the impact of differences in feature representation methods of heterogeneous models on knowledge distillation, thereby improving the stability and sufficiency of knowledge distillation; on the other hand, it establishes a many-to-many matching relationship between the prediction results of the teacher model and the student model using real labels as intermediate information, reducing the impact of differences in the number and organization of predicted instances on knowledge distillation, and maximizing the transmission of instance segmentation knowledge from the teacher model, thereby improving the instance segmentation accuracy of the student model.

[0152] The following description continues to illustrate the exemplary structure of the knowledge distillation apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the knowledge distillation apparatus 455 of the memory 450 may include: The output acquisition module 4551 is used to acquire U first predicted instances output by the instance segmentation teacher model for the sample image, and to acquire V second predicted instances output by the instance segmentation student model for the sample image, where U and V are unequal positive integers. The instance alignment module 4552 is used to align the U first predicted instances and the V second predicted instances to L real instances of the sample image, so as to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance, where L, M and N are positive integers. The loss acquisition module 4553 is used to calculate the instance distillation loss based on the difference between the M first prediction instances and the N second prediction instances; The loss acquisition module 4553 is also used to calculate the distillation loss result based on the distillation loss of the L instances corresponding to the L real instances; The parameter adjustment module 4554 is used to adjust the model parameters of the instance segmentation student model based on the distillation loss of the results.

[0153] In some embodiments, the instance alignment module 4552 is further configured to perform the following processing for each of the L real instances: selecting K first predicted instances initially aligned with the real instance from the U first predicted instances, and selecting K second predicted instances initially aligned with the real instance from the V second predicted instances, where K is a positive integer; aligning and deduplicating the first predicted instances based on the K first predicted instances initially aligned with each of the L real instances to obtain the M first predicted instances aligned with each of the real instances; aligning and deduplicating the second predicted instances based on the K second predicted instances initially aligned with each of the L real instances to obtain the N second predicted instances aligned with each of the real instances.

[0154] In some embodiments, the instance alignment module 4552 is further configured to: select, from the U first predicted instances, a first predicted instance whose center point is within the real detection box of the real instance; calculate the intersection-union ratio (IU) of the predicted detection box and the real detection box for each selected first predicted instance; integrate the IU IU and the classification score of the first predicted instance to obtain an alignment index value; and, based on the alignment index value, select, from the selected plurality of first predicted instances, the K first predicted instances that are initially aligned with the real instance.

[0155] In some embodiments, the instance alignment module 4552 is further configured to: determine the first predicted instance initially aligned with a plurality of real instances based on the K first predicted instances initially aligned to each of the L real instances; calculate the complete intersection-union ratio (CIU) of the first predicted instances initially aligned with the plurality of real instances; determine the real instance aligned to the first predicted instance from the plurality of real instances initially aligned to the first predicted instance based on the CIU; and determine the M first predicted instances aligned to each of the real instances based on the real instances aligned to the first predicted instance.

[0156] In some embodiments, the loss acquisition module 4553 is further configured to perform the following processing on each of the M first prediction instances and each of the N second prediction instances: calculating a classification distillation loss based on the difference between the predicted category of the first prediction instance and the predicted category of the second prediction instance; calculating a detection distillation loss based on the difference between the predicted detection box of the first prediction instance and the predicted detection box of the second prediction instance; calculating a segmentation distillation loss based on the difference between the predicted segmentation mask of the first prediction instance and the predicted segmentation mask of the second prediction instance; and integrating the M*N classification distillation losses, the M*N detection distillation losses, and the M*N segmentation distillation losses to obtain the instance distillation loss.

[0157] In some embodiments, the parameter adjustment module 4554 is further configured to: obtain first features at I scales output by the instance segmentation student model for the sample image; obtain second features at J scales output by the instance segmentation teacher model for the sample image, where I and J are unequal positive integers; align the second features at J scales to the first features at I scales to obtain third features at I scales; calculate feature distillation loss based on the difference between the first features at I scales and the third features at I scales; and adjust the model parameters of the instance segmentation student model based on the feature distillation loss and the result distillation loss.

[0158] In some embodiments, the parameter adjustment module 4554 is further configured to: upsample the second feature starting from the lowest scale among the J scales to obtain the p-th upsampled feature, where p is a variable representing the number of upsampling times; integrate the second feature with the same scale as the p-th upsampled feature among the second features of the J scales and the p-th upsampled feature into a fourth feature; when upsampling to the highest scale among the I scales, obtain the fourth feature of the I scale; downsample starting from the fused feature of the second feature at least one scale above the highest scale of the I scale among the J scales to obtain the q-th downsampled feature, where q is a variable representing the number of downsampling times; integrate the fourth feature with the same scale as the q-th downsampled feature among the fourth features of the I scales and the q-th downsampled feature into a third feature; when downsampling to the lowest scale among the I scales, obtain the third feature of the I scale.

[0159] In some embodiments, the parameter adjustment module 4554 is further configured to: determine multiple first features at multiple I scales and multiple third features at multiple I scales corresponding to multiple sample images based on the first features at multiple I scales and the third features at multiple I scales corresponding to the sample images; calculate a first feature similarity among the multiple first features at multiple I scales corresponding to multiple sample images; calculate a second feature similarity among the multiple third features at multiple I scales corresponding to multiple sample images; calculate a feature relationship distillation loss based on the difference between the first feature similarity and the first feature similarity; and adjust the model parameters of the instance segmentation student model based on the feature relationship distillation loss, the feature distillation loss, and the result distillation loss.

[0160] This application provides a computer program product including computer-executable instructions or a computer program stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the knowledge distillation method for instance segmentation described above in this application.

[0161] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the knowledge distillation method for instance segmentation provided in this application. For example, ... Figure 3 The knowledge distillation method for instance segmentation is shown.

[0162] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0163] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0164] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0165] As an example, computer-executable instructions can be deployed to execute on a single electronic device (in which case, this single electronic device is a knowledge distillation device), or to execute on multiple electronic devices located in one location (in which case, multiple electronic devices located in one location are knowledge distillation devices), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in which case, multiple electronic devices distributed across multiple locations and interconnected via a communication network are knowledge distillation devices).

[0166] It is understood that in the embodiments of this application, data such as sample images are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required. The collection, use and processing of related data must comply with relevant laws, regulations and standards, and conform to the principles of legality, legitimacy and necessity. It does not involve obtaining data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0167] In summary, this embodiment, after obtaining U first predicted instances of the instance segmentation teacher model and V second predicted instances of the instance segmentation student model for the sample image, aligns the U first predicted instances and V second predicted instances to L real instances of the sample image, using real instances as intermediate information for the alignment of the first and second predicted instances, to obtain M first predicted instances aligned with N second predicted instances. Thus, even when the number of predicted instances in the instance segmentation teacher model and the instance segmentation student model differs, alignment of the predicted instances in the instance segmentation teacher model and the instance segmentation student model is achieved. This allows for the calculation of the result distillation loss based on the difference between the N second predicted instances and the M first predicted instances, thereby achieving knowledge distillation. Therefore, it improves the feasibility and applicability of knowledge distillation based on the result distillation loss. Furthermore, by using a bidirectional feature pyramid network to align the multi-scale features of the teacher model to the multi-scale features of the student model in terms of both the number and scale, the accuracy of feature alignment is improved, thereby enhancing the accuracy of knowledge distillation.

[0168] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A knowledge distillation method for instance segmentation, characterized in that, The method includes: Obtain U first predicted instances output by the instance segmentation teacher model for the sample image, and obtain V second predicted instances output by the instance segmentation student model for the sample image, where U and V are unequal positive integers; Align the U first predicted instances and the V second predicted instances to the L real instances of the sample image to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance, where L, M, and N are positive integers; Instance distillation loss is calculated based on the differences between the M first prediction instances and the N second prediction instances; The distillation loss is calculated based on the distillation loss of the L instances corresponding to the L real instances; The model parameters of the instance segmentation student model are adjusted based on the distillation loss results.

2. The method according to claim 1, characterized in that, The step of aligning the U first predicted instances and the V second predicted instances to L real instances of the sample image to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance includes: For each of the L real instances, perform the following processing: select K first predicted instances that are initially aligned with the real instance from the U first predicted instances, and select K second predicted instances that are initially aligned with the real instance from the V second predicted instances, where K is a positive integer; Based on the K first predicted instances initially aligned to each of the L real instances, the first predicted instances are aligned and deduplicated to obtain the M first predicted instances aligned to each of the real instances. Based on the K second predicted instances initially aligned to each of the L real instances, the second predicted instances are aligned and deduplicated to obtain the N second predicted instances aligned to each of the real instances.

3. The method according to claim 2, characterized in that, The step of selecting K first predicted instances that are initially aligned with the real instance from the U first predicted instances includes: From the U first prediction instances, select the first prediction instance whose center point of the prediction detection box is within the real detection box of the real instance; Calculate the intersection-union ratio (IoU) between the predicted detection box and the ground truth detection box for each selected first prediction instance; The intersection-union ratio and the classification score of the first predicted instance are integrated to obtain the alignment index value; Based on the alignment index value, select K first prediction instances that are initially aligned with the real instance from among the selected plurality of first prediction instances.

4. The method according to claim 2, characterized in that, The process of aligning and deduplicating the K first predicted instances initially aligned to each of the L real instances to obtain the M first predicted instances aligned to each of the real instances includes: Based on the K first predicted instances initially aligned to each of the L real instances, determine the first predicted instances initially aligned to the multiple real instances; Calculate the full intersection-union ratio of the first predicted instance that is initially aligned with the plurality of said real instances; Based on the complete intersection-union ratio, the real instance aligned to the first predicted instance is determined from the plurality of real instances initially aligned to the first predicted instance; Based on the real instances aligned with the first predicted instances, the M first predicted instances aligned with each of the real instances are determined.

5. The method according to claim 1, characterized in that, The calculation of instance distillation loss based on the differences between the M first predicted instances and the N second predicted instances includes: For each of the M first prediction instances and each of the N second prediction instances, the following processing is performed: a classification distillation loss is calculated based on the difference between the predicted category of the first prediction instance and the predicted category of the second prediction instance; a detection distillation loss is calculated based on the difference between the predicted detection box of the first prediction instance and the predicted detection box of the second prediction instance; and a segmentation distillation loss is calculated based on the difference between the predicted segmentation mask of the first prediction instance and the predicted segmentation mask of the second prediction instance. The instance distillation loss is obtained by integrating the M*N classification distillation losses, the M*N detection distillation losses, and the M*N segmentation distillation losses.

6. The method according to any one of claims 1 to 5, characterized in that, The adjustment of the model parameters of the instance segmentation student model based on the distillation loss of the results includes: Obtain the first features at I scales output by the instance segmentation student model for the sample image, and obtain the second features at J scales output by the instance segmentation teacher model for the sample image, where I and J are unequal positive integers; Align the second features at the J scales to the first features at the I scales to obtain the third features at the I scales; Based on the difference between the first feature at the I scale and the third feature at the I scale, the feature distillation loss is calculated; The model parameters of the instance segmentation student model are adjusted based on the feature distillation loss and the result distillation loss.

7. The method according to claim 6, characterized in that, Aligning the second features at the J scales to the first features at the I scales to obtain the third features at the I scale includes: The second feature is upsampled starting from the lowest of the J scales to obtain the p-th upsampled feature, where p is a variable representing the number of upsampling times; The second feature with the same scale as the p-th upsampled feature among the J-scale second features, along with the p-th upsampled feature, are integrated into a fourth feature; When upsampling is performed to the highest of the I scales, the fourth feature of the I scales is obtained; Downsampling is performed starting from the fused features of the second features at least one scale higher than the highest scale of the I scales in the J scales to obtain the q-th downsampled feature, where q is a variable representing the number of downsampling times; The fourth feature with the same scale as the q-th downsampled feature in the I-scale and the q-th downsampled feature are integrated into the third feature; When sampling down to the lowest of the I scales, the third feature of the I scales is obtained.

8. The method according to claim 6, characterized in that, The adjustment of the model parameters of the instance segmentation student model based on the feature distillation loss and the result distillation loss includes: Based on the first feature and the third feature at the I scale corresponding to the sample image, determine the first feature at the I scale and the third feature at the I scale corresponding to the multiple sample images; Calculate the first feature similarity among the first features of the I scales corresponding to the multiple sample images; Calculate the second feature similarity among the third features at multiple I scales corresponding to the multiple sample images; Based on the difference between the first feature similarity and the first feature similarity, the feature relationship distillation loss is calculated; The model parameters of the instance segmentation student model are adjusted based on the feature relation distillation loss, the feature distillation loss, and the result distillation loss.

9. A knowledge distillation apparatus for instance segmentation, characterized in that, The knowledge distillation apparatus includes: The output acquisition module is used to acquire U first predicted instances output by the instance segmentation teacher model for the sample image, and to acquire V second predicted instances output by the instance segmentation student model for the sample image, where U and V are unequal positive integers. The instance alignment module is used to align the U first predicted instances and the V second predicted instances to L real instances of the sample image, so as to obtain M first predicted instances aligned with each real instance and N second predicted instances aligned with each real instance, where L, M and N are positive integers. The loss acquisition module is used to calculate the instance distillation loss based on the difference between the M first prediction instances and the N second prediction instances; The loss acquisition module is also used to calculate the result distillation loss based on the distillation losses of the L instances corresponding to the L real instances; The parameter adjustment module is used to adjust the model parameters of the instance segmentation student model based on the distillation loss of the results.

10. An electronic device for knowledge distillation, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the knowledge distillation method for instance segmentation as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the knowledge distillation method for instance segmentation as described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the knowledge distillation method for instance segmentation as described in any one of claims 1 to 8 is implemented.

Images