Image processing model training method, device, apparatus, medium and program product

Abstract

The present disclosure relates to a method, apparatus, device, medium and program product for training an image processing model. The method comprises: obtaining a first feature map and a second feature map of a training image, wherein the first feature map is output by a first image processing model, and the second feature map is output by a second image processing model; generating a third feature map according to part of the features in the first feature map and part of the features in the second feature map; determining a value of a loss function corresponding to the second image processing model according to the third feature map and the first feature map; and training the second image processing model according to the value of the loss function.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method, apparatus, electronic device, storage medium, and program product for an image processing model. Background Technology

[0002] Knowledge distillation (KD) refers to the process of extracting knowledge from a pre-trained teacher model and incorporating it into a student model. Common knowledge distillation methods utilize the logits output by the teacher model as the knowledge learned by the student model; this knowledge is often referred to as "soft labels" or "dark knowledge." Related techniques have also proposed feature-based knowledge distillation methods, which use the feature maps output from the intermediate layers of the teacher model as the knowledge learned by the student model.

[0003] In related technologies, improving the performance of the teacher model does not necessarily lead to an improvement in the performance of the distilled student model. That is, there is a gap between the teacher and student models (i.e., the teacher model and the student model), and performance improvements in the student model face a bottleneck; after reaching a certain level, it is difficult to continue optimizing.

[0004] For mobile devices lacking computing power, only small-scale student models can typically be deployed. In related technologies, the poor performance of student models hinders their deployment, and low-precision student models often fail to meet user needs. If the gap between teacher and student models can be bridged, the performance of student models will improve along with the performance of teacher models, accelerating the deployment of more student models and providing users with a more comfortable experience. Summary of the Invention

[0005] This disclosure provides a training technique for an image processing model.

[0006] According to one aspect of this disclosure, a method for training an image processing model is provided, comprising:

[0007] Obtain a first feature map and a second feature map of the training image, wherein the first feature map is output by a first image processing model and the second feature map is output by a second image processing model;

[0008] A third feature map is generated based on some features in the first feature map and some features in the second feature map;

[0009] Based on the third feature map and the first feature map, determine the value of the loss function corresponding to the second image processing model;

[0010] The second image processing model is trained based on the value of the loss function.

[0011] By acquiring a first feature map and a second feature map of the training image, where the first feature map is output by a first image processing model and the second feature map is output by a second image processing model, a third feature map is generated based on some features from the first and second feature maps. Based on the third and first feature maps, the value of the loss function corresponding to the second image processing model is determined, and the second image processing model is trained based on the value of the loss function. Thus, a portion of the feature map extracted by the first image processing model is used as prior knowledge for the second image processing model, enabling the second image processing model to mimic the features output by the first image processing model. This allows the second image processing model to improve along with the performance of the first image processing model, thereby bridging the gap between teacher-student models and addressing the issue that the performance of the second image processing model no longer improves as the performance of the first model improves. Consequently, a larger-scale, higher-performance first image processing model can be used to distill a higher-performance second image processing model.

[0012] In one possible implementation, generating a third feature map based on partial features from the first feature map and partial features from the second feature map includes:

[0013] From the first feature map, a first set of feature blocks for generating a third feature map is determined, and from the second feature map, a second set of feature blocks for generating the third feature map is determined, wherein the first set of feature blocks represents a set of feature blocks in the first feature map used to generate the third feature map, and the first set of feature blocks includes some features of the first feature map; the second set of feature blocks represents a set of feature blocks in the second feature map used to generate the third feature map, and the second set of feature blocks includes some features of the second feature map.

[0014] The third feature map is generated based on the first feature block set and the second feature block set.

[0015] In this implementation, a first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map. The third feature map is then generated based on the first and second feature block sets. This divides the first and second feature maps into smaller units using feature blocks as the smallest unit, and generates the third feature map based on some feature blocks in the first and second feature maps. By using the prior feature blocks provided by the teacher model, the second image processing model can be trained, which helps to improve the accuracy of the trained second image processing model.

[0016] In one possible implementation, determining a first set of feature blocks from the first feature map for generating a third feature map, and determining a second set of feature blocks from the second feature map for generating the third feature map, includes:

[0017] Based on the first feature map and the second feature map, determine the mask ratio used to merge the first feature map and the second feature map;

[0018] Determine the mask area based on the mask ratio;

[0019] Based on the mask region, a first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map, wherein the positions of the feature blocks in the first set of feature blocks and the second set of feature blocks are complementary.

[0020] In this implementation, a mask ratio for merging the first and second feature maps is determined based on the first and second feature maps. A mask region is then determined based on the mask ratio. A first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map. The feature blocks in the first and second feature block sets are complementary in position. Therefore, the mask ratio for merging the first and second feature maps is determined based on the first and second feature maps, rather than using a fixed mask ratio. That is, the mask ratio for merging feature map pairs is dynamically adjusted based on the similarity information (e.g., similarity) of the feature map pairs. For example, different intermediate layer pairs output different feature map pairs, and their similarity information is also different, which may lead to the use of different mask ratios. Similarly, the same intermediate layer pair may output different feature map pairs in different training epochs, which may also lead to the use of different mask ratios. By using a mask ratio dynamically determined based on the feature map pairs, it is beneficial to reduce the gap between the second image processing model and the first image processing model, and to improve the accuracy of the trained second image processing model.

[0021] In one possible implementation, determining the mask ratio for merging the first feature map and the second feature map based on the first feature map and the second feature map includes:

[0022] Determine the similarity information between the first feature map and the second feature map;

[0023] Based on the similarity information, a mask ratio for merging the first feature map and the second feature map is determined.

[0024] In this implementation, by determining the similarity information between the first feature map and the second feature map, and based on the similarity information, determining the mask ratio for merging the first feature map and the second feature map, the accuracy of the trained second image processing model is improved.

[0025] In one possible implementation,

[0026] Determining the similarity information between the first feature map and the second feature map includes: determining the kernel alignment (CKA) similarity index between the first feature map and the second feature map;

[0027] The step of determining the mask ratio for merging the first feature map and the second feature map based on the similarity information includes: determining the mask ratio for merging the first feature map and the second feature map based on the CKA similarity index.

[0028] In this implementation, by determining the CKA similarity index between the first feature map and the second feature map, and based on the CKA similarity index, determining the mask ratio for merging the first feature map and the second feature map, the accuracy of the trained second image processing model is improved.

[0029] In one possible implementation, determining the similarity information between the first feature map and the second feature map includes:

[0030] Align the first feature map with the second feature map;

[0031] Determine the similarity information between the aligned first feature map and the second feature map.

[0032] In this implementation, by aligning the first feature map and the second feature map and determining the similarity information between the aligned first feature map and the second feature map, the similarity information between the first feature map and the second feature map can be determined more accurately.

[0033] In one possible implementation, aligning the first feature map with the second feature map includes:

[0034] In response to the fact that the number of channels of the second feature map is different from that of the first feature map, the second feature map is convolved so that the number of channels of the convolved second feature map is the same as that of the first feature map, and / or, in response to the fact that the size of the second feature map is different from that of the first feature map, the second feature map is bilinearly interpolated so that the size of the bilinearly interpolated second feature map is the same as that of the first feature map.

[0035] or,

[0036] In response to the second feature map having a different number of channels than the first feature map, the first feature map is convolved to make the number of channels in the convolved first feature map the same as that in the second feature map, and / or, in response to the second feature map having a different size than the first feature map, the first feature map is bilinearly interpolated to make the size of the bilinearly interpolated first feature map the same as that of the second feature map.

[0037] In this implementation, in response to the difference in the number of channels between the second feature map and the first feature map, convolution processing is performed on the second feature map, making the number of channels in the convolution-processed second feature map the same as that in the first feature map, thereby aligning the number of channels in the first and second feature maps; in response to the difference in size between the second feature map and the first feature map, bilinear interpolation is performed on the second feature map, making the size of the bilinearly interpolated second feature map the same as that in the first feature map, thereby aligning the size of the first and second feature maps; in response to the difference in the number of channels between the second feature map and the first feature map, convolution processing is performed on the first feature map, making the number of channels in the convolution-processed first feature map the same as that in the second feature map, thereby aligning the number of channels in the first and second feature maps; in response to the difference in size between the second feature map and the first feature map, bilinear interpolation is performed on the first feature map, making the size of the bilinearly interpolated first feature map the same as that in the second feature map, thereby aligning the size of the first and second feature maps.

[0038] In one possible implementation, determining a first set of feature blocks for generating a third feature map from the first feature map based on the mask region, and determining a second set of feature blocks for generating the third feature map from the second feature map, includes:

[0039] Based on the location information of the masked area or the location information outside the masked area, a first set of feature blocks for generating the third feature map is determined from the first feature map;

[0040] From the second feature map, feature blocks whose positions are complementary to those of the feature blocks in the first feature block set are selected to obtain a second feature block set used to generate the third feature map.

[0041] In this implementation, a first set of feature blocks for generating a third feature map is determined from the first feature map by using the position information corresponding to the masked area or the position information outside the masked area. Then, feature blocks that are complementary in position to feature blocks in the first feature block set are selected from the second feature map to obtain a second set of feature blocks for generating the third feature map. Thus, the masked feature blocks in the second feature map are filled by feature blocks at corresponding positions in the first feature map, thereby enabling the first image processing model, which acts as the teacher model, to provide some prior knowledge to the second image processing model, which acts as the student model.

[0042] In one possible implementation, generating the third feature map based on the first feature block set and the second feature block set includes:

[0043] Position encoding is performed on the first feature block set and the second feature block set respectively to obtain the first position information corresponding to the first feature block set and the second position information corresponding to the second feature block set;

[0044] By combining the first location information, the first feature block set is encoded to obtain the first feature block encoding result;

[0045] By combining the second position information, the second feature block set is encoded to obtain the second feature block encoding result;

[0046] A third feature map is generated based on the encoding results of the first and second feature blocks.

[0047] In this implementation, positional encoding is performed on the first feature block set and the second feature block set respectively to obtain first positional information corresponding to the first feature block set and second positional information corresponding to the second feature block set. The first feature block set is then encoded in combination with the first positional information to obtain a first feature block encoding result. The second feature block set is then encoded in combination with the second positional information to obtain a second feature block encoding result. A third feature map is generated based on the first and second feature block encoding results. By combining the positional information of the feature blocks in the first and second feature block sets to generate the third feature map, the third feature map can contain the positional information of the feature blocks in the first and second feature maps. This allows the prior features containing positional information provided by the teacher model to be used to train the second image processing model, which helps to improve the accuracy of the trained second image processing model.

[0048] In one possible implementation, determining the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map includes:

[0049] The third feature map is positionally encoded to obtain the third position information corresponding to the third feature map;

[0050] The third location information and the third feature map are input into the decoding network to obtain the fourth feature map;

[0051] The value of the loss function corresponding to the second image processing model is determined based on the fourth feature map and the first feature map.

[0052] In this implementation, the third feature map is positionally encoded to obtain the third position information corresponding to the third feature map. The third feature map with the added third position information is then input into the decoding network to obtain the fourth feature map. Based on the fourth feature map and the first feature map, the value of the loss function corresponding to the second image processing model is determined, thereby training the second image processing model and improving its accuracy.

[0053] In one possible implementation,

[0054] The step of obtaining the first feature map and the second feature map of the training image includes: obtaining at least two first feature maps corresponding to the training image extracted by at least two first intermediate layers of the first image processing model, and at least two second feature maps corresponding to the training image extracted by at least two second intermediate layers of the second image processing model, wherein the at least two first feature maps and the at least two second feature maps form at least two feature map pairs, and any feature map pair in the at least two feature map pairs includes a first feature map and a second feature map;

[0055] The step of generating a third feature map based on partial features in the first feature map and partial features in the second feature map includes: for any feature map pair among the at least two feature map pairs, generating a third feature map corresponding to the feature map pair based on partial features in the first feature map and partial features in the second feature map of the feature map pair;

[0056] Determining the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map includes: determining the value of the loss function corresponding to the second image processing model based on the third feature map corresponding to the at least two feature map pairs and the first feature map in the at least two feature map pairs.

[0057] In this implementation, the second image processing model is trained by using at least two feature map pairs output by at least two intermediate layers of the first and second image processing models. This enables the second image processing model to learn richer hidden layer features from the first image processing model, thereby further reducing the gap between the second and first image processing models.

[0058] In one possible implementation, both the first image processing model and the second image processing model are used for image classification;

[0059] After the second image processing model has been trained, the method further includes:

[0060] Obtain the image to be classified;

[0061] The image to be classified is processed by the second image processing model to obtain the feature map corresponding to the image to be classified;

[0062] The second image processing model processes the feature map corresponding to the image to be classified to obtain the classification result corresponding to the image to be classified.

[0063] In this implementation, the image to be classified is processed by the trained second image processing model to obtain the feature map corresponding to the image to be classified, and the feature map corresponding to the image to be classified is processed by the second image processing model to obtain the classification result corresponding to the image to be classified, thereby improving the accuracy of image classification of the image to be classified.

[0064] In one possible implementation, both the first image processing model and the second image processing model are used for target detection;

[0065] After the second image processing model has been trained, the method further includes:

[0066] Acquire the image to be detected;

[0067] The image to be detected is processed by the second image processing model to obtain the feature map corresponding to the image to be detected;

[0068] The second image processing model is used to process the feature map corresponding to the image to be detected, thereby obtaining the target detection result corresponding to the image to be detected.

[0069] In this implementation, the image to be detected is processed by the trained second image processing model to obtain the feature map corresponding to the image to be detected. The feature map corresponding to the image to be detected is then processed by the second image processing model to obtain the target detection result corresponding to the image to be detected, thereby improving the accuracy of target detection in the image to be detected.

[0070] According to one aspect of this disclosure, a training apparatus for an image processing model is provided, comprising:

[0071] The acquisition module is used to acquire a first feature map and a second feature map of the training image, wherein the first feature map is output by a first image processing model and the second feature map is output by a second image processing model.

[0072] A generation module is used to generate a third feature map based on some features in the first feature map and some features in the second feature map;

[0073] The determining module is used to determine the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map;

[0074] The training module is used to train the second image processing model based on the value of the loss function.

[0075] In one possible implementation, the generation module is used to:

[0076] From the first feature map, a first set of feature blocks for generating a third feature map is determined, and from the second feature map, a second set of feature blocks for generating the third feature map is determined, wherein the first set of feature blocks represents a set of feature blocks in the first feature map used to generate the third feature map, and the first set of feature blocks includes some features of the first feature map; the second set of feature blocks represents a set of feature blocks in the second feature map used to generate the third feature map, and the second set of feature blocks includes some features of the second feature map.

[0077] The third feature map is generated based on the first feature block set and the second feature block set.

[0078] In one possible implementation, the generation module is used to:

[0079] Based on the first feature map and the second feature map, determine the mask ratio used to merge the first feature map and the second feature map;

[0080] Determine the mask area based on the mask ratio;

[0081] Based on the mask region, a first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map, wherein the positions of the feature blocks in the first set of feature blocks and the second set of feature blocks are complementary.

[0082] In one possible implementation, the generation module is used to:

[0083] Determine the similarity information between the first feature map and the second feature map;

[0084] Based on the similarity information, a mask ratio for merging the first feature map and the second feature map is determined.

[0085] In one possible implementation, the generation module is used to:

[0086] Determine the intermediate kernel alignment CKA similarity index between the first feature map and the second feature map;

[0087] Based on the CKA similarity index, the mask ratio used to merge the first feature map and the second feature map is determined.

[0088] In one possible implementation, the generation module is used to:

[0089] Align the first feature map with the second feature map;

[0090] Determine the similarity information between the aligned first feature map and the second feature map.

[0091] In one possible implementation, the generation module is used to:

[0092] In response to the fact that the number of channels of the second feature map is different from that of the first feature map, the second feature map is convolved so that the number of channels of the convolved second feature map is the same as that of the first feature map, and / or, in response to the fact that the size of the second feature map is different from that of the first feature map, the second feature map is bilinearly interpolated so that the size of the bilinearly interpolated second feature map is the same as that of the first feature map.

[0093] or,

[0094] In response to the second feature map having a different number of channels than the first feature map, the first feature map is convolved to make the number of channels in the convolved first feature map the same as that in the second feature map, and / or, in response to the second feature map having a different size than the first feature map, the first feature map is bilinearly interpolated to make the size of the bilinearly interpolated first feature map the same as that of the second feature map.

[0095] In one possible implementation, the determining module is used to:

[0096] Based on the location information of the masked area or the location information outside the masked area, a first set of feature blocks for generating the third feature map is determined from the first feature map;

[0097] From the second feature map, feature blocks whose positions are complementary to those of the feature blocks in the first feature block set are selected to obtain a second feature block set used to generate the third feature map.

[0098] In one possible implementation, the generation module is used to:

[0099] Position encoding is performed on the first feature block set and the second feature block set respectively to obtain the first position information corresponding to the first feature block set and the second position information corresponding to the second feature block set;

[0100] By combining the first location information, the first feature block set is encoded to obtain the first feature block encoding result;

[0101] By combining the second position information, the second feature block set is encoded to obtain the second feature block encoding result;

[0102] A third feature map is generated based on the encoding results of the first and second feature blocks.

[0103] In one possible implementation, the determining module is used to:

[0104] The third feature map is positionally encoded to obtain the third position information corresponding to the third feature map;

[0105] The third location information and the third feature map are input into the decoding network to obtain the fourth feature map;

[0106] The value of the loss function corresponding to the second image processing model is determined based on the fourth feature map and the first feature map.

[0107] In one possible implementation,

[0108] The acquisition module is used to: obtain at least two first feature maps corresponding to the training image extracted by at least two first intermediate layers of the first image processing model, and at least two second feature maps corresponding to the training image extracted by at least two second intermediate layers of the second image processing model, wherein the at least two first feature maps and the at least two second feature maps form at least two feature map pairs, and any feature map pair in the at least two feature map pairs includes a first feature map and a second feature map;

[0109] The generation module is used to: for any of the at least two feature map pairs, generate a third feature map corresponding to the feature map pair based on some features in the first feature map of the feature map pair and some features in the second feature map of the feature map pair;

[0110] The determining module is used to: determine the value of the loss function corresponding to the second image processing model based on the third feature map corresponding to the at least two feature map pairs and the first feature map in the at least two feature map pairs.

[0111] In one possible implementation, both the first image processing model and the second image processing model are used for image classification;

[0112] The device further includes:

[0113] The classification module is used to acquire an image to be classified; process the image to be classified using the second image processing model to obtain a feature map corresponding to the image to be classified; and process the feature map corresponding to the image to be classified using the second image processing model to obtain a classification result corresponding to the image to be classified.

[0114] In one possible implementation, both the first image processing model and the second image processing model are used for target detection;

[0115] The device further includes:

[0116] The target detection module is used to acquire an image to be detected; process the image to be detected using the second image processing model to obtain a feature map corresponding to the image to be detected; and process the feature map corresponding to the image to be detected using the second image processing model to obtain a target detection result corresponding to the image to be detected.

[0117] According to one aspect of this disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the executable instructions stored in the memory to perform the method described above.

[0118] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0119] According to one aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device performs the above-described method.

[0120] In this embodiment, a first feature map and a second feature map of the training image are obtained. The first feature map is output by a first image processing model, and the second feature map is output by a second image processing model. A third feature map is generated based on some features in the first feature map and some features in the second feature map. The value of the loss function corresponding to the second image processing model is determined based on the third feature map and the first feature map. The second image processing model is trained based on the value of the loss function. Thus, a portion of the feature map extracted by the first image processing model is used as prior knowledge for the second image processing model, enabling the second image processing model to mimic the features output by the first image processing model. This allows the second image processing model to improve as the performance of the first image processing model improves. In other words, it can solve the gap between teacher and student models and the problem that the performance of the second image model no longer improves as the performance of the first image model improves. Therefore, it is possible to use a larger-scale, higher-performance first image processing model to distill a higher-performance second image processing model.

[0121] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0122] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0123] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0124] Figure 1 A flowchart illustrating the training method of the image processing model provided in the embodiments of this disclosure is shown.

[0125] Figure 2 A block diagram of a training apparatus for an image processing model provided in an embodiment of this disclosure is shown.

[0126] Figure 3 A block diagram of a sub-device 1900 provided in an embodiment of this disclosure is shown. Detailed Implementation

[0127] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0128] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0129] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0130] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0131] This disclosure provides a training method, apparatus, electronic device, storage medium, and program product for an image processing model. It involves acquiring a first feature map and a second feature map of a training image. The first feature map is output by a first image processing model, and the second feature map is output by a second image processing model. A third feature map is generated based on partial features from the first and second feature maps. The value of a loss function corresponding to the second image processing model is determined based on the third and first feature maps. The second image processing model is then trained based on the value of the loss function. This utilizes a portion of the feature map extracted by the first image processing model as prior knowledge for the second image processing model, enabling the second model to mimic the features output by the first model. This allows the second image processing model to improve along with the performance of the first model, thus bridging the gap between teacher-student models and addressing the issue that the performance of the second model no longer improves as the performance of the first model improves. Furthermore, it allows the use of a larger-scale, higher-performance first image processing model to distill a higher-performance second image processing model.

[0132] The training method of the image processing model provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0133] Figure 1A flowchart illustrating a training method for an image processing model provided in an embodiment of this disclosure is shown. In one possible implementation, the entity executing the image processing model training method may be an image processing model training device. For example, the image processing model training method may be executed by a terminal device, a server, or other electronic equipment. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the image processing model training method may be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the training method of the image processing model includes steps S11 to S14.

[0134] In step S11, a first feature map and a second feature map of the training image are obtained, wherein the first feature map is output by a first image processing model and the second feature map is output by a second image processing model.

[0135] In step S12, a third feature map is generated based on some features in the first feature map and some features in the second feature map.

[0136] In step S13, the value of the loss function corresponding to the second image processing model is determined based on the third feature map and the first feature map.

[0137] In step S14, the second image processing model is trained based on the value of the loss function.

[0138] In the embodiments of this disclosure, both the first image processing model and the second image processing model are models for image processing, and both can be used for the same image processing task. For example, both the first image processing model and the second image processing model are used for image classification; or both are used for object detection; or both are used for image segmentation; or both are used for feature extraction; and so on.

[0139] In this embodiment, the first image processing model is a teacher model, and the second image processing model is a student model. The second image processing model is lighter than the first image processing model. For example, compared to the first image processing model, the second image processing model has a simpler network structure and / or fewer parameters. That is, compared to the second image processing model, the first image processing model has a more complex network structure and / or more parameters. Because the first image processing model has a more complex network structure and / or more parameters, it can usually be trained to achieve better performance. As the scale of the first image processing model increases, its performance will improve. Since this embodiment uses some features of the feature map extracted by the first image processing model as prior knowledge for the second image processing model, enabling the second image processing model to mimic the features output by the first image processing model, this embodiment can make the second image processing model improve as the performance of the first image processing model improves. That is, it can bridge the gap between the teacher and student models and solve the problem that the performance of the second image model no longer improves as the performance of the first image model improves. Since the performance of the second image processing model can be improved as the performance of the first image processing model is improved, a larger-scale, better-performing first image processing model can be used to distill a better-performing second image processing model.

[0140] This disclosure does not limit the network structure used by the first image processing model and the second image processing model. The first image processing model and the second image processing model may use the same type of network structure or different types of network structures.

[0141] In one possible implementation, both the first image processing model and the second image processing model are used for image classification.

[0142] As an example of this implementation, the first image processing model and the second image processing model can use the same type of network structure. For example, the first image processing model can use ResNet152, and the second image processing model can use ResNet18; or the first image processing model can use WRN40-2, and the second image processing model can use WRN16-2; or the first image processing model can use WRN40-2, and the second image processing model can use WRN40-1; or the first image processing model can use ResNet56, and the second image processing model can use ResNet20; or the first image processing model can use ResNet110, and the second image processing model can use ResNet20; or the first image processing model can use ResNet110, and the second image processing model can use ResNet32; or the first image processing model can use VGG13, and the second image processing model can use VGG8; or the first image processing model can use ResNet34, and the second image processing model can use ResNet18; and so on.

[0143] As another example of this implementation, the first and second image processing models can employ different types of network architectures. For instance, the first image processing model could use ResNet50, and the second image processing model could use MobileNetV2.

[0144] In another possible implementation, both the first and second image processing models are used for object detection. As an example of this implementation, the first and second image processing models can each employ RetinaNet. As another example, the first and second image processing models can each employ Faster-RCNN. In this implementation, the first and second image processing models can use the same type of network architecture or different types of network architectures. For example, the first image processing model can use ResNet101-FPN, and the second image processing model can use ResNet50-FPN; or the first image processing model can use ResNet152-FPN, and the second image processing model can use ResNet50-FPN; the first image processing model can use ResNet101-FPN, and the second image processing model can use ResNet18-FPN; and so on.

[0145] In this embodiment of the disclosure, the first feature map can be a feature map output by a first network layer of a first image processing model, and the second feature map can be a feature map output by a second network layer of a second image processing model, wherein the first network layer and the second network layer are corresponding network layers in the first image processing model and the second image processing model. The number of first network layers can be one or more, and correspondingly, the number of first feature maps can also be one or more; the number of second network layers can also be one or more, and correspondingly, the number of second feature maps can also be one or more.

[0146] In one possible implementation, the first network layer may include an intermediate layer of the first image processing model, and the second network layer may include an intermediate layer of the second image processing model. For example, the first network layer may include a first intermediate layer of the first image processing model, and the second network layer may include a second intermediate layer of the second image processing model. The first and second intermediate layers are corresponding intermediate layers in the first and second image processing models, and they can form an intermediate layer pair. Accordingly, the first feature map output by the first intermediate layer and the second feature map output by the second intermediate layer can form a feature map pair. As an example of this implementation, the first intermediate layer may be the last layer before the downsampling layer in the first image processing model, and the second intermediate layer may be the last layer before the downsampling layer in the second image processing model. Of course, those skilled in the art can flexibly choose the first and second intermediate layers according to the actual application scenario requirements, and no limitation is made here.

[0147] In this implementation, T intermediate layer pairs can be selected from the first image processing model and the second image processing model. The second image processing model can be trained based on the T feature map pairs output from these T intermediate layer pairs, where T is an integer greater than or equal to 1. As an example of this implementation, the T first feature maps output from the last layer before the last T downsampling layers of the first image processing model can be obtained. Obtain the T second feature maps from the output of the last layer before the last T downsampling layers of the second image processing model. And form T feature map pairs.

[0148] As an example of this implementation, two intermediate layer pairs can be selected from the first image processing model and the second image processing model. These two intermediate layer pairs include: a first intermediate layer pair consisting of the last layer before the last downsampling layer of the first image processing model and the last layer before the last downsampling layer of the second image processing model; and a second intermediate layer pair consisting of the last layer before the second-to-last downsampling layer of the first image processing model and the last layer before the second-to-last downsampling layer of the second image processing model. The first feature map output by the last layer before the last downsampling layer of the first image processing model can be denoted as... The second feature map output from the last layer before the last downsampling layer of the second image processing model can be denoted as: The first feature map pair corresponding to the first intermediate layer can be denoted as: The first feature map output from the last layer before the penultimate downsampling layer of the first image processing model can be denoted as: The second feature map output from the last layer before the penultimate downsampling layer of the second image processing model can be denoted as... The second feature map pair corresponding to the second intermediate layer can be denoted as...

[0149] As another example of this implementation, three intermediate layer pairs can be selected from the first image processing model and the second image processing model. These three intermediate layer pairs include: a first intermediate layer pair consisting of the last layer before the last downsampling layer of the first image processing model and the last layer before the last downsampling layer of the second image processing model; a second intermediate layer pair consisting of the last layer before the second-to-last downsampling layer of the first image processing model and the last layer before the second-to-last downsampling layer of the second image processing model; and a third intermediate layer pair consisting of the last layer before the third-to-last downsampling layer of the first image processing model and the last layer before the third-to-last downsampling layer of the second image processing model. The first feature map output by the last layer before the last downsampling layer of the first image processing model can be denoted as... The second feature map output from the last layer before the last downsampling layer of the second image processing model can be denoted as: The first feature map pair corresponding to the first intermediate layer can be denoted as: The first feature map output from the last layer before the penultimate downsampling layer of the first image processing model can be denoted as: The second feature map output from the last layer before the penultimate downsampling layer of the second image processing model can be denoted as... The second feature map pair corresponding to the second intermediate layer can be denoted as... The first feature map output from the last layer before the third-to-last downsampling layer of the first image processing model can be denoted as... The second feature map output from the last layer before the third-to-last downsampling layer of the second image processing model can be denoted as... The third feature map pair corresponding to the third intermediate layer can be denoted as...

[0150] As another example of this implementation, one intermediate layer pair can be selected from the first image processing model and the second image processing model. This intermediate layer pair consists of the last layer before the last downsampling layer of the first image processing model and the last layer before the last downsampling layer of the second image processing model. The first feature map output by the last layer before the last downsampling layer of the first image processing model can be denoted as... The second feature map output from the last layer before the last downsampling layer of the second image processing model can be denoted as: The first feature map pair corresponding to the first intermediate layer can be denoted as:

[0151] In another possible implementation, the first network layer may include the output layer of the first image processing model, and the second network layer may include the output layer of the second image processing model. In this implementation, both the first and second image processing models can be used for feature extraction; that is, the outputs of both the first and second image processing models can be feature maps.

[0152] In another possible implementation, the first network layer may include intermediate layers and an output layer of the first image processing model, and the second network layer may include intermediate layers and an output layer of the second image processing model. In this implementation, both the first and second image processing models can be used for feature extraction; that is, the outputs of both the first and second image processing models can be feature maps.

[0153] In one possible implementation, generating a third feature map based on partial features in the first feature map and partial features in the second feature map includes: determining a first set of feature blocks for generating the third feature map from the first feature map, and determining a second set of feature blocks for generating the third feature map from the second feature map, wherein the first set of feature blocks represents a set of feature blocks in the first feature map used to generate the third feature map, and the first set of feature blocks includes partial features of the first feature map; the second set of feature blocks represents a set of feature blocks in the second feature map used to generate the third feature map, and the second set of feature blocks includes partial features of the second feature map; and generating the third feature map based on the first set of feature blocks and the second set of feature blocks.

[0154] In this implementation, a feature block can represent an image block obtained by partitioning a feature map. Partitioning the first feature map yields a set of feature blocks corresponding to the first feature map, where any two feature blocks in the first feature map's feature block set do not overlap. Similarly, partitioning the second feature map yields a set of feature blocks corresponding to the second feature map, where any two feature blocks in the second feature map's feature block set do not overlap. The number of channels in a feature block can be the same as the number of channels in the feature map. For example, if the first feature map has C1 channels, then any feature block in the set of feature blocks corresponding to the first feature map can also have C1 channels; similarly, if the second feature map has C2 channels, then any feature block in the set of feature blocks corresponding to the second feature map can also have C2 channels.

[0155] In one example, the first feature map can be... Second feature map Divided into sizes P i ×P i The feature blocks. For example, the first feature map. Second feature map All dimensions are H i ×W i Therefore, the first feature map can be... Second feature map Divided into M i There are feature blocks, where M i =(H i ×W i ) / (P i ×P i For example, the first feature map. Second feature map The number of channels is C. i Therefore, the size of the resulting feature block can be P. i×P i The number of channels can be C i In one example, a convolution kernel size of P can be used. i ×P i Step size is P i The convolutional layer, for the first feature map Second feature map Divide them separately.

[0156] In this implementation, a subset of feature blocks can be selected from the feature block set corresponding to the first feature map to form a first feature block set for generating the third feature map; similarly, a subset of feature blocks can be selected from the feature block set corresponding to the second feature map to form a second feature block set for generating the third feature map. The feature blocks in the first and second feature block sets are complementary in position. After obtaining the first and second feature block sets, they can be processed to generate the third feature map. For example, a pre-defined network's first and second feature block sets can be used for processing to generate the third feature map. Alternatively, the first and second feature block sets can be concatenated to generate the third feature map.

[0157] In this implementation, a first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map. The third feature map is then generated based on the first and second feature block sets. This divides the first and second feature maps into smaller units using feature blocks as the smallest unit, and generates the third feature map based on some feature blocks in the first and second feature maps. By using the prior feature blocks provided by the teacher model, the second image processing model can be trained, which helps to improve the accuracy of the trained second image processing model.

[0158] As an example of this implementation, determining a first set of feature blocks for generating a third feature map from the first feature map and a second set of feature blocks for generating the third feature map from the second feature map includes: determining a mask ratio for merging the first and second feature maps based on the first and second feature maps; determining a mask region based on the mask ratio; determining a first set of feature blocks for generating the third feature map from the first feature map and a second set of feature blocks for generating the third feature map from the second feature map based on the mask region, wherein the positions of the feature blocks in the first and second feature map sets are complementary.

[0159] In one example, the first feature map Second feature map Each includes M i There are 1 feature block, with a mask ratio of α. i Therefore, the mask region can include M. i ×α i Each feature block.

[0160] In this example, the mask area can be randomly determined based on the mask ratio. Alternatively, the mask area can be determined based on the mask ratio and a preset mask rule.

[0161] In this example, a mask ratio for merging the first and second feature maps is determined based on the first and second feature maps. A mask region is then determined based on this mask ratio. A first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map. The feature blocks in the first and second feature map sets are complementary in position. Therefore, the mask ratio for merging the first and second feature maps is determined based on both the first and second feature maps, rather than using a fixed mask ratio. That is, the mask ratio for merging feature map pairs is dynamically adjusted based on the similarity information (e.g., similarity) of the feature map pairs. For example, different intermediate layer pairs output different feature map pairs, and their similarity information is also different, which may lead to different mask ratios. Similarly, the same intermediate layer pair may output different feature map pairs in different training epochs, which may also lead to different mask ratios. By using a mask ratio dynamically determined based on the feature map pairs, it is beneficial to reduce the gap between the second and first image processing models, and to improve the accuracy of the trained second image processing model.

[0162] In one example, determining the mask ratio for merging the first and second feature maps based on the first and second feature maps includes: determining similarity information between the first and second feature maps; and determining the mask ratio for merging the first and second feature maps based on the similarity information. In this example, the similarity information between the first and second feature maps can be any information capable of representing the similarity between the first and second feature maps. In this example, by determining the similarity information between the first and second feature maps and determining the mask ratio for merging the first and second feature maps based on the similarity information, the accuracy of the trained second image processing model can be improved.

[0163] In other examples, the mask ratio for merging the first and second feature maps can be determined based on the correlation between the first and second feature maps, or based on other information about the first and second feature maps, without limitation here.

[0164] In one example, determining the similarity information between the first feature map and the second feature map includes: determining the CKA (Centered Kernel Alignment) similarity index between the first feature map and the second feature map; determining the mask ratio for merging the first feature map and the second feature map based on the similarity information includes: determining the mask ratio for merging the first feature map and the second feature map based on the CKA similarity index.

[0165] In one example, Equation 1 can be used to determine the first feature map. With the second feature map CKA similarity index between them:

[0166]

[0167] in, X′ represents the matrix after transforming the diagonal of X to 0, Y′ represents the matrix after transforming the diagonal of Y to 0, 1 represents the identity matrix, and n represents the length H of the feature map. i and width W i The product of , where tr represents the trace of the matrix.

[0168] In one example, used to merge the first feature map With the second feature map Mask ratio α i Equation 2 can be used to determine:

[0169] α i =1-CKA Equation 2.

[0170] In this example, by determining the CKA similarity index between the first feature map and the second feature map, and based on the CKA similarity index, determining the mask ratio for merging the first feature map and the second feature map, the accuracy of the trained second image processing model is improved.

[0171] In other examples, the similarity information between the first feature map and the second feature map can also be measured using cosine similarity, etc., which is not limited here.

[0172] In one example, determining the similarity information between the first feature map and the second feature map includes: aligning the first feature map and the second feature map; and determining the similarity information between the aligned first feature map and the second feature map. In this example, aligning the first feature map and the second feature map can mean making the number of channels and / or the size of the first feature map and the second feature map the same. By aligning the first feature map and the second feature map and determining the similarity information between the aligned first feature map and the second feature map, the similarity information between the first feature map and the second feature map can be determined more accurately.

[0173] In one example, aligning the first feature map and the second feature map includes: in response to the second feature map having a different number of channels than the first feature map, performing convolution processing on the second feature map so that the number of channels in the convolution-processed second feature map is the same as that in the first feature map; and / or, in response to the second feature map having a different size than the first feature map, performing bilinear interpolation on the second feature map so that the size of the bilinearly interpolated second feature map is the same as that in the first feature map; or, in response to the second feature map having a different number of channels than the first feature map, performing convolution processing on the first feature map so that the number of channels in the convolution-processed first feature map is the same as that in the second feature map; and / or, in response to the second feature map having a different size than the first feature map, performing bilinear interpolation on the first feature map so that the size of the bilinearly interpolated first feature map is the same as that in the second feature map.

[0174] In one example, in response to a difference in the number of channels between the second feature map and the first feature map, the second feature map can be convolved to make the number of channels in the convolved second feature map the same as that in the first feature map, and / or, in response to a difference in size between the second feature map and the first feature map, the second feature map can be bilinearly interpolated to make the size of the bilinearly interpolated second feature map the same as that in the first feature map. For example, if the feature map... The first feature map in With the second feature map With different channel numbers, the second feature map can be... Perform convolution processing to make the second feature map after convolution processing With the first feature map The number of channels is the same, where 1≤i≤T; if the feature map pairs The first feature map in With the second feature map Different sizes allow for different second feature maps. Perform bilinear interpolation to make the second feature map after bilinear interpolation... With the first feature map They are the same size.

[0175] In another example, in response to the second feature map having a different number of channels than the first feature map, the first feature map can be convolved to make the number of channels in the convolved first feature map the same as that in the second feature map, and / or, in response to the second feature map having a different size than the first feature map, the first feature map can be bilinearly interpolated to make the size of the bilinearly interpolated first feature map the same as that of the second feature map. For example, if the feature map... The first feature map in With the second feature map If the number of channels is different, then the first feature map can be... Perform convolution processing to make the first feature map after convolution processing With the second feature map The number of channels is the same, where 1≤i≤T; if the feature map pairs The first feature map in With the second feature map Different sizes allow for different first feature maps. Perform bilinear interpolation to make the first feature map after bilinear interpolation... With the second feature map They are the same size.

[0176] In this example, in response to the difference in the number of channels between the second feature map and the first feature map, the second feature map is convolved to make the number of channels in the convolved second feature map the same as that in the first feature map, thereby aligning the number of channels in the first and second feature maps; in response to the difference in size between the second feature map and the first feature map, bilinear interpolation is performed on the second feature map to make the size of the bilinearly interpolated second feature map the same as that in the first feature map, thereby aligning the size of the first and second feature maps; in response to the difference in the number of channels between the second feature map and the first feature map, the first feature map is convolved to make the number of channels in the convolved first feature map the same as that in the second feature map, thereby aligning the number of channels in the first and second feature maps; in response to the difference in size between the second feature map and the first feature map, bilinear interpolation is performed on the first feature map to make the size of the bilinearly interpolated first feature map the same as that in the second feature map, thereby aligning the size of the first and second feature maps.

[0177] In one example, determining a first set of feature blocks for generating a third feature map from the first feature map based on the mask region, and determining a second set of feature blocks for generating the third feature map from the second feature map, includes: determining a first set of feature blocks for generating the third feature map from the first feature map based on the position information of the mask region or the position information outside the mask region; and selecting feature blocks from the second feature map whose positions are complementary to those of the feature blocks in the first set of feature blocks to obtain a second set of feature blocks for generating the third feature map.

[0178] In one example, a first set of feature blocks for generating a third feature map can be determined from the first feature map based on the location information of the masked region; from the second feature map, feature blocks whose positions are complementary to those of the feature blocks in the first set of feature blocks are selected to obtain a second set of feature blocks for generating the third feature map. That is, based on the location information of the masked region, a first set of feature blocks for generating a third feature map can be determined according to the feature blocks in the first feature map that belong to the masked region; and based on the location information outside the masked region, a second set of feature blocks for generating a third feature map can be determined according to the feature blocks in the second feature map that do not belong to the masked region.

[0179] In another example, a first set of feature blocks for generating the third feature map can be determined from the first feature map based on the location information outside the masked region; and from the second feature map, feature blocks whose positions are complementary to those in the first set of feature blocks are selected to obtain a second set of feature blocks for generating the third feature map. That is, a first set of feature blocks for generating the third feature map can be determined from the first feature map based on the location information outside the masked region; and a second set of feature blocks for generating the third feature map can be determined from the second feature map based on the location information of the masked region.

[0180] In this example, a first set of feature blocks for generating a third feature map is determined from the first feature map by using the position information corresponding to the masked area or the position information outside the masked area. Then, from the second feature map, feature blocks whose positions are complementary to those of the feature blocks in the first set of feature blocks are selected to obtain a second set of feature blocks for generating the third feature map. Thus, the masked feature blocks in the second feature map are filled by feature blocks at corresponding positions in the first feature map, thereby enabling the first image processing model, which acts as the teacher model, to provide some prior knowledge to the second image processing model, which acts as the student model.

[0181] As another example of this implementation, determining a first set of feature blocks for generating a third feature map from the first feature map and a second set of feature blocks for generating the third feature map from the second feature map includes: determining a mask region according to a preset mask ratio; determining a first set of feature blocks for generating the third feature map from the first feature map and a second set of feature blocks for generating the third feature map from the second feature map based on the mask region. In this example, the mask ratio can be a fixed value.

[0182] As another example of this implementation, determining a first set of feature blocks for generating a third feature map from the first feature map and a second set of feature blocks for generating the third feature map from the second feature map includes: randomly determining a first set of feature blocks for generating a third feature map from the first feature map; and determining a second set of feature blocks for generating the third feature map from the second feature map based on the positions of the feature blocks in the first set of feature blocks; wherein the positions of the feature blocks in the first set of feature blocks and the second set of feature blocks are complementary.

[0183] As another example of this implementation, determining a first set of feature blocks for generating a third feature map from the first feature map, and determining a second set of feature blocks for generating the third feature map from the second feature map, includes: dividing the feature block set corresponding to the first feature map into multiple first feature block subsets, wherein any one of the multiple first feature block subsets includes at least two adjacent feature blocks; selecting a preset number of feature blocks from each of the multiple first feature block subsets to obtain a first feature block set; and selecting feature blocks whose positions are complementary to those of the feature blocks in the first feature block set from the feature block set corresponding to the second feature map to obtain a second feature block set. For example, each first feature block subset may include four adjacent feature blocks, and the preset number may be one.

[0184] As an example of this implementation, generating the third feature map based on the first feature block set and the second feature block set includes: performing positional encoding on the first feature block set and the second feature block set respectively to obtain first positional information corresponding to the first feature block set and second positional information corresponding to the second feature block set; encoding the first feature block set in combination with the first positional information to obtain a first feature block encoding result; encoding the second feature block set in combination with the second positional information to obtain a second feature block encoding result; and generating the third feature map based on the first feature block encoding result and the second feature block encoding result.

[0185] In this example, positional encoding methods such as sine and cosine positional encoding can be used to positionally encode the first feature block set to obtain first positional information, and the second feature block set can be positionally encoded to obtain second positional information. Here, the first positional information can represent the positional encoding result corresponding to the first feature block set, and the second positional information can represent the positional encoding result corresponding to the second feature block set.

[0186] In this example, a first encoding network corresponding to a first image processing model can be used to encode the first feature block set after adding the first position information, resulting in the first feature block encoding result; similarly, a second encoding network corresponding to a second image processing model can be used to encode the second feature block set after adding the second position information, resulting in the second feature block encoding result. For example, the first feature block encoding result can be denoted as f. i t The encoding result of the second feature block can be denoted as f. i s In one example, the first and second encoding networks can each be 6-layer multi-head self-attention networks. Of course, other network structures can also be used for the first and second encoding networks; for example, the number of layers in the self-attention network can be fewer or more, which is not limited here. The parameters of the first and second encoding networks can be updated as the second image processing model is trained; that is, the first and second encoding networks can be trained together with the second image processing model.

[0187] In this example, after obtaining the first feature block encoding result and the second feature block encoding result, the first feature block encoding result and the second feature block encoding result can be merged according to the relative position information between the feature blocks in the second feature map to obtain the third feature map.

[0188] In this example, by performing positional encoding on the first feature block set and the second feature block set respectively, first positional information corresponding to the first feature block set and second positional information corresponding to the second feature block set are obtained. Combining the first positional information, the first feature block set is encoded to obtain the first feature block encoding result. Combining the second positional information, the second feature block set is encoded to obtain the second feature block encoding result. Based on the first feature block encoding result and the second feature block encoding result, a third feature map is generated. Thus, by combining the positional information of the feature blocks in the first and second feature block sets to generate the third feature map, the third feature map can contain the positional information of the feature blocks in the first and second feature maps. This allows the prior features containing positional information provided by the teacher model to be used to train the second image processing model, which helps to improve the accuracy of the trained second image processing model.

[0189] In another possible implementation, generating a third feature map based on partial features from the first feature map and partial features from the second feature map includes: determining a first set of pixels from the first feature map for generating the third feature map, and determining a second set of pixels from the second feature map for generating the third feature map, wherein the first set of pixels represents the set of pixels in the first feature map used to generate the third feature map, and the second set of pixels represents the set of pixels in the second feature map used to generate the third feature map; and generating the third feature map based on the first set of pixels and the second set of pixels. In this implementation, the first and second feature maps can be divided into units of pixels.

[0190] In one possible implementation, determining the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map includes: performing position encoding on the third feature map to obtain third position information corresponding to the third feature map; inputting the third position information and the third feature map into a decoding network to obtain a fourth feature map; and determining the value of the loss function corresponding to the second image processing model based on the fourth feature map and the first feature map.

[0191] In this implementation, positional encoding methods such as sine and cosine positional encoding can be used to positionally encode the third feature map, obtaining the third positional information corresponding to the third feature map. In this implementation, the decoding network can consist of six multi-head self-attention layers and one multilayer perceptron. Of course, other network structures can also be used for the decoding network, which are not limited here. The parameters of the decoding network can be updated as the second image processing model is trained; that is, the decoding network can be trained together with the second image processing model. In this implementation, the value of the loss function corresponding to the second image processing model can be determined using L2 loss or L1 loss, etc. When at least two feature map pairs exist, the value of the loss function corresponding to the second image processing model can be determined based on the at least two feature map pairs.

[0192] In this implementation, the third feature map is positionally encoded to obtain the third position information corresponding to the third feature map. The third feature map with the added third position information is then input into the decoding network to obtain the fourth feature map. Based on the fourth feature map and the first feature map, the value of the loss function corresponding to the second image processing model is determined, thereby training the second image processing model and improving its accuracy.

[0193] In one possible implementation, obtaining the first feature map and the second feature map of the training image includes: obtaining at least two first feature maps corresponding to the training image extracted by at least two first intermediate layers of the first image processing model, and at least two second feature maps corresponding to the training image extracted by at least two second intermediate layers of the second image processing model, wherein the at least two first feature maps and the at least two second feature maps form at least two feature map pairs, and any feature map pair in the at least two feature map pairs includes a first feature map and a second feature map; generating a third feature map based on partial features in the first feature map and partial features in the second feature map includes: for any feature map pair in the at least two feature map pairs, generating a third feature map corresponding to the feature map pair based on partial features in the first feature map and partial features in the second feature map of the feature map pair; determining the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map includes: determining the value of the loss function corresponding to the second image processing model based on the third feature map corresponding to the at least two feature map pairs and the first feature map in the at least two feature map pairs.

[0194] In this implementation, the second image processing model is trained using at least two feature map pairs. For any feature map pair, the position of the corresponding third feature map can be encoded to obtain the third position information. The third position information and the third feature map can be input into a decoding network to obtain a fourth feature map, and the difference information between the fourth feature map and the first feature map in the feature map pair can be determined. Similarly, the difference information between the fourth feature map and the first feature map corresponding to each feature map pair can be determined. After determining the difference information corresponding to each feature map pair in the at least two feature map pairs, the value of the loss function corresponding to the second image processing model can be determined based on the weighted sum of the difference information corresponding to the at least two feature map pairs. In this implementation, by using at least two feature map pairs output from at least two intermediate layers in the first and second image processing models to train the second image processing model, the second image processing model can learn richer hidden layer features of the first image processing model, thereby further reducing the gap between the second and first image processing models.

[0195] In one possible implementation, the training of the second image processing model can also be supervised by combining the logits distillation method, thereby providing distillation results.

[0196] In one possible implementation, during the training of the second image processing model, the difference information between the prediction results of the second image processing model and the labeled data corresponding to the training images can be combined to supervise the second image processing model and improve its accuracy.

[0197] In one possible implementation, both the first image processing model and the second image processing model are used for image classification. After the second image processing model is trained, the method further includes: acquiring an image to be classified; processing the image to be classified using the second image processing model to obtain a feature map corresponding to the image to be classified; and processing the feature map corresponding to the image to be classified using the second image processing model to obtain a classification result corresponding to the image to be classified. In this implementation, the image to be classified can be any image that needs to be classified. By processing the image to be classified using the trained second image processing model to obtain a feature map corresponding to the image to be classified, and then processing the feature map corresponding to the image to be classified using the second image processing model to obtain a classification result corresponding to the image to be classified, the accuracy of image classification can be improved.

[0198] In one possible implementation, both the first image processing model and the second image processing model are used for object detection. After the second image processing model is trained, the method further includes: acquiring an image to be detected; processing the image to be detected using the second image processing model to obtain a feature map corresponding to the image to be detected; and processing the feature map corresponding to the image to be detected using the second image processing model to obtain an object detection result corresponding to the image to be detected. In this implementation, the image to be detected can be any image for which object detection is required. By processing the image to be detected using the trained second image processing model to obtain a feature map corresponding to the image to be detected, and then processing the feature map corresponding to the image to be detected using the second image processing model to obtain an object detection result corresponding to the image to be detected, the accuracy of object detection on the image to be detected can be improved.

[0199] The image processing model training method provided in this disclosure can be applied to fields such as computer vision.

[0200] The training method of the image processing model provided in this disclosure is illustrated below through a specific application scenario. In this application scenario, the first image processing model can be ResNet152, and the second image processing model can be ResNet18.

[0201] The training image can be input into the first image processing model to obtain the first feature map of the last layer output before the last downsampling layer of the first image processing model. and the first feature map output from the last layer before the penultimate downsampling layer of the first image processing model. The training image can be input into the second image processing model to obtain the second feature map of the last layer output before the last downsampling layer of the second image processing model. And the second feature map output from the last layer before the penultimate downsampling layer of the second image processing model. Among them, the first feature map Second feature map Forming the first feature map pair First feature map Second feature map Composition of the second feature map pair

[0202] For the first feature map pair exist and With different channel numbers, it is possible to... Perform convolution processing, so that the convolutional process... With the The number of channels is the same. and In cases of different sizes, it is possible to... Perform bilinear interpolation to make the bilinear interpolated result With the They are the same size.

[0203] In alignment and After that, you can and Divided into sizes P 1 ×P 1 Feature blocks. For example, aligned and All dimensions are H 1 ×W 1 ,So, and M can be obtained by dividing it separately. 1 =(H 1 ×W 1 ) / (P 1 ×P 1 ) feature blocks.

[0204] Based on equations 1, 2, and 3 above, the formula used for merging can be determined. and Mask ratio α i It can be based on the mask ratio α. i The mask area is randomly determined. This can be based on... The feature blocks belonging to the mask region are used to determine the first set of feature blocks for generating the third feature map F1; it can be based on... The feature blocks that do not belong to the mask region are used to determine the second set of feature blocks for generating F1.

[0205] The first feature block set can be positionally encoded to obtain the first position information corresponding to the first feature block set. Furthermore, the first encoding network corresponding to the first image processing model can be used to encode the first feature block set after adding the first position information, resulting in the first feature block encoding result. The second feature block set can be positionally encoded to obtain the second positional information corresponding to the second feature block set. Furthermore, the second encoding network corresponding to the second image processing model can be used to encode the second feature block set after adding the second positional information, resulting in the second feature block encoding result. The first and second encoding networks can each be 6-layer multi-head self-attention networks. (The sentence is incomplete and requires further context.) and After that, it can be based on The relative positional information between feature blocks in the data is merged. and The third feature map F1 is obtained.

[0206] Positional encoding can be performed on F1 to obtain the third positional information corresponding to F1. The F1 with the added third positional information can then be input into the decoding network to obtain the fourth feature map.

[0207] Similarly, for the second feature map pair The fourth feature map can be obtained.

[0208] according to and The differences between them, and and The difference information between them can be used to obtain the first loss function corresponding to the second image processing model. The value of .

[0209] In addition, the logits distillation method can be combined to determine the second loss function corresponding to the second image processing model based on the difference between the logits output by the second image processing model and the logits output by the first image processing model. The value of and, based on the difference between the prediction results of the second image processing model and the labeled data corresponding to the training images, the third loss function corresponding to the second image processing model is obtained. The value of .

[0210] In one example, Equation 4 can be used to determine the loss function corresponding to the second image processing model. Value:

[0211]

[0212] Where α represents The corresponding weight, β represents The corresponding weights, α and β, can be determined empirically.

[0213] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0214] In addition, this disclosure also provides a training device for an image processing model, an electronic device, a computer-readable storage medium, and a computer program product. All of the above can be used to implement any of the image processing model training methods provided in this disclosure. The corresponding technical solutions and effects can be found in the relevant descriptions in the method section, and will not be repeated here.

[0215] Figure 2 A block diagram of a training apparatus for an image processing model provided in an embodiment of this disclosure is shown. Figure 2 As shown, the training device for the image processing model includes:

[0216] The acquisition module 21 is used to acquire a first feature map and a second feature map of the training image, wherein the first feature map is output by a first image processing model and the second feature map is output by a second image processing model.

[0217] Generation module 22 is used to generate a third feature map based on some features in the first feature map and some features in the second feature map;

[0218] The determining module 23 is used to determine the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map;

[0219] Training module 24 is used to train the second image processing model based on the value of the loss function.

[0220] In one possible implementation, the generation module 22 is used for:

[0221] From the first feature map, a first set of feature blocks for generating a third feature map is determined, and from the second feature map, a second set of feature blocks for generating the third feature map is determined, wherein the first set of feature blocks represents a set of feature blocks in the first feature map used to generate the third feature map, and the first set of feature blocks includes some features of the first feature map; the second set of feature blocks represents a set of feature blocks in the second feature map used to generate the third feature map, and the second set of feature blocks includes some features of the second feature map.

[0222] The third feature map is generated based on the first feature block set and the second feature block set.

[0223] In one possible implementation, the generation module 22 is used for:

[0224] Based on the first feature map and the second feature map, determine the mask ratio used to merge the first feature map and the second feature map;

[0225] Determine the mask area based on the mask ratio;

[0226] Based on the mask region, a first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map, wherein the positions of the feature blocks in the first set of feature blocks and the second set of feature blocks are complementary.

[0227] In one possible implementation, the generation module 22 is used for:

[0228] Determine the similarity information between the first feature map and the second feature map;

[0229] Based on the similarity information, a mask ratio for merging the first feature map and the second feature map is determined.

[0230] In one possible implementation, the generation module 22 is used for:

[0231] Determine the intermediate kernel alignment CKA similarity index between the first feature map and the second feature map;

[0232] Based on the CKA similarity index, the mask ratio used to merge the first feature map and the second feature map is determined.

[0233] In one possible implementation, the generation module 22 is used for:

[0234] Align the first feature map with the second feature map;

[0235] Determine the similarity information between the aligned first feature map and the second feature map.

[0236] In one possible implementation, the generation module 22 is used for:

[0237] In response to the fact that the number of channels of the second feature map is different from that of the first feature map, the second feature map is convolved so that the number of channels of the convolved second feature map is the same as that of the first feature map, and / or, in response to the fact that the size of the second feature map is different from that of the first feature map, the second feature map is bilinearly interpolated so that the size of the bilinearly interpolated second feature map is the same as that of the first feature map.

[0238] or,

[0239] In response to the second feature map having a different number of channels than the first feature map, the first feature map is convolved to make the number of channels in the convolved first feature map the same as that in the second feature map, and / or, in response to the second feature map having a different size than the first feature map, the first feature map is bilinearly interpolated to make the size of the bilinearly interpolated first feature map the same as that of the second feature map.

[0240] In one possible implementation, the determining module 23 is used to:

[0241] Based on the location information of the masked area or the location information outside the masked area, a first set of feature blocks for generating the third feature map is determined from the first feature map;

[0242] From the second feature map, feature blocks whose positions are complementary to those of the feature blocks in the first feature block set are selected to obtain a second feature block set used to generate the third feature map.

[0243] In one possible implementation, the generation module 22 is used for:

[0244] Position encoding is performed on the first feature block set and the second feature block set respectively to obtain the first position information corresponding to the first feature block set and the second position information corresponding to the second feature block set;

[0245] By combining the first location information, the first feature block set is encoded to obtain the first feature block encoding result;

[0246] By combining the second position information, the second feature block set is encoded to obtain the second feature block encoding result;

[0247] A third feature map is generated based on the encoding results of the first and second feature blocks.

[0248] In one possible implementation, the determining module 23 is used to:

[0249] The third feature map is positionally encoded to obtain the third position information corresponding to the third feature map;

[0250] The third location information and the third feature map are input into the decoding network to obtain the fourth feature map;

[0251] The value of the loss function corresponding to the second image processing model is determined based on the fourth feature map and the first feature map.

[0252] In one possible implementation,

[0253] The acquisition module 21 is used to: acquire at least two first feature maps corresponding to the training image extracted by at least two first intermediate layers of the first image processing model, and at least two second feature maps corresponding to the training image extracted by at least two second intermediate layers of the second image processing model, wherein the at least two first feature maps and the at least two second feature maps form at least two feature map pairs, and any feature map pair in the at least two feature map pairs includes a first feature map and a second feature map;

[0254] The generation module 22 is used to: for any of the at least two feature map pairs, generate a third feature map corresponding to the feature map pair based on some features in the first feature map of the feature map pair and some features in the second feature map of the feature map pair;

[0255] The determining module 23 is used to: determine the value of the loss function corresponding to the second image processing model based on the third feature map corresponding to the at least two feature map pairs and the first feature map in the at least two feature map pairs.

[0256] In one possible implementation, both the first image processing model and the second image processing model are used for image classification;

[0257] The device further includes:

[0258] The classification module is used to acquire an image to be classified; process the image to be classified using the second image processing model to obtain a feature map corresponding to the image to be classified; and process the feature map corresponding to the image to be classified using the second image processing model to obtain a classification result corresponding to the image to be classified.

[0259] In one possible implementation, both the first image processing model and the second image processing model are used for target detection;

[0260] The device further includes:

[0261] The target detection module is used to acquire an image to be detected; process the image to be detected using the second image processing model to obtain a feature map corresponding to the image to be detected; and process the feature map corresponding to the image to be detected using the second image processing model to obtain a target detection result corresponding to the image to be detected.

[0262] In this embodiment, a first feature map and a second feature map of the training image are obtained. The first feature map is output by a first image processing model, and the second feature map is output by a second image processing model. A third feature map is generated based on some features in the first feature map and some features in the second feature map. The value of the loss function corresponding to the second image processing model is determined based on the third feature map and the first feature map. The second image processing model is trained based on the value of the loss function. Thus, a portion of the feature map extracted by the first image processing model is used as prior knowledge for the second image processing model, enabling the second image processing model to mimic the features output by the first image processing model. This allows the second image processing model to improve as the performance of the first image processing model improves. In other words, it can solve the gap between teacher and student models and the problem that the performance of the second image model no longer improves as the performance of the first image model improves. Therefore, it is possible to use a larger-scale, higher-performance first image processing model to distill a higher-performance second image processing model.

[0263] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation and technical effects can be referred to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0264] This disclosure also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.

[0265] This disclosure also proposes a computer program including computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device executes the above-described method.

[0266] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, the processor in the electronic device executes the above-described method.

[0267] This disclosure also provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the executable instructions stored in the memory to perform the above-described method.

[0268] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0269] Figure 3 A block diagram of a sub-device 1900 provided in an embodiment of this disclosure is shown. For example, electronic device 1900 may be provided as a server. (Refer to...) Figure 3 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0270] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0271] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0272] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0273] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0274] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0275] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0276] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0277] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0278] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0279] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0280] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0281] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0282] If the technical solution of this disclosure involves personal information, the product applying the technical solution of this disclosure has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this disclosure involves sensitive personal information, the product applying the technical solution of this disclosure has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to indicate that the user has entered the scope of personal information collection and that personal information will be collected. If the user voluntarily enters the collection scope, it is deemed to have consented to the collection of their personal information; or on the personal information processing device, with clear signs / information informing the user of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0283] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for an image processing model, characterized in that, include: Obtain a first feature map and a second feature map of the training image, wherein the first feature map is output by a first image processing model and the second feature map is output by a second image processing model; A third feature map is generated based on some features in the first feature map and some features in the second feature map; Based on the third feature map and the first feature map, determine the value of the loss function corresponding to the second image processing model; The second image processing model is trained based on the value of the loss function; The step of generating a third feature map based on partial features from the first feature map and partial features from the second feature map includes: From the first feature map, a first set of feature blocks for generating a third feature map is determined, and from the second feature map, a second set of feature blocks for generating the third feature map is determined, wherein the first set of feature blocks represents a set of feature blocks in the first feature map used to generate the third feature map, and the first set of feature blocks includes some features of the first feature map; the second set of feature blocks represents a set of feature blocks in the second feature map used to generate the third feature map, and the second set of feature blocks includes some features of the second feature map. The third feature map is generated based on the first feature block set and the second feature block set.

2. The method according to claim 1, characterized in that, The step of determining a first set of feature blocks for generating a third feature map from the first feature map, and determining a second set of feature blocks for generating the third feature map from the second feature map, includes: Based on the first feature map and the second feature map, determine the mask ratio used to merge the first feature map and the second feature map; Determine the mask area based on the mask ratio; Based on the mask region, a first set of feature blocks for generating a third feature map is determined from the first feature map, and a second set of feature blocks for generating the third feature map is determined from the second feature map, wherein the positions of the feature blocks in the first set of feature blocks and the second set of feature blocks are complementary.

3. The method according to claim 2, characterized in that, The step of determining the mask ratio for merging the first feature map and the second feature map based on the first feature map and the second feature map includes: Determine the similarity information between the first feature map and the second feature map; Based on the similarity information, a mask ratio for merging the first feature map and the second feature map is determined.

4. The method according to claim 3, characterized in that, Determining the similarity information between the first feature map and the second feature map includes: determining the kernel alignment (CKA) similarity index between the first feature map and the second feature map; The step of determining the mask ratio for merging the first feature map and the second feature map based on the similarity information includes: determining the mask ratio for merging the first feature map and the second feature map based on the CKA similarity index.

5. The method according to claim 3, characterized in that, Determining the similarity information between the first feature map and the second feature map includes: Align the first feature map with the second feature map; Determine the similarity information between the aligned first feature map and the second feature map.

6. The method according to claim 5, characterized in that, Aligning the first feature map with the second feature map includes: In response to the fact that the number of channels of the second feature map is different from that of the first feature map, the second feature map is convolved so that the number of channels of the convolved second feature map is the same as that of the first feature map, and / or, in response to the fact that the size of the second feature map is different from that of the first feature map, the second feature map is bilinearly interpolated so that the size of the bilinearly interpolated second feature map is the same as that of the first feature map. or, In response to the second feature map having a different number of channels than the first feature map, the first feature map is convolved to make the number of channels in the convolved first feature map the same as that in the second feature map, and / or, in response to the second feature map having a different size than the first feature map, the first feature map is bilinearly interpolated to make the size of the bilinearly interpolated first feature map the same as that of the second feature map.

7. The method according to any one of claims 2 to 6, characterized in that, The step of determining a first set of feature blocks for generating a third feature map from the first feature map based on the mask region, and determining a second set of feature blocks for generating the third feature map from the second feature map, includes: Based on the location information of the masked area or the location information outside the masked area, a first set of feature blocks for generating the third feature map is determined from the first feature map; From the second feature map, feature blocks whose positions are complementary to those of the feature blocks in the first feature block set are selected to obtain a second feature block set used to generate the third feature map.

8. The method according to any one of claims 1 to 6, characterized in that, The step of generating the third feature map based on the first feature block set and the second feature block set includes: Position encoding is performed on the first feature block set and the second feature block set respectively to obtain the first position information corresponding to the first feature block set and the second position information corresponding to the second feature block set; By combining the first location information, the first feature block set is encoded to obtain the first feature block encoding result; By combining the second position information, the second feature block set is encoded to obtain the second feature block encoding result; A third feature map is generated based on the encoding results of the first and second feature blocks.

9. The method according to any one of claims 1 to 6, characterized in that, The step of determining the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map includes: The third feature map is positionally encoded to obtain the third position information corresponding to the third feature map; The third location information and the third feature map are input into the decoding network to obtain the fourth feature map; The value of the loss function corresponding to the second image processing model is determined based on the fourth feature map and the first feature map.

10. The method according to any one of claims 1 to 6, characterized in that, The step of obtaining the first feature map and the second feature map of the training image includes: obtaining at least two first feature maps corresponding to the training image extracted by at least two first intermediate layers of the first image processing model, and at least two second feature maps corresponding to the training image extracted by at least two second intermediate layers of the second image processing model, wherein the at least two first feature maps and the at least two second feature maps form at least two feature map pairs, and any feature map pair in the at least two feature map pairs includes a first feature map and a second feature map; The step of generating a third feature map based on partial features in the first feature map and partial features in the second feature map includes: for any feature map pair among the at least two feature map pairs, generating a third feature map corresponding to the feature map pair based on partial features in the first feature map and partial features in the second feature map of the feature map pair; Determining the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map includes: determining the value of the loss function corresponding to the second image processing model based on the third feature map corresponding to the at least two feature map pairs and the first feature map in the at least two feature map pairs.

11. The method according to any one of claims 1 to 6, characterized in that, Both the first image processing model and the second image processing model are used for image classification; After the second image processing model has been trained, the method further includes: Obtain the image to be classified; The image to be classified is processed by the second image processing model to obtain the feature map corresponding to the image to be classified; The second image processing model processes the feature map corresponding to the image to be classified to obtain the classification result corresponding to the image to be classified.

12. The method according to any one of claims 1 to 6, characterized in that, Both the first image processing model and the second image processing model are used for target detection; After the second image processing model has been trained, the method further includes: Acquire the image to be detected; The image to be detected is processed by the second image processing model to obtain the feature map corresponding to the image to be detected; The second image processing model is used to process the feature map corresponding to the image to be detected, thereby obtaining the target detection result corresponding to the image to be detected.

13. A training device for an image processing model, characterized in that, include: The acquisition module is used to acquire a first feature map and a second feature map of the training image, wherein the first feature map is output by a first image processing model and the second feature map is output by a second image processing model. A generation module is used to generate a third feature map based on some features in the first feature map and some features in the second feature map; The determining module is used to determine the value of the loss function corresponding to the second image processing model based on the third feature map and the first feature map; The training module is used to train the second image processing model based on the value of the loss function; The generation module is used for: From the first feature map, a first set of feature blocks for generating a third feature map is determined, and from the second feature map, a second set of feature blocks for generating the third feature map is determined, wherein the first set of feature blocks represents a set of feature blocks in the first feature map used to generate the third feature map, and the first set of feature blocks includes some features of the first feature map; the second set of feature blocks represents a set of feature blocks in the second feature map used to generate the third feature map, and the second set of feature blocks includes some features of the second feature map. The third feature map is generated based on the first feature block set and the second feature block set.

14. An electronic device, characterized in that, include: One or more processors; Memory used to store executable instructions; The one or more processors are configured to invoke executable instructions stored in the memory to perform the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 12.

16. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 12.

Images