Model training, image processing method, device and storage medium

Abstract

Embodiments of the present application provide a model training method, an image processing method, a device and a storage medium. In the process of model training based on the training method of knowledge distillation, after obtaining the feature maps output by the teacher model and the student model according to the portrait image, the structural features of the portrait image can be utilized to determine the feature correlation loss of the student model and the teacher model through the structural correlation of the foreground region and / or the background region in the portrait image. The feature correlation loss can be used to express the gap between the student model and the teacher model in feature expression. Thus, the student model is prompted to continuously learn the structural correlation of different regions and the feature expression ability for different regions under the guidance of the teacher model, thereby improving the distinguishing ability of the student model for the foreground and the background region and further improving the image segmentation accuracy of the student model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a model training, image processing method, device and storage medium. Background Technology

[0002] With the widespread adoption of artificial intelligence technology, deploying neural network models on mobile devices has become a strong demand. For example, in applications such as education, meetings, and live streaming, portrait background segmentation is gradually becoming an essential function. Uploading the images or videos to be segmented to a server for portrait segmentation increases processing time and network transmission costs. However, some mobile devices and personal computers without dedicated video memory have limited computing power, which cannot meet the needs of real-time processing of large models.

[0003] Currently, there is a neural network model training method based on knowledge distillation. Knowledge distillation can transfer some of the representational capabilities of a complex neural network (teacher network) to a less complex neural network (student network) during training, thereby improving the prediction performance of the student network without changing its training time. However, the segmentation accuracy of student networks trained using existing knowledge distillation methods is relatively poor, and a new solution is needed. Summary of the Invention

[0004] This application provides a model training method, image processing method, device, and storage medium for improving the image segmentation accuracy of student networks.

[0005] This application provides an image processing method, comprising: acquiring a portrait image to be processed; inputting the portrait image to be processed into an image segmentation model to obtain a segmentation result of the portrait image; wherein, the image segmentation model is trained based on a feature correlation loss between the image segmentation model and a teacher model; the feature correlation loss includes: foreground correlation error and / or, background correlation error; wherein, the foreground correlation error is determined based on the structural correlation of the foreground region in the portrait image sample, from a first feature map extracted by the teacher model and a second feature map extracted by the image segmentation model; the background correlation error is determined based on the structural correlation of the background region in the portrait image sample, from the first feature map and the second feature map.

[0006] This application also provides a model training method, comprising: inputting a portrait image into a teacher model and a student model respectively to obtain a first feature map and a second feature map; determining the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the structural correlation of the foreground region in the portrait image; and / or determining the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the structural correlation of the background region in the portrait image; determining the feature correlation loss of the teacher model and the student model based on the foreground correlation error and / or the background correlation error; training the student model based on the feature correlation loss, and outputting the trained student model as an image segmentation model when the student model meets a preset convergence condition.

[0007] Optionally, determining the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the structural correlation of the foreground region in the portrait image includes: determining the foreground region in the portrait image based on the segmentation ground truth value marked on the portrait image; determining the first foreground feature correlation degree of the foreground region in the portrait image on the first feature map and the second foreground feature correlation degree of the foreground region in the second feature map; and determining the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the difference between the first foreground feature correlation degree and the second foreground feature correlation degree.

[0008] Optionally, determining the first foreground feature correlation of the foreground region in the portrait image in the first feature map includes: for any first foreground position point and a second foreground position point in the portrait image, determining the feature values ​​corresponding to the first foreground position point and the second foreground position point in the first feature map respectively, to obtain the feature vector of the first foreground position point and the feature vector of the second foreground position point; and determining the foreground feature correlation of the first foreground position point and the second foreground position point based on the similarity between the feature vectors of the first foreground position point and the feature vectors of the second foreground position point.

[0009] Optionally, determining the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the structural correlation of the background region in the portrait image includes: determining the background region in the portrait image based on the segmentation ground truth value marked on the portrait image; determining the first background feature correlation degree of the background region in the portrait image on the first feature map and the second background feature correlation degree on the second feature map, respectively; and determining the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the difference between the first background feature correlation degree and the second background feature correlation degree.

[0010] Optionally, determining the feature relevance loss of the teacher model and the student model based on the foreground relevance error and / or the background relevance error includes: calculating the global feature relevance of the first feature map and the global feature relevance of the second feature map; determining the global spatial domain loss based on the difference between the global feature relevance of the first feature map and the global feature relevance of the second feature map; and determining the feature relevance loss of the teacher model and the student model based on the weighted result of the foreground relevance error, the background relevance error, and the global spatial domain loss.

[0011] Optionally, calculating the global feature correlation of the first feature map includes: for any channel of the first feature map, performing mean normalization on the feature values ​​corresponding to the pixels in the portrait image in the feature map to obtain the global feature correlation of the feature map.

[0012] This application also provides a terminal device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: perform the steps in the image processing method provided in this application.

[0013] This application embodiment also provides a server, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: perform the steps in the model training method provided in this application embodiment.

[0014] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the method provided in this application.

[0015] In this embodiment, during model training using the knowledge distillation-based training method, after obtaining the feature maps output by the teacher and student models based on the portrait image, the structural features of the portrait image can be utilized. By analyzing the structural correlation between the foreground and / or background regions in the portrait image, the feature correlation loss between the student and teacher models can be determined. This feature correlation loss can be used to express the gap between the student and teacher models in feature representation. This encourages the student model to continuously learn the structural correlation of different regions and its ability to represent features in different regions under the guidance of the teacher model, thereby improving the student model's ability to distinguish between foreground and background regions and further enhancing the image segmentation accuracy of the student model. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A schematic diagram illustrating the model training process provided for an exemplary embodiment of this application;

[0018] Figure 2 A schematic diagram of the model training process provided for another exemplary embodiment of this application;

[0019] Figure 3 A schematic flowchart of an image processing method provided for an exemplary embodiment of this application;

[0020] Figure 4 A schematic diagram of the structure of a terminal device provided in an exemplary embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of a server provided for an exemplary embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0024] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0025] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0026] To address the technical problem of low segmentation accuracy in existing knowledge distillation-based segmentation models, this application provides a solution in some embodiments. The technical solutions provided by each embodiment of this application are described in detail below with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart illustrating a model training method provided in an exemplary embodiment of this application. The method may include, for example: Figure 1 The steps shown are as follows:

[0028] Step 101: Input the portrait images into the teacher model and the student model respectively to obtain the first feature map and the second feature map.

[0029] Step 102: Determine the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the structural correlation of the foreground region in the portrait image; and / or, determine the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the structural correlation of the background region in the portrait image.

[0030] Step 103: Determine the feature correlation loss of the teacher model and the student model based on the foreground correlation error and / or the background correlation error.

[0031] Step 104: Train the student model according to the feature correlation loss, and when the student model meets the preset convergence condition, output the trained student model as the image segmentation model.

[0032] In this embodiment, knowledge distillation is used to train the student model. Knowledge distillation employs a teacher-student model, using a teacher model to assist in the training of the student model. The teacher model refers to a neural network model with higher computational complexity, while the student model refers to a lightweight neural network model with lower computational complexity. The teacher model has stronger learning capabilities and can transfer its learned knowledge to the student model, which has relatively weaker learning capabilities, thereby enhancing the student model's generalization ability. The student model typically has a smaller data volume and lower computational cost, making it more suitable for running on the terminal side. After training, the student model can be flexibly deployed on various terminal devices and run in a lightweight manner.

[0033] The first feature map is obtained by the teacher model extracting features from the portrait image, and the second feature map is obtained by the student model extracting features from the portrait image. The first and second feature maps have the same size.

[0034] A portrait image consists of a foreground region and a background region. The foreground region is the set of pixels in the image that contains the main subject, while the background region is the set of pixels in the image that does not contain the main subject. For example, in a portrait image, the set of pixels that contains the human body is the foreground region, and the set of pixels that does not contain the human body is the background region.

[0035] In this system, the portrait image is labeled with ground truth values ​​for different regions. For example, in portrait segmentation, the region containing the portrait can be labeled as 1, and the background region without the portrait can be labeled as 0. In portrait matting, the region inside the human body can be labeled as 1, the region without the portrait can be labeled as 0, and the regions with glasses, hair strands, or motion blur can be labeled with values ​​between 0 and 1. Based on the labeled ground truth values, different regions in the image can be identified, and the structural correlations between these regions can be used for model training.

[0036] The structural correlation of the foreground region in a portrait image refers to the structural commonality of the foreground in different portrait images. Similarly, the structural correlation of the background region in a portrait image refers to the structural commonality of the background region. From a macroscopic perspective, when the dataset of portrait images is large, the foreground in different portrait images typically possesses structural features such as the head, limbs, torso, and facial features. In contrast, the background region in a portrait image is usually cluttered and lacks these structural features. These structural features in the foreground distinguish it from the background region, thus giving pixels in the foreground a certain degree of structural correlation. Correspondingly, pixels in the background region of a portrait image also exhibit a certain degree of structural correlation.

[0037] In this model, the teacher model extracts features that express the correlation of pixels in the foreground of a portrait. The student model, through knowledge transfer and under the guidance of the teacher model, learns this feature representation ability, enabling its extracted features to express the correlation of pixels in the foreground. Similarly, the teacher model extracts features that express the correlation of pixels in the background region. The student model, through knowledge transfer and under the guidance of the teacher model, learns this feature representation ability, enabling its extracted features to express the correlation of pixels in the background region.

[0038] In some embodiments, during training, after acquiring the first feature map and the second feature map, the foreground correlation error of the first and second feature maps in the foreground region can be determined based on the structural correlation of the foreground region in the portrait image. The foreground correlation error represents the difference in foreground feature extraction capabilities between the teacher model and the student model. Based on the foreground correlation error, the feature correlation loss of the teacher model and the student model can be determined, and the student model can be trained based on this correlation loss. Furthermore, the foreground feature representation ability learned by the teacher model can be transferred to the student model, improving the student model's ability in foreground feature extraction, thereby enhancing the student model's foreground recognition capability.

[0039] In other embodiments, during training, after acquiring the first and second feature maps, the background correlation error between the first and second feature maps in the background region can be determined based on the structural correlation of the background region in the portrait image. The foreground correlation error is used to express the difference in background feature extraction capabilities between the teacher model and the student model. Based on the background correlation error, the feature correlation loss of the teacher model and the student model can be determined, and the student model can be trained based on this correlation loss. Furthermore, the background feature representation ability learned by the teacher model can be transferred to the student model, improving the student model's ability to extract background features, thereby enhancing the student model's background recognition capability.

[0040] In some other embodiments, during training, the foreground correlation error and background correlation error can be acquired. Based on these errors, the feature correlation loss of the teacher model and the student model can be determined, and the student model can be trained based on this correlation loss. Furthermore, the foreground and background feature representation capabilities learned by the teacher model can be transferred to the student model, improving its ability to extract foreground and background features, thereby enhancing its ability to recognize and distinguish foreground and background.

[0041] During the training of the student model based on foreground correlation error and / or background correlation error, after each training round, it can be determined whether the student model meets a preset convergence condition. This convergence condition may include: the total loss of the student model is less than a certain set threshold, or the total loss of the student model fluctuates within a certain range. When the student model meets the preset convergence condition, the trained student model is output as the image segmentation model.

[0042] In the model training process based on knowledge distillation, after obtaining the feature maps output by the teacher and student models from the portrait image, the structural features of the portrait image can be utilized. By analyzing the structural correlation between the foreground and / or background regions in the portrait image, the feature correlation loss between the student and teacher models can be determined. This feature correlation loss can be used to express the gap between the student and teacher models in feature representation. This prompts the student model to continuously learn the structural correlations of different regions and its ability to represent features in different regions under the guidance of the teacher model, thereby improving the student model's ability to distinguish between foreground and background regions and further enhancing the image segmentation accuracy of the student model.

[0043] The following will combine Figure 2 The model training method provided in the embodiments of this application will be further illustrated by example.

[0044] In some optional embodiments, when determining the foreground relevance error of the first feature map and the second feature map in the foreground region, the foreground region in the portrait image can be determined based on the foreground ground truth value annotated in the portrait image, and the foreground relevance of the foreground region in the portrait image in the first feature map (hereinafter referred to as the first foreground relevance) and the foreground relevance in the second feature map (hereinafter referred to as the second foreground relevance) can be determined respectively. Based on the first foreground relevance and the second foreground relevance, the foreground relevance error of the first feature map and the second feature map in the foreground region is determined, such as... Figure 2 As shown.

[0045] The feature correlation of the foreground region can be determined by calculating the feature correlation between any two points (i.e., pixels) within the foreground region. The feature correlation between any two points represents the correlation between their feature values. For example, if two pixels in a portrait image both belong to the human torso, their feature correlation is high; if two pixels belong to the human torso and the background respectively, their feature correlation is low. Based on the first feature map extracted from the teacher model, the feature correlation between any two pixels in the foreground can be calculated, forming a correlation map. Based on the second feature map extracted from the student model, the feature correlation between any two pixels in the foreground can also be calculated, forming a correlation map for subsequent knowledge distillation.

[0046] The feature correlation between any two pixels can be calculated from the feature values ​​between those pixels. An example will be provided below.

[0047] Optionally, when determining the first foreground correlation of the foreground region in the portrait image in the first feature map, the similarity between pairwise pixels in the foreground region can be calculated in the first feature map. Taking any first foreground position point and a second foreground position point in the portrait image as examples, the feature value corresponding to the first foreground position point in the first feature map can be determined, obtaining the feature vector of the first foreground position point. Similarly, the feature value corresponding to the second foreground position point in the first feature map can be determined, obtaining the feature vector of the second foreground position point. When the first feature map is a multi-channel feature map, the feature vector of the first foreground position point can contain multiple feature values, arranged in channel order. Similarly, the feature vector of the second foreground position point can also contain multiple feature values, arranged in channel order.

[0048] The foreground relevance of the first and second foreground points is determined based on the similarity between their feature vectors. This similarity can be calculated using at least one algorithm selected from cosine distance, Euclidean distance, and Jaccard distance; this embodiment is not limited to this. In some exemplary embodiments, the similarity between the feature vectors of the first and second foreground points in the first feature map extracted by the teacher model can be expressed using the following formula:

[0049]

[0050] in, Let represent the eigenvector extracted by the teacher model at the foreground position point i. R represents the eigenvector extracted by the teacher model at the foreground position point j; tea-fg (Φ) i，j This represents the similarity between the feature vectors of foreground point i and foreground point j in the feature map extracted by the teacher's model.

[0051] In the second feature map extracted by the student model, the similarity between the feature vectors of the first foreground point and the feature vectors of the second foreground point can be expressed by the following formula:

[0052]

[0053] in, Let represent the eigenvector extracted by the student model at the foreground position point i. R represents the eigenvector extracted by the student model at the foreground position point j; s t u_fg (Φ) i，j This represents the similarity between the feature vectors of foreground point i and foreground point j in the feature map extracted by the student model.

[0054] Based on the above implementation method, the similarity between the feature vectors of any pair of foreground points in the portrait image in the first feature map and the similarity between the feature vectors of any pair of foreground points in the second feature map can be calculated. Then, based on the similarity between the feature vectors of any pair of foreground points in the portrait image, a first foreground feature correlation can be determined, and based on the similarity between the feature vectors of any pair of foreground points in the portrait image, a second foreground feature correlation can be determined. Next, based on the first and second foreground feature correlations, the foreground correlation error between the first and second feature maps in the foreground region can be calculated, such as... Figure 2 As shown in the figure, the foreground relevance error describes the error in the feature relevance between the foreground features extracted by the teacher model and the student model. Knowledge distillation based on the foreground relevance error allows the knowledge learned by the teacher model in foreground feature extraction to be transferred to the student model, thereby improving the student model's ability in foreground feature extraction.

[0055] Optionally, the foreground correlation error can be determined based on the difference between the first foreground feature correlation degree and the second foreground feature correlation degree. The larger the difference between the first foreground feature correlation degree and the second foreground feature correlation degree, the larger the foreground correlation error. In some embodiments, the L2 norm can be used to calculate the first foreground feature correlation degree and the second foreground feature correlation degree to obtain the foreground correlation error. In the calculation of the L2 norm, the square root of the difference between the first foreground feature correlation degree and the second foreground feature correlation degree can be summed, and the square root of the summation result can be taken to obtain a correlation error with a positive result. Optionally, the specific formula for calculating the foreground correlation error based on the L2 norm is as follows:

[0056] Loss fg (i, j) = |R tea_fg (Φ) i，j -R stu_fg (Φ) i，j ||2, i, j ∈ foreground region

[0057] Among them, Loss fg (i, j) represents the foreground correlation error of foreground position points i and j.

[0058] The larger the foreground correlation error, the greater the difference in foreground feature representation ability between the teacher's model and the student's model. Therefore, during the training of the student model, the foreground correlation error can be converged as the optimization objective, and the parameters of the student model can be continuously adjusted until the foreground correlation error converges to the target range. Furthermore, the structural features of the foreground region can be learned in portrait images, which is beneficial for more accurately identifying pixels in the foreground region from portrait images.

[0059] In some optional embodiments, when determining the background correlation error of the first feature map and the second feature map in the background region, the background region in the portrait image can be determined based on the ground truth value of the background annotated in the portrait image, and the background feature correlation of the background region in the portrait image in the first feature map (hereinafter referred to as the first background feature correlation) and the background feature correlation in the second feature map (hereinafter referred to as the second background feature correlation) can be determined respectively, such as... Figure 2 As shown. Based on the first background feature correlation and the second background feature correlation, the background correlation error between the first feature map and the second feature map in the background region is determined.

[0060] Taking any first background location point and a second background location point in a portrait image as examples, the feature value corresponding to the first background location point in the first feature map can be determined, thus obtaining the feature vector of the first background location point. Similarly, the feature value corresponding to the second background location point in the first feature map can be determined, thus obtaining the feature vector of the second background location point. When the first feature map is a multi-channel feature map, the feature vector of the first background location point can contain multiple feature values, arranged in channel order. Likewise, the feature vector of the second background location point can also contain multiple feature values, arranged in channel order.

[0061] Based on the similarity between the feature vectors of the first background location point and the feature vectors of the second background location point, the background feature correlation between the first background location point and the second background location point can be determined. In some exemplary embodiments, in the first feature map extracted by the teacher model, the similarity between the feature vectors of the first background location point and the feature vectors of the second background location point can be expressed by the following formula:

[0062]

[0063] in, This represents the eigenvector extracted by the teacher model at background location point i. This represents the eigenvector extracted by the teacher model at the background location point j; This represents the similarity between the feature vectors of background location point i and background location point j in the feature map extracted by the teacher's model.

[0064] In the second feature map extracted by the student model, the similarity between the feature vectors of the first background location points and the feature vectors of the second background location points can be expressed by the following formula:

[0065]

[0066] in, Let represent the eigenvector extracted by the student model at background location point i. R represents the eigenvector extracted by the student model at background location point j; stu_bg (Φ) i，j This represents the similarity between the feature vectors of background location point i and background location point j in the feature map extracted by the student model.

[0067] Based on the above implementation method, the similarity of feature vectors between any two background locations in the portrait image in the first feature map and the similarity of feature vectors between any two background locations in the second feature map can be calculated. Then, a first background feature correlation can be determined based on the similarity of feature vectors between any two background locations in the portrait image, and a second background feature correlation can be determined based on the similarity of feature vectors between any two background locations in the portrait image. Next, the foreground correlation error between the first and second feature maps in the background region can be calculated based on the first and second background feature correlations. The background correlation error describes the error in feature correlation between the background features extracted by the teacher model and the student model. Knowledge distillation based on the background correlation error allows the knowledge learned by the teacher model in background feature extraction to be transferred to the student model, thereby improving the student model's ability in background feature extraction.

[0068] Optionally, the background correlation error can be determined based on the difference between the first background feature correlation degree and the second background feature correlation degree. The larger the difference between the first background feature correlation degree and the second background feature correlation degree, the larger the background correlation error. In some embodiments, the L2 norm can be used to calculate the first background feature correlation degree and the second background feature correlation degree to obtain the background correlation error. In the calculation of the L2 norm, the square roots of the differences between the first background feature correlation degree and the second background feature correlation degree can be summed, and the square root of the summation result can be taken to obtain a correlation error with a positive result. Optionally, the specific formula for calculating the background correlation error based on the L2 norm is as follows:

[0069] Loss bg (i, j) = ||R tea_bg (Φ) i，j -R stu_bg (Φ) i，j ||2, i, j ∈ background region

[0070] Among them, Loss bg (i, j) represents the background correlation error of background location point i and background location point j.

[0071] The larger the background correlation error, the greater the difference in background feature representation ability between the teacher's model and the student's model. Therefore, during the training of the student model, the background correlation error can be converged as the optimization objective, and the parameters of the student model can be continuously adjusted until the background correlation error converges to the target range. Furthermore, the structural features of the background region can be learned in the portrait image, which is beneficial for more accurately identifying pixels of the background region from the portrait image.

[0072] It is worth noting that in some optional embodiments, when determining the feature relevance loss of the teacher model and the student model, the global difference between the first feature map and the second feature map can be further considered. Here, the global difference describes the difference in the ability of the teacher model and the student model to extract global features. By adding global difference as an optimization objective on top of foreground knowledge distillation and background knowledge distillation, the loss of global correlation information can be reduced, thereby further improving the feature extraction ability of the student model. An exemplary description follows.

[0073] Optionally, such as Figure 2 As shown, the global feature correlation degree of the first feature map and the global feature correlation degree of the second feature map can be calculated, and the global spatial domain loss is determined based on the difference between the global feature correlation degree of the first feature map and the global feature correlation degree of the second feature map. The global feature correlation degree is calculated for a single channel feature map and reflects the correlation between global features of the single channel feature map. Furthermore, the following will use a feature map of any channel as an example to illustrate an optional implementation method for determining the global feature correlation degree.

[0074] For any channel of the first feature map, the feature values ​​corresponding to the pixels in the portrait image in that feature map can be normalized to obtain the global feature correlation degree of that feature map. Optionally, assume that the first feature map is an h×w×C matrix, where h is the height of the feature map, w is the width of the first feature map, and C is the number of channels of the feature map. The global feature correlation degree of the feature map of channel c in the first feature map can be calculated using the following formula:

[0075]

[0076] in, This represents the feature value corresponding to position point i in the feature map of channel c extracted by the teacher network.

[0077] Accordingly, for any channel of the feature map extracted from the student model in the second feature map, the feature values ​​corresponding to the pixels in the portrait image in that feature map can be normalized to obtain the global feature correlation of that feature map. The second feature map has the same size as the first feature map, that is, the second feature map is also an h×w×C matrix. The global feature correlation of the feature map of channel c in the second feature map can be calculated using the following formula:

[0078]

[0079] in, This represents the feature value corresponding to position point i in the feature map of channel c extracted from the student network.

[0080] After determining the global feature correlation degree of the feature maps of multiple channels in the first feature map and the global feature correlation degree of the feature maps of multiple channels in the second feature map based on the aforementioned embodiments, the global spatial loss of the student model and the teacher model can be determined according to the global feature correlation degree of the feature maps of multiple channels in the first feature map and the global feature correlation degree of the feature maps of multiple channels in the second feature map.

[0081] In some optional embodiments, the global spatial loss can be calculated using KL divergence (Kullback-Leibler, also known as relative entropy), as shown in the following formula:

[0082]

[0083] Among them, Loss global Let C represent the global spatial loss, and C be the total number of channels in the feature map. In some optional embodiments, the feature correlation loss of the teacher model and the student model can be determined based on the weighted result of the foreground correlation error, background correlation error, and global spatial loss.

[0084] Optionally, for different image processing tasks, a loss function corresponding to the task type (such as...) can be combined. Figure 2 The task loss shown is used to determine the final training loss of the student model. For example, for image segmentation, the weighted result of foreground correlation error, background correlation error, global spatial domain loss, and segmentation loss can be calculated as the final training loss. Similarly, for image matting, the weighted result of foreground correlation error, background correlation error, global spatial domain loss, and matting loss can be calculated as the final training loss. Likewise, for image stylization, the weighted result of foreground correlation error, background correlation error, global spatial domain loss, and stylization loss can be calculated as the final training loss. In other words, the final training loss is... total It can be expressed using the following formula:

[0085] Loss total =α1·Loss fg +α2·Loss bg +α3·Loss global +Loss

[0086] Wherein, α1, α2, and α3 are weighting coefficients used to control the weight values ​​of the image processing task and the knowledge distillation task. Loss is the loss function of the image processing task. For example, Loss can be the cross-entropy loss corresponding to the image segmentation task, the L1 loss of the matting task, the L2 loss of the super-resolution task, or the style loss of the stylization task. This embodiment does not impose any restrictions.

[0087] Based on this implementation method, during the training process of the student model, global spatial domain loss is introduced for knowledge distillation. This allows the student model to learn to classify pixels as foreground or background while simultaneously learning to extract global correlation information on portrait images. This improves the overall feature extraction capability of the student model, enabling it to perform better in other image processing tasks (such as image segmentation and matting) after classifying pixels as foreground or background.

[0088] Figure 3 This is a schematic flowchart of an exemplary embodiment of the present application providing an image processing method, which may include:

[0089] Step 301: Obtain the portrait image to be processed.

[0090] Step 302: Input the portrait image to be processed into the image segmentation model to obtain the segmentation result of the portrait image; wherein, the image segmentation model is trained based on the feature correlation loss between the image segmentation model and the teacher model; the feature correlation loss includes: foreground correlation error, and / or, background correlation error; wherein, the foreground correlation error is determined from the first feature map extracted from the teacher model and the second feature map extracted from the image segmentation model based on the structural correlation of the foreground region in the portrait image sample; the background correlation error is determined from the first feature map and the second feature map based on the structural correlation of the background region in the portrait image sample.

[0091] The image segmentation model can run on a terminal device, which may include a user's mobile phone, tablet, or smart wearable device; this embodiment is not limited to any particular device. The image segmentation model is trained using a knowledge distillation method. As a student model, it can transfer knowledge under the guidance of the teacher model, thereby learning better feature representations. Specific training methods can be found in the preceding embodiments and will not be repeated here. In this embodiment, the image segmentation model running on the terminal device is trained using a knowledge distillation method. The segmentation model, as a student model, can be a lightweight model with fewer parameters, requiring less computation and data, thus allowing it to run smoothly on the terminal device and reducing network and time costs during image processing. During the training process of the image segmentation model, after obtaining the feature maps output by the teacher and student models based on the portrait image, the structural features of the portrait image can be used to determine the feature correlation loss between the student and teacher models through the structural correlation between the foreground and / or background regions in the portrait image. This feature correlation loss can be used to express the difference in feature representation between the student and teacher models. This prompts the student model to continuously learn the structural correlations of different regions and the ability to express the features of different regions under the guidance of the teacher model, thereby improving the student model's ability to distinguish between foreground and background regions. This allows the image segmentation model to achieve high image segmentation accuracy while meeting the requirements of lightweight design.

[0092] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0093] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0095] Figure 4 This illustration shows a structural diagram of a terminal device provided in an exemplary embodiment of this application. This terminal device is applicable to the image processing method provided in the foregoing embodiments. Figure 4 As shown, the terminal device includes: a memory 401, a processor 402, and a communication component 403.

[0096] Memory 401 is used to store computer programs and can be configured to store various other data to support operation on the terminal device. Examples of this data include instructions for any application or method used to operate on the terminal device.

[0097] Processor 402, coupled to memory 401, is used to execute a computer program in memory 401 for: acquiring a portrait image to be processed; inputting the portrait image to be processed into an image segmentation model to obtain a segmentation result of the portrait image; wherein the image segmentation model is trained based on a feature correlation loss between the image segmentation model and a teacher model; the feature correlation loss includes: foreground correlation error and / or background correlation error; wherein the foreground correlation error is determined from a first feature map extracted from the teacher model and a second feature map extracted from the image segmentation model based on the structural correlation of the foreground region in the portrait image sample; the background correlation error is determined from the first feature map and the second feature map based on the structural correlation of the background region in the portrait image sample.

[0098] Furthermore, such as Figure 4 As shown, the terminal device also includes other components such as a power supply component 404, a display component 405, and an audio component 406. Figure 4 The diagram only shows some components and does not mean that the terminal device only includes... Figure 4 The components shown.

[0099] Display component 405 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0100] Audio component 405 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device in which the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0101] In this embodiment, the image segmentation model running on the terminal device is trained using a knowledge distillation method. The segmentation model, acting as a student model, can be a lightweight model with fewer parameters, requiring less computation and data, thus allowing it to run smoothly on the terminal device and reducing network and time costs during image processing. During the training process, after obtaining the feature maps output by the teacher and student models based on the portrait image, the structural features of the portrait image can be utilized. By analyzing the structural correlation between the foreground and / or background regions in the portrait image, the feature correlation loss between the student and teacher models can be determined. This feature correlation loss can be used to express the difference in feature representation between the student and teacher models. This encourages the student model to continuously learn the structural correlations of different regions and its ability to represent features in different regions under the guidance of the teacher model, thereby improving the student model's ability to distinguish between foreground and background regions. This allows the image segmentation model to achieve high image segmentation accuracy while meeting the requirements of lightweight design.

[0102] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a terminal device in the above method embodiments.

[0103] Figure 5 This illustration shows a structural diagram of a server provided in an exemplary embodiment of this application, which is suitable for the model training method provided in the foregoing embodiments. Figure 5 As shown, the server includes: a memory 501, a processor 502, and a communication component 503.

[0104] Memory 501 is used to store computer programs and can be configured to store various other data to support operations on the server. Examples of this data include instructions for any application or method used to operate on the server.

[0105] Processor 502, coupled to memory 501, is used to execute a computer program in memory 501 for: inputting portrait images into a teacher model and a student model respectively to obtain a first feature map and a second feature map; determining the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the structural correlation of the foreground region in the portrait image; and / or determining the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the structural correlation of the background region in the portrait image; determining the feature correlation loss of the teacher model and the student model based on the foreground correlation error and / or the background correlation error; training the student model based on the feature correlation loss, and outputting the trained student model as an image segmentation model when the student model meets a preset convergence condition.

[0106] Optionally, when determining the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the structural correlation of the foreground region in the portrait image, the processor 502 is specifically configured to: determine the foreground region in the portrait image based on the segmentation ground truth value marked on the portrait image; determine the first foreground feature correlation degree of the foreground region in the portrait image on the first feature map and the second foreground feature correlation degree of the foreground region in the second feature map respectively; and determine the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map based on the difference between the first foreground feature correlation degree and the second foreground feature correlation degree.

[0107] Optionally, when determining the first foreground feature correlation of the foreground region in the portrait image in the first feature map, the processor 502 specifically performs the following steps: for any first foreground location point and second foreground location point in the portrait image, determine the feature values ​​corresponding to the first foreground location point and the second foreground location point in the first feature map, respectively, to obtain the feature vector of the first foreground location point and the feature vector of the second foreground location point; and determine the foreground feature correlation of the first foreground location point and the second foreground location point based on the similarity between the feature vectors of the first foreground location point and the feature vectors of the second foreground location point.

[0108] Optionally, when determining the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the structural correlation of the background region in the portrait image, the processor 502 is specifically configured to: determine the background region in the portrait image based on the segmentation ground truth value marked on the portrait image; determine the first background feature correlation degree of the background region in the portrait image on the first feature map and the second background feature correlation degree on the second feature map respectively; and determine the background correlation error of the background region in the portrait image on the first feature map and the second feature map based on the difference between the first background feature correlation degree and the second background feature correlation degree.

[0109] Optionally, when determining the feature correlation loss of the teacher model and the student model based on the foreground correlation error and / or the background correlation error, the processor 502 specifically performs the following steps: calculating the global feature correlation degree of the first feature map and the global feature correlation degree of the second feature map; determining the global spatial domain loss based on the difference between the global feature correlation degree of the first feature map and the global feature correlation degree of the second feature map; and determining the feature correlation loss of the teacher model and the student model based on the weighted result of the foreground correlation error, the background correlation error, and the global spatial domain loss.

[0110] Optionally, when calculating the global feature correlation degree of the first feature map, the processor 502 specifically performs mean normalization processing on the feature values ​​corresponding to the pixels in the portrait image in the feature map for any channel of the first feature map, to obtain the global feature correlation degree of the feature map.

[0111] Furthermore, such as Figure 5 As shown, the server also includes other components such as the power supply component 504. Figure 5 The diagram only shows some components and does not mean that the server only includes... Figure 5 The components shown.

[0112] In this embodiment, during the training process of the image segmentation model, after the server obtains the feature maps output by the teacher model and the student model based on the portrait image, it can utilize the structural features of the portrait image and determine the feature correlation loss between the student model and the teacher model through the structural correlation between the foreground and / or background regions in the portrait image. This feature correlation loss can be used to express the gap between the student model and the teacher model in feature representation. Therefore, under the guidance of the teacher model, the student model continuously learns the structural correlation of different regions and its feature representation ability for different regions, thereby improving the student model's ability to distinguish between foreground and background regions. This allows the image segmentation model to achieve high image segmentation accuracy while meeting the requirements of lightweight design.

[0113] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the server in the above method embodiments.

[0114] exist Figure 4 as well as Figure 5 In this context, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0115] exist Figure 4 as well as Figure 5In this context, the communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0116] exist Figure 4 as well as Figure 5 In this context, a power supply component is used to provide power to various components of the device in which it resides. A power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0117] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0118] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0122] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0123] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0125] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image processing method, characterized in that, include: Acquire the portrait image to be processed; The portrait image to be processed is input into the image segmentation model to obtain the segmentation result of the portrait image; The image segmentation model is trained based on the feature correlation loss between the image segmentation model and the teacher model; the feature correlation loss includes: foreground correlation error, and / or, background correlation error. The foreground correlation error is determined based on the structural correlation of the foreground region in the portrait image sample, from the first feature map extracted by the teacher model and the second feature map extracted by the image segmentation model; the background correlation error is determined based on the structural correlation of the background region in the portrait image sample, from the first feature map and the second feature map. The foreground relevance error is calculated based on the first foreground feature relevance of the foreground region in the first feature map and the second foreground feature relevance of the foreground region in the second feature map. The first foreground feature relevance is determined by calculating the similarity of the feature vectors of any two pixels in the foreground region in the first feature map, and the second foreground feature relevance is determined by calculating the similarity of the feature vectors of any two pixels in the foreground region in the second feature map. The background correlation error is calculated based on the first background feature correlation of the background region in the first feature map and the second background feature correlation of the background region in the second feature map. The first background feature correlation is determined by calculating the similarity of the feature vectors of any two pixels in the background region in the first feature map, and the second background feature correlation is determined by calculating the similarity of the feature vectors of any two pixels in the background region in the second feature map.

2. A model training method for image segmentation, characterized in that, include: The portrait images are input into the teacher model and the student model respectively to obtain the first feature map and the second feature map; Based on the structural correlation of the foreground region in the portrait image, the foreground correlation error of the foreground region in the portrait image on the first feature map and the second feature map is determined; And / or, based on the structural correlation of the background region in the portrait image, determine the background correlation error of the background region in the portrait image on the first feature map and the second feature map; Based on the foreground correlation error and / or the background correlation error, determine the feature correlation loss of the teacher model and the student model; The student model is trained based on the feature correlation loss, and when the student model meets the preset convergence condition, the trained student model is output as an image segmentation model. The foreground correlation error is calculated based on the first foreground feature correlation of the foreground region in the portrait image in the first feature map and the second foreground feature correlation of the foreground region in the second feature map. The first foreground feature correlation is determined by calculating the similarity of the feature vectors of any first foreground position point and any second foreground position point in the foreground region in the first feature map. The second foreground feature correlation is determined by calculating the similarity of the feature vectors of any first foreground position point and any second foreground position point in the foreground region in the second feature map. The background correlation error is calculated based on the first background feature correlation of the background region in the first feature map and the second background feature correlation of the background region in the second feature map. The first background feature correlation is calculated by comparing the feature vectors of any first background location point and any second background location point in the background region in the first feature map. The second background feature correlation is calculated by comparing the feature vectors of any first background location point and any second background location point in the background region in the second feature map.

3. The method according to claim 2, characterized in that, The foreground correlation error is determined based on the difference between the first foreground feature correlation and the second foreground feature correlation.

4. The method according to claim 2, characterized in that, The background correlation error is determined based on the difference between the first background feature correlation and the second background feature correlation.

5. The method according to any one of claims 2-4, characterized in that, Based on the foreground correlation error and / or the background correlation error, determine the feature correlation loss of the teacher model and the student model, including: Calculate the global feature correlation degree of the first feature map and the global feature correlation degree of the second feature map; The global spatial loss is determined based on the difference between the global feature correlation degree of the first feature map and the global feature correlation degree of the second feature map. The feature correlation loss of the teacher model and the student model is determined based on the weighted result of the foreground correlation error, the background correlation error, and the global spatial domain loss.

6. The method according to claim 5, characterized in that, Calculating the global feature correlation of the first feature map includes: For any channel of the first feature map, the feature values ​​corresponding to the pixels in the portrait image in the feature map are normalized by mean normalization to obtain the global feature correlation of the feature map.

7. A terminal device, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps in the method of claim 1.

8. A server, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for performing the steps of the method according to any one of claims 2-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 1-6.

Images