A lip segmentation method and system based on spatial weighting

By using a spatial weighting method in lip region segmentation, the focus weight of pixel points is calculated and the segmentation network is optimized, the problem of inaccurate segmentation in the prior art is solved, and the segmentation accuracy of lip region is significantly improved.

CN114092496BActive Publication Date: 2024-11-19XIAN UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111444534.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-30
Publication Date
2024-11-19
Estimated Expiration
2041-11-30

AI Technical Summary

Technical Problem

The existing lip area segmentation method has the problem of inaccurate segmentation, especially at the transition position of the lips and skin, and due to the problem of category imbalance, deep learning methods are difficult to achieve the expected results.

Method used

The spatial weighting-based lip-part cutting method is adopted to calculate the attention weight of pixel points by labeling the boundary prior information in the image and predicting the data distribution of feature maps, strengthen the weight of the boundary area and the center position of the category, and weaken the weight of the transition area. Use the Adam optimizer to optimize the segmentation network and improve segmentation accuracy.

Benefits of technology

It alleviates the problem of category imbalance, improves the accuracy of lip area segmentation, and enhances the segmentation effect of the transition position of the lips and skin.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114092496B_ABST
    Figure CN114092496B_ABST
Patent Text Reader

Abstract

The present invention relates to a lip segmentation method and system based on spatial weighting, the method comprising: obtaining an original image and an annotated image; inputting the original image into a segmentation network to obtain a feature map; calculating a weighted prediction loss according to the feature map and the annotated image; optimizing the parameters of the segmentation network using an Adam optimizer according to the weighted prediction loss to obtain an optimized segmentation network; and segmenting the image to be tested using the optimized segmentation network to obtain a segmented image. The method of the present invention alleviates the problem of class imbalance and improves the accuracy of lip region segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of lip region segmentation, and in particular to a lip region segmentation method and system based on spatial weighting. Background Art

[0002] As an upstream link of tasks such as identity authentication and semantic recognition, the accuracy of lip segmentation has an important impact on subsequent results. Accurate segmentation results can effectively eliminate the interference of noise around the lips, such as beards, and can also provide information about the inner and outer contours of the lips, thereby realizing high-level semantic information representation. Traditional lip segmentation methods usually distinguish the lip area from the skin area based on color information, but traditional lip segmentation methods are extremely dependent on image clarity and imaging lighting conditions, and usually cannot meet the actual task requirements. With the rapid development of deep learning technology, deep learning-based methods have achieved better performance than traditional lip segmentation methods, and work represented by the UNet segmentation network has achieved great success. However, due to the low proportion of the lip area in the face image, there are more pixels in the non-lip area category in the annotated image, and fewer pixels in the lip area category. Segmentation methods based on this type of annotated data usually face the problem of category imbalance. Therefore, the deep learning-based method is more accurate for the segmentation of lip areas with significant differences, but for the transition position between the lips and the skin, the segmentation results cannot meet expectations. Therefore, the existing lip area segmentation methods have the problem of inaccurate segmentation. Summary of the invention

[0003] The purpose of the present invention is to provide a lip segmentation method and system based on spatial weighting to solve the problem of inaccurate segmentation of the lip region segmentation method in the prior art.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A lip segmentation method based on spatial weighting, comprising:

[0006] Get the original image and the annotated image;

[0007] Inputting the original image into a segmentation network to obtain a feature map;

[0008] Calculating a weighted prediction loss based on the feature map and the annotated image;

[0009] According to the weighted prediction loss, the parameters of the segmentation network are optimized using an Adam optimizer to obtain an optimized segmentation network;

[0010] The optimized segmentation network is used to segment the image to be tested to obtain a segmented image.

[0011] Optionally, after inputting the original image into a segmentation network to obtain a feature map, the method further includes:

[0012] According to the feature map, the prediction probability of the pixel point in the original image is calculated; the prediction probability is the probability that the pixel point in the original image belongs to the lip point;

[0013] According to the prediction probability, a predicted segmentation image is obtained.

[0014] Optionally, the calculating the weighted prediction loss according to the feature map and the annotated image specifically includes:

[0015] Generating an attention weight of a pixel at a first spatial position according to the annotated image;

[0016] Performing segment-wise mapping on the pixel attention weights at the first spatial position to obtain pixel attention weights at the second spatial position;

[0017] Generating an attention weight of a pixel at a third spatial position according to the feature map;

[0018] The pixel attention weight at the second spatial position is combined with the pixel attention weight at the third spatial position to obtain a pixel position attention weight;

[0019] A weighted prediction loss is calculated based on the pixel position attention weight, the predicted probability and the actual probability; the actual probability is the probability that a pixel point in the annotated image belongs to a lip point.

[0020] Optionally, generating the attention weight of the pixel point at the first spatial position according to the annotated image specifically includes:

[0021] Using the formula Calculate the attention weight of the pixel at the first spatial position; where w con represents the attention weight of the pixel at the first spatial position; φ b Indicates that (x c ,y c ) is the center and the size of the window neighborhood is k × k. The segmentation category is different from the center pixel category. represents the coordinates of the ith pixel in the window neighborhood that is different from the central pixel category; φ represents the coordinate set of all pixels in the window neighborhood; (x i ,y i ) represents the coordinates of the i-th pixel in the window; (x c ,y c ) represents the coordinate position of the central pixel; σ represents the bandwidth of the Gaussian function.

[0022] Optionally, performing segmented mapping on the pixel attention weights at the first spatial position to obtain the pixel attention weights at the second spatial position specifically includes:

[0023] Using the formula The attention weights of the pixels at the first spatial position are segmented and mapped; wherein δ represents the threshold for identifying the pixels as noise; w con' is the attention weight of the pixel at the second spatial position.

[0024] Optionally, generating the attention weight of the pixel at the third spatial position according to the feature map specifically includes:

[0025] Clustering the pixels at different positions in the feature map using an unsupervised clustering method to obtain categories of the pixels; the categories of the pixels include pixels in the lip area and pixels in the skin area;

[0026] Using the formula Calculate the cluster center of the pixel points in the lip area; where f L Represents the cluster center of the pixels in the lip area, Indicates that the pixel points in the feature map are segmented into a set of pixel points in the lip area, N L represents the number of pixels segmented into the lip category, and D represents the number of channels of the predicted feature map;

[0027] Using the formula Calculate the cluster center of the pixels in the skin area; where f S Represents the cluster center of the pixels in the skin area, Indicates that the pixel points in the feature map are segmented into a set of skin area pixels, N S Indicates the number of pixels segmented into skin categories;

[0028] Using the formula Calculate the cluster center of the pixels in the transition area;

[0029] Using the formula The attention weight of the pixel point at the third spatial position is calculated according to the cluster center of the pixel points in the lip area, the cluster center of the pixel points in the skin area, and the cluster center of the pixel points in the transition area; wherein, w fea represents the attention weight of the pixel at the third spatial position; f i ∈L represents f i In the unsupervised clustering, the pixels are segmented into lip area pixels; f i ∈S represents f i In unsupervised clustering, pixels are segmented into skin area pixels; ρ is the weight coefficient of transition area pixels.

[0030] Optionally, the fusing the second spatial position pixel attention weight and the third spatial position pixel attention weight to obtain the pixel position attention weight specifically includes:

[0031] Using the formula w=αw con' +(1-α)w fea , calculate the pixel position attention weight; where w represents the pixel position attention weight, α is the fusion coefficient; w con' is the attention weight of the pixel at the second spatial position.

[0032] Optionally, the calculating the weighted prediction loss according to the pixel position attention weight, the prediction probability and the actual probability specifically includes:

[0033] Using the formula Calculate the weighted prediction loss; where loss represents the weighted prediction loss, N represents the number of training samples, and the training samples include the original image and the annotated image; M represents the number of pixels in the image, and w ij represents the attention weight of the jth pixel position in the i-th training sample, y ij represents the actual probability of the jth pixel in the i-th labeled image, Represents the predicted probability of the jth pixel in the i-th original image.

[0034] A lip segmentation system based on spatial weighting, comprising:

[0035] An image acquisition module, used to acquire original images and annotate images;

[0036] A prediction module, used for inputting the original image into a segmentation network to obtain a feature map;

[0037] A loss calculation module, used to calculate the weighted prediction loss according to the feature map and the annotated image;

[0038] An optimization module, used for optimizing the parameters of the segmentation network using an Adam optimizer according to the weighted prediction loss to obtain an optimized segmentation network;

[0039] The segmentation module is used to segment the image to be tested using the optimized segmentation network to obtain a segmented image.

[0040] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0041] The present invention utilizes the prior information of the boundaries in the annotated image and the data distribution of the predicted feature map to form a pixel position attention weight for each pixel according to the neighborhood characteristics and feature distribution characteristics of the pixel points, strengthens the pixel weights located in the boundary area and the category center position, weakens the transition weights between the different segmentation center areas of the lip, alleviates the problem of category imbalance, and improves the accuracy of lip area segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A flow chart of a lip segmentation method based on spatial weighting provided by the present invention;

[0044] Figure 2 A schematic diagram of a lip segmentation method based on spatial weighting provided by the present invention;

[0045] Figure 3 A schematic diagram of the segmentation principle provided by the present invention;

[0046] Figure 4 A schematic diagram of a method for generating position weights based on annotated boundary characteristics provided by the present invention;

[0047] Figure 5 The weights of the central pixels at different positions provided by the present invention;

[0048] Figure 6 A flow chart of a method for generating spatial position attention weights based on a feature map provided by the present invention;

[0049] Figure 7 A schematic diagram of attention weights and fusion weights generated based on different methods provided by the present invention;

[0050] Figure 8 A schematic diagram of the segmentation result of lip segmentation using the lip segmentation method based on spatial weighting provided by the present invention;

[0051] Fig. 9 A structural diagram of a lip segmentation system based on spatial weighting provided by the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The purpose of the present invention is to provide a lip segmentation method and system based on spatial weighting to solve the problem of inaccurate segmentation of the lip region segmentation method in the prior art.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Specific implementation method (I)

[0056] Figure 1 A flow chart of a lip segmentation method based on spatial weighting provided by the present invention is shown in FIG. Figure 1 As shown, a lip segmentation method based on spatial weighting includes:

[0057] Step 101: Obtaining an original image and an annotated image. In practical applications, the original image is manually annotated to obtain an annotated image.

[0058] Step 102: Input the original image into a segmentation network and output a feature map.

[0059] In this specific implementation, after step 102, the following steps are further included:

[0060] The prediction probability of a pixel point in the original image is calculated based on the feature map. The prediction probability is the probability that the pixel point in the original image belongs to a lip point.

[0061] According to the prediction probability, a predicted segmentation image is obtained.

[0062] Step 103: Calculate a weighted prediction loss based on the feature map and the labeled image.

[0063] In this specific implementation, step 103 specifically includes:

[0064] Generate a pixel attention weight at a first spatial position according to the annotated image.

[0065] In this specific implementation, generating the attention weight of the pixel point at the first spatial position according to the annotated image specifically includes:

[0066] Using the formula Calculate the attention weight of the pixel at the first spatial position.con represents the attention weight of the pixel at the first spatial position; φ b Indicates that (x c ,y c ) is the center and the size of the window neighborhood is k × k. The segmentation category is different from the center pixel category. represents the coordinates of the ith pixel in the window neighborhood that is different from the central pixel category; φ represents the coordinate set of all pixels in the window neighborhood; (x i ,y i ) represents the coordinates of the i-th pixel in the window; (x c ,y c ) represents the coordinate position of the central pixel; σ represents the bandwidth of the Gaussian function.

[0067] The step of performing segmented mapping on the pixel attention weights at the first spatial position to obtain the pixel attention weights at the second spatial position specifically includes:

[0068] Using the formula The attention weights of the pixels at the first spatial position are segmented and mapped; wherein δ represents the threshold for identifying the pixels as noise; w con' is the attention weight of the pixel at the second spatial position.

[0069] Generate an attention weight for a pixel at a third spatial position according to the feature map.

[0070] In this specific implementation, generating the attention weight of the pixel at the third spatial position according to the feature map specifically includes:

[0071] The pixels at different positions in the feature map are clustered using an unsupervised clustering method to obtain categories of the pixels. The categories of the pixels include lip area pixels and skin area pixels.

[0072] Using the formula Calculate the cluster center of the pixel points in the lip area. Where, f L Represents the cluster center of the pixels in the lip area, Indicates that the pixel points in the feature map are segmented into a set of pixel points in the lip area, N L It represents the number of pixels segmented into lip area pixels, and D represents the number of predicted feature map channels.

[0073] Using the formula Calculate the cluster center of the pixels in the skin area. S Represents the cluster center of the pixels in the skin area, Indicates that the pixel points in the feature map are segmented into a set of skin area pixels, N SIndicates the number of pixels segmented into skin area pixels.

[0074] Using the formula Calculate the cluster center of the pixels in the transition area.

[0075] Using the formula The attention weight of the pixel point at the third spatial position is calculated according to the cluster center of the pixel points in the lip area, the cluster center of the pixel points in the skin area, and the cluster center of the pixel points in the transition area. fea represents the attention weight of the pixel at the third spatial position; f i ∈L represents f i In the unsupervised clustering, the pixels are segmented into lip area pixels; f i ∈S represents f i In unsupervised clustering, pixels are segmented into skin area pixels; ρ is the weight coefficient of transition area pixels.

[0076] The pixel attention weight at the second spatial position and the pixel attention weight at the third spatial position are fused to obtain a pixel position attention weight.

[0077] In this specific implementation, the second spatial position pixel point attention weight and the third spatial position pixel point attention weight are fused to obtain the pixel position attention weight, specifically including:

[0078] Using the formula w=αw con' +(1-α)w fea , calculate the pixel position attention weight. Among them, w represents the pixel position attention weight, α is the fusion coefficient; w con' is the attention weight of the pixel at the second spatial position.

[0079] A weighted prediction loss is calculated according to the pixel position attention weight, the predicted probability and the actual probability. The actual probability is the probability that the pixel point in the annotated image belongs to the lip point.

[0080] In this specific implementation, the weighted prediction loss is calculated according to the pixel position attention weight, the prediction probability and the actual probability, specifically including:

[0081] Using the formula Calculate the weighted prediction loss. Wherein, loss represents the weighted prediction loss, N represents the number of training samples, and the training samples include the original image and the annotated image; M represents the number of pixels in the original image, and w ij represents the attention weight of the jth pixel position in the i-th training sample, y ij represents the actual probability of the jth pixel in the i-th labeled image, Represents the predicted probability of the jth pixel in the i-th original image.

[0082] Step 104: According to the weighted prediction loss, the parameters of the segmentation network are optimized using the Adam optimizer to obtain an optimized segmentation network.

[0083] Step 105: segment the image to be tested using the optimized segmentation network to obtain a segmented image. Specific implementation method (II)

[0085] Figure 2 A schematic diagram of a lip segmentation method based on spatial weighting provided by the present invention is shown in FIG. Figure 2 As shown, the second spatial position pixel attention weight is generated according to the boundary of the annotated image, the input original image is predicted through the segmentation network, and the feature map is output. The third spatial position pixel attention weight is calculated according to the distribution of each pixel in the predicted output feature map, and the second spatial position pixel attention weight and the third spatial position pixel attention weight are fused to obtain the pixel position attention weight. According to the feature map and the annotated image and combined with each pixel position attention weight, the weighted prediction loss is calculated to train the segmentation network. Finally, the trained segmentation network is used to segment the image to be tested to obtain the segmentation result.

[0086] The purpose of the present invention is to predict the lip region structure from a given image. Figure 3 In the segmentation method shown in the figure, the segmentation network uses the UNet network as the basic backbone network, and the segmentation network uses spatial weighted binary cross entropy as the optimization target. The spatial weight is generated through two groups of modules: (1) The attention weight of the pixel point at the second spatial position is generated based on the boundary characteristics of the annotated image; (2) The attention weight of the pixel point at the third spatial position is generated based on the predicted feature map. The final spatial weight is formed by the fusion of the above two weights.

[0087] UNet maps the input original image to a 64-channel feature map of the same scale as the original image through the encoder and decoder structure, and finally converts the 64-channel feature map into the final segmented image through 1×1 convolution. The original UNet network uses the cross entropy between the predicted segmented image and the actual segmented image as the network optimization target. The weights of the pixels at each position of the lip are set to be consistent. However, since the lip area accounts for a relatively low proportion in the image, the same weight will cause an imbalance problem in the positive and negative category samples during the model learning process.

[0088] The core idea of ​​the method for generating attention weights for pixels at the second spatial position based on the annotated boundary is that pixels at the category boundary are more important for constructing a segmentation decision network than pixels far away from the category boundary. The transition area from the skin to the lip at the lip boundary is not clearly distinguished. If the system can correctly segment this area, it will also have a good effect on distinguishing the more obvious lip core area and the skin core area. Based on this, the present invention designs a method for evaluating the importance of pixel positions, that is, first generating attention weights for pixels at the first spatial position according to the boundary characteristics of the annotated image, such as Figure 4 As shown in the figure, the importance of a pixel is evaluated by taking the pixel as the center pixel, taking a k×k window as the basis for evaluating the pixel weight, and using the distance metric between the segmentation category of the neighborhood pixels in the window and the segmentation category of the center pixel as the evaluation. The specific definition is as follows:

[0089]

[0090] Among them, φ b Indicates that (x c ,y c ) is the center and the size of the window neighborhood is k × k. The segmentation category is different from the center pixel category. represents the coordinates of a pixel in the window neighborhood that is different from the central pixel category; φ represents the set of all pixel coordinates in the window neighborhood; (x i ,y i ) represents the coordinates of a pixel point in the window; (x c ,y c ) represents the coordinate position of the central pixel; σ represents the bandwidth of the Gaussian function.

[0091] like Figure 5 As shown in , the center pixels at different positions have different weights, ranging from 0 to 1. When the categories of the neighboring pixels are all different from the category of the center pixel, the attention weight of the pixel at the first spatial position is 1; when the category of the center pixel is the same as that of the neighboring pixels, the attention weight of the pixel at the first spatial position is 0. However, in Figure 5 In (a), the central pixel is an isolated segmentation point, which usually corresponds to noise in the lip segmentation task. In order to remove the influence of segmentation noise, the present invention regards pixels whose attention weight of the first spatial position pixel is greater than a certain value as noise, and resets the weight of the noise pixel to 0. Figure 5As shown in (c), the center pixel neighborhood is all pixels of the same category. At this time, the center pixel can be considered to be located in the core area of ​​the segmentation target. At this time, the attention weight of the pixel at the first spatial position is 0. The weighted loss of the center pixel will not affect the model learning. In order to avoid losing the learning effect of the center pixel and shielding the interference of noise pixels, the following segmented mapping is performed on the generated attention weight of the pixel at the first spatial position to obtain the attention weight of the pixel at the second spatial position:

[0092]

[0093] Among them, δ represents the threshold for identifying whether a pixel is noise. If the attention weight of the pixel at the first spatial position is greater than the threshold, it is considered to be a noise point, and the attention weight of the pixel at the first spatial position is reset to 0; when the attention weight of the pixel at the first spatial position is equal to the threshold, the attention weight of the pixel at the first spatial position is updated to 1, which means that the pixel position is at the category boundary and should be given the maximum weight during the segmentation network training process; when the attention weight of the pixel at the first spatial position is 0, it means that the current pixel is the core area of ​​the segmentation target and should be given certain consideration during the segmentation network training. Through the above mapping, the attention weight of the pixel at the first spatial position is updated to 0.5. Through this segmented mapping, the segmentation noise is shielded, and the influence of the boundary and core areas is also considered in different situations.

[0094] The window size k for calculating the attention weight of the pixel point at the first spatial position also has a great influence on the weight construction. When the window is small, the attention weight generation of the pixel point at the first spatial position pays more attention to the short-distance detail information around the central pixel point; when the window is large, the attention weight generation of the pixel point at the first spatial position pays more attention to the structural information around the central pixel point. In order to comprehensively consider the detail information and structural information, the present invention adopts the following multi-scale method to construct the final attention weight of the pixel point at the first spatial position based on the boundary characteristics:

[0095]

[0096] in, and represents the center pixel weight calculated based on windows of different scales, k1<k2<k3. k1 Focus on detail weight, and w k3 Focus on structural weights.

[0097] The method for generating pixel attention weights at the second spatial position based on boundary characteristics only relies on the information of the annotated image and ignores the detailed information such as the texture of the input original image. In addition, the method for generating pixel attention weights at the second spatial position based on boundary characteristics assigns the smallest weight to the pixel value at the center of the category, and the category center sample defines the data distribution of typical samples of this type, which theoretically plays an important role in forming the decision space. Therefore, the present invention designs a method for generating pixel attention weights at the third spatial position based on a feature map, which uses the feature map of the input original image to comprehensively evaluate the sample distribution position, and greater attention should be given to the pixel values ​​at the center of the category and at the boundary of the category. The specific process is as follows: Figure 6 shown.

[0098] Since the predicted feature map has no actual category annotation, the pixels at different positions in the feature map are clustered according to the feature representation of the pixels through an unsupervised clustering method. The present invention uses the mean shift algorithm for classification (the clustering algorithm is not limited to the mean shift algorithm). Most pixels can be correctly classified through the unsupervised clustering method, and a small number of pixels located at the boundary may be segmented incorrectly. However, the present invention does not need to obtain accurate segmentation results in this step. Unlike other clustering methods, after the present invention determines the category of the pixel point, the cluster centers of different segmentation categories are calculated separately. Assume Indicates that the pixels in the feature map are segmented into a set of pixels in the lip area. Indicates that the pixels in the feature map are segmented into a set of skin area pixels. L Indicates the number of pixels segmented into lip area pixels, N S represents the number of pixels segmented into skin area pixels, and D represents the number of channels of the predicted feature map. Then the cluster centers of the lip area pixels and the skin area pixels can be calculated as follows:

[0099]

[0100]

[0101] The cluster center of the pixels in the transition area is the median of the cluster center of the pixels in the lip area and the cluster center of the pixels in the skin area:

[0102]

[0103] After calculating the three cluster centers, the pixel x i The attention weight of the pixel at the third spatial position can be calculated as follows:

[0104]

[0105] Among them, f i∈L represents f i In the unsupervised clustering, pixels are segmented into the lip area; f i ∈S represents f i In unsupervised clustering, the pixels are segmented into the skin area; ρ is the weight coefficient of the pixels in the transition area. The attention weight of the pixel at the third spatial position is calculated based on the weighted sum of the Gaussian distance between the current sample data and a certain type of cluster center and the Gaussian distance between the current sample data and the cluster center of the pixels in the transition area. In this way, in forming the attention weight of the pixel at the third spatial position, not only the distance of the current sample from the category center is considered, but also the distance from the transition center. When the sample is closer to the category center, the attention weight of the pixel at the third spatial position is larger; when the category is closer to the transition center, the attention weight of the pixel at the third spatial position is also larger.

[0106] like Figure 7 As shown, the final pixel position attention weight comprehensively considers the second spatial position pixel point attention weight generated based on the boundary feature and the third spatial position pixel point attention weight generated based on the feature map, and is calculated using the following fusion method:

[0107] w=αw con' +(1-α)w fea

[0108] Among them, α is the fusion coefficient.

[0109] After calculating the fusion weights, the segmentation network is optimized using weighted binary cross entropy, as follows:

[0110]

[0111] Among them, loss represents the weighted prediction loss, N represents the number of training samples, and the training samples include the original image and the annotated image; M represents the number of pixels in the original image, and w ij represents the attention weight of the jth pixel position in the i-th training sample, y ij represents the actual probability of the jth pixel in the i-th labeled image (if it is a lip pixel y ij =1, otherwise y ij =0), represents the predicted probability of the jth pixel in the i-th original image ( The larger the value, the more likely the prediction result is a lip pixel, otherwise it is a skin pixel).

[0112] After obtaining the weighted prediction loss, the Adam optimizer is used for optimization to obtain the optimized segmentation network parameters. The segmentation network with the optimized segmentation network parameters is directly used for image segmentation.

[0113] The segmentation result of the lip segmentation method based on spatial weighting of the present invention is as follows: Figure 8 shown.

[0114] Fig. 9 A structure diagram of a lip segmentation system based on spatial weighting provided by the present invention, such as Fig. 9 As shown, a lip segmentation system based on spatial weighting includes:

[0115] The image acquisition module 901 is used to acquire the original image and the annotated image.

[0116] The prediction module 902 is used to input the original image into a segmentation network to obtain a feature map.

[0117] The loss calculation module 903 is used to calculate the weighted prediction loss according to the feature map and the annotated image.

[0118] The optimization module 904 is used to optimize the parameters of the segmentation network using the Adam optimizer according to the weighted prediction loss to obtain an optimized segmentation network.

[0119] The segmentation module 905 is used to segment the image to be tested using the optimized segmentation network to obtain a segmented image.

[0120] The main work of the present invention is to design a lip segmentation method and system based on spatial weighting, the core of which includes a weight generation method based on annotated images and a weight generation method based on feature maps. Through the generated fusion weights, different attention is given to pixels at different positions during the training process, thereby solving the problem of imbalance in segmentation training data categories and inaccurate boundary segmentation.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0122] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A lip segmentation method based on spatial weighting, characterized in that: include: Get the original image and the annotated image; Inputting the original image into a segmentation network to obtain a feature map; Calculating a weighted prediction loss based on the feature map and the annotated image; According to the weighted prediction loss, the parameters of the segmentation network are optimized using an Adam optimizer to obtain an optimized segmentation network; Using the optimized segmentation network to segment the image to be tested to obtain a segmented image; After the original image is input into the segmentation network to obtain the feature map, the method further includes: According to the feature map, the prediction probability of the pixel point in the original image is calculated; the prediction probability is the probability that the pixel point in the original image belongs to the lip point; According to the predicted probability, a predicted segmented image is obtained; Calculating the weighted prediction loss according to the feature map and the annotated image specifically includes: Generating an attention weight of a pixel at a first spatial position according to the annotated image; Performing segment-wise mapping on the pixel attention weights at the first spatial position to obtain pixel attention weights at the second spatial position; Generating an attention weight of a pixel at a third spatial position according to the feature map; The pixel attention weight at the second spatial position is combined with the pixel attention weight at the third spatial position to obtain a pixel position attention weight; Calculating a weighted prediction loss according to the pixel position attention weight, the prediction probability and the actual probability; the actual probability is the probability that the pixel point in the annotated image belongs to the lip point; Generating a pixel attention weight at a first spatial position according to the annotated image specifically includes: Using the formula Calculate the attention weight of the pixel at the first spatial position; where w con represents the attention weight of the pixel at the first spatial position; φ b Indicates that (x c ,y c ) is the center and the size of the window neighborhood is k × k. The segmentation category is different from the center pixel category. represents the coordinates of the ith pixel in the window neighborhood that is different from the central pixel category; φ represents the coordinate set of all pixels in the window neighborhood; (x i ,y i ) represents the coordinates of the i-th pixel in the window; (x c ,y c ) represents the coordinate position of the central pixel; σ represents the bandwidth of the Gaussian function; The pixel attention weights at the first spatial position are segmented and mapped to obtain pixel attention weights at the second spatial position, specifically comprising: Using the formula The attention weight of the pixel point at the first spatial position is segmented and mapped; wherein δ represents the threshold for identifying the pixel point as noise; wcon' is the attention weight of the pixel point at the second spatial position; Generating an attention weight of a pixel at a third spatial position according to the feature map specifically includes: Clustering the pixels at different positions in the feature map using an unsupervised clustering method to obtain categories of the pixels; the categories of the pixels include pixels in the lip area and pixels in the skin area; Using the formula Calculate the cluster center of the pixel points in the lip area; where f L Represents the cluster center of the pixels in the lip area, Indicates that the pixel points in the feature map are segmented into a set of pixel points in the lip area, N L represents the number of pixels segmented into the lip category, and D represents the number of channels of the predicted feature map; Using the formula Calculate the cluster center of the pixels in the skin area; where f S Represents the cluster center of the pixels in the skin area, Indicates that the pixel points in the feature map are segmented into a set of skin area pixels, N S Indicates the number of pixels segmented into skin categories; Using the formula Calculate the cluster center of the pixels in the transition area; Using the formula The attention weight of the pixel point at the third spatial position is calculated according to the cluster center of the pixel points in the lip area, the cluster center of the pixel points in the skin area, and the cluster center of the pixel points in the transition area; wherein, w fea represents the attention weight of the pixel at the third spatial position; f i ∈L represents f i In unsupervised clustering, the pixels are segmented into lip area pixels; f i ∈S represents f i In unsupervised clustering, the pixels are segmented into skin area pixels; ρ is the weight coefficient of transition area pixels; The pixel attention weight at the second spatial position and the pixel attention weight at the third spatial position are merged to obtain the pixel position attention weight, specifically comprising: Use the formula w=αw con '+(1-α)w fea , calculate the pixel position attention weight; where w represents the pixel position attention weight, and α is the fusion coefficient; Calculating the weighted prediction loss according to the pixel position attention weight, the predicted probability and the actual probability specifically includes: Using the formula Calculate the weighted prediction loss; where loss represents the weighted prediction loss, N represents the number of training samples, and the training samples include the original image and the annotated image; M represents the number of pixels in the image, and w ij represents the attention weight of the jth pixel position in the i-th training sample, y ij represents the actual probability of the jth pixel in the i-th labeled image, Represents the predicted probability of the jth pixel in the i-th original image.

2. A lip segmentation system based on spatial weighting, characterized in that: include: An image acquisition module, used to acquire original images and annotate images; A prediction module, used for inputting the original image into a segmentation network to obtain a feature map; A loss calculation module, used to calculate the weighted prediction loss according to the feature map and the annotated image; An optimization module, used for optimizing the parameters of the segmentation network using an Adam optimizer according to the weighted prediction loss to obtain an optimized segmentation network; A segmentation module, used to segment the image to be tested using the optimized segmentation network to obtain a segmented image; After the original image is input into the segmentation network to obtain the feature map, the method further includes: According to the feature map, the prediction probability of the pixel point in the original image is calculated; the prediction probability is the probability that the pixel point in the original image belongs to the lip point; According to the predicted probability, a predicted segmented image is obtained; Calculating the weighted prediction loss according to the feature map and the annotated image specifically includes: Generating an attention weight of a pixel at a first spatial position according to the annotated image; Performing segment-wise mapping on the pixel attention weights at the first spatial position to obtain pixel attention weights at the second spatial position; Generating an attention weight of a pixel at a third spatial position according to the feature map; The pixel attention weight at the second spatial position is combined with the pixel attention weight at the third spatial position to obtain a pixel position attention weight; Calculating a weighted prediction loss according to the pixel position attention weight, the prediction probability and the actual probability; the actual probability is the probability that the pixel point in the annotated image belongs to the lip point; Generating a pixel attention weight at a first spatial position according to the annotated image specifically includes: Using the formula Calculate the attention weight of the pixel at the first spatial position; where w con represents the attention weight of the pixel at the first spatial position; φ b Indicates that (x c ,y c ) is the center and the size of the window neighborhood is k × k. The segmentation category is different from the center pixel category. represents the coordinates of the ith pixel in the window neighborhood that is different from the central pixel category; φ represents the coordinate set of all pixels in the window neighborhood; (x i ,y i ) represents the coordinates of the i-th pixel in the window; (x c ,y c ) represents the coordinate position of the central pixel; σ represents the bandwidth of the Gaussian function; The pixel attention weights at the first spatial position are segmented and mapped to obtain pixel attention weights at the second spatial position, specifically comprising: Using the formula The attention weight of the pixel point at the first spatial position is segmented and mapped; wherein δ represents the threshold for identifying the pixel point as noise; wcon' is the attention weight of the pixel point at the second spatial position; Generating an attention weight of a pixel at a third spatial position according to the feature map specifically includes: Clustering the pixels at different positions in the feature map using an unsupervised clustering method to obtain categories of the pixels; the categories of the pixels include pixels in the lip area and pixels in the skin area; Using the formula Calculate the cluster center of the pixel points in the lip area; where f L Represents the cluster center of the pixels in the lip area, Indicates that the pixel points in the feature map are segmented into a set of pixel points in the lip area, N L represents the number of pixels segmented into the lip category, and D represents the number of channels of the predicted feature map; Using the formula Calculate the cluster center of the pixels in the skin area; where f S Represents the cluster center of the pixels in the skin area, Indicates that the pixel points in the feature map are segmented into a set of skin area pixels, N S Indicates the number of pixels segmented into skin categories; Using the formula Calculate the cluster center of the pixels in the transition area; Using the formula The attention weight of the pixel point at the third spatial position is calculated according to the cluster center of the pixel points in the lip area, the cluster center of the pixel points in the skin area, and the cluster center of the pixel points in the transition area; wherein, w fea represents the attention weight of the pixel at the third spatial position; f i ∈L represents f i In unsupervised clustering, the pixels are segmented into lip area pixels; f i ∈S represents f i In unsupervised clustering, the pixels are segmented into skin area pixels; ρ is the weight coefficient of transition area pixels; The pixel attention weight at the second spatial position and the pixel attention weight at the third spatial position are merged to obtain the pixel position attention weight, specifically comprising: Use the formula w=αw con '+(1-α)w fea , calculate the pixel position attention weight; where w represents the pixel position attention weight, and α is the fusion coefficient; Calculating the weighted prediction loss according to the pixel position attention weight, the predicted probability and the actual probability specifically includes: Using the formula Calculate the weighted prediction loss; where loss represents the weighted prediction loss, N represents the number of training samples, and the training samples include the original image and the annotated image; M represents the number of pixels in the image, and w ij represents the attention weight of the jth pixel position in the i-th training sample, y ij represents the actual probability of the jth pixel in the i-th labeled image, Represents the predicted probability of the jth pixel in the i-th original image.