Auricle anaphora segmentation method and system

Abstract

The invention provides an auricle anaphora segmentation method and system, and the method comprises the steps: inputting a to-be-segmented human ear image and text description into an auricle anaphora segmentation model which comprises a text coding module, a text-guided visual coding module and visual decoding module, and an angle transformation module; the text coding module obtains a text feature # imgabs0 #; a visual coding module realizes fusion of text features and image features through a structure organized into four stages, a visual encoder in each stage generates visual features # imgabs1 #, a cross-modal sensing module aligns # imgabs2 # and # imgabs3 # to obtain multi-modal features # imgabs4 # and # imgabs5 #, each element in the multi-modal features # imgabs4 # and # imgabs5 # is weighted by an attention gating module to obtain weighted multi-modal features # imgabs6 #, and the weighted multi-modal features # imgabs6 # and # imgabs5 # are weighted by an attention gating module. Adding the elements and # imgabs7 # to generate an enhanced visual feature # imgabs8 #, inputting # imgabs9 # into a visual decoding module to gradually recover the spatial resolution of the image, and further fusing the text and the visual feature to output a multi-scale feature; the angle transformation module carries out angle transformation on the multi-scale features and outputs a segmentation mask of a text description related region. According to the invention, the auricle can be subjected to anaphora segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of auricle reference segmentation, and in particular to an auricle reference segmentation method and system. Background Art

[0002] As a biological feature, the human ear is considered to be a highly reliable method of identity recognition because of its relatively stable shape, significant individual differences, and is not easily affected by expressions and age changes. It can be used as an auxiliary means of identity recognition when facial biometrics are limited and face recognition is inconvenient. In addition, in the medical field, it is believed that by observing the color, shape, texture, etc. of the auricle, combined with changes in specific reflection points, it can assist in judging the function of internal organs or physical health. In the intelligent process combining AI technology, instance segmentation based on auricle images and image feature analysis of regions of interest are very important links. In the process of image segmentation, the referential segmentation method that adds language or text information to guide the segmentation results is in great demand in actual application scenarios of human-computer interaction.

[0003] However, the "auricle inclination" (i.e., the rotation angle of the external ear on the side of the skull) of different individuals varies from 0 to 70 degrees, and the auricle includes roughly divided parts such as the helix, antihelix, tragus, antitragus, earlobe, cavum concha and cymba concha, which can be divided into dozens of parts in detail. These parts are visually densely distributed, that is, the structures are closely adjacent, which will cause the masks of various regions to overlap during image segmentation, resulting in poor segmentation results. The similar texture features of various parts of the human ear further increase the difficulty of the segmentation task. In addition, the actual segmentation task may be affected by a variety of interference factors. For example, changes in illumination can cause the ear features to be blurred, and changes in the posture of the human ear may cause the shape of the auricle to deform. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides an auricle reference segmentation method, and the technical solution is as follows:

[0005] In one aspect, a method for segmenting an auricle is provided, the method comprising:

[0006] S1, obtaining the human ear image to be segmented and the corresponding text description;

[0007] S2, inputting the human ear image to be segmented and the corresponding text description into an auricle reference segmentation model based on text guidance and angle transformation, wherein the auricle reference segmentation model includes a text encoding module, a text-guided visual encoding module, a text-guided visual decoding module and an angle transformation module;

[0008] The text encoding module embeds the text description into a high-dimensional word vector to obtain text features ;

[0009] The text-guided visual encoding module realizes the fusion of text features and image features by organizing into a four-stage structure. The four stages are connected in series, and the output of the previous stage is used as the input of the next stage to realize multi-stage feature fusion. Each stage includes a visual encoder, a cross-modal perception module and an attention gating module. The visual encoder of each stage generates visual features. The cross-modal perception module aligns and fuses the visual features across modalities. With text features , get multimodal features , the multimodal features Each element in is weighted by the attention gating module to obtain weighted multimodal features , by element and original visual features Added together, they produce a set of enhanced visual features that embed linguistic information , the smallest scale enhanced visual features produced by the last stage inputting the text-guided visual decoding module;

[0010] The text-guided visual decoding module gradually restores the spatial resolution of the image, and further integrates text features and visual features to provide high-quality feature representation for the final segmentation task and output multi-scale features;

[0011] The angle transformation module performs angle transformation on the multi-scale features and outputs a segmentation mask of an area related to the text description.

[0012] Optionally, the text encoding module uses BERT to perform text encoding and embeds the text description into a high-dimensional word vector to obtain the text feature: ,in and N represent the number of word embedding channels and the number of words contained in the text description, respectively.

[0013] Optionally, the visual encoder adopts a Swin-Transformer encoder, and the input feature map is divided into windows of fixed size. The features in each window are modeled through a multi-head self-attention mechanism to capture the relationship between local areas. After the modeling is completed at the window level, the position of the window will be periodically translated to achieve global context association across windows. In the specific encoding process, the feature map is first mapped to a high-dimensional space through a linear embedding module, and then feature extraction and nonlinear transformation are performed layer by layer through a sliding window attention module and a multi-layer perceptron to enhance the expression ability of high-level semantic features and output the visual feature map. .

[0014] Optionally, the cross-modal perception module uses the visual features As a query, the text feature As the key and value, we use the scaled dot product attention to calculate the process of supervising visual features with text features, and promote the fusion of multimodal information. After calculating the scaled dot product attention, we will obtain the features With input visual features Perform the projection operation again to obtain and , and then perform a dot product operation to finally obtain the multimodal feature , the process formula is as follows:

[0015] (1)

[0016] (2)

[0017] (3)

[0018] (4)

[0019] (5)

[0020] (6)

[0021] (7)

[0022] in, , , , , is the projection function, Represents the dot product operation, key function Sum value function Is has 1×1 convolution with the number of output channels, query function It is also a 1×1 convolution, and the number of output channels remains the same as the text feature. When the attention is calculated, the two-dimensional visual feature map is cut and arranged into one-dimensional sequence data to adapt to the subsequent attention calculation operation. After the attention calculation is completed, the one-dimensional sequence data is restored to the form of a two-dimensional feature map.

[0023] Optionally, the attention gating module learns a set of attention gating modules based on the multimodal features. The weight map is used to rescale Each element in , to prevent multimodal features Overwhelming visual features The visual signal in the image is used to generate the image and allow an adaptive amount of textual semantic information to flow to the next stage. The specific process is as follows:

[0024] The multimodal features Perform 1×1 convolution and ReLu activation, and then perform 1×1 convolution and Tanh activation again. In this process, 1×1 convolution adjusts the number of channels to achieve the reorganization and weight distribution of channel information without changing the spatial resolution of the input feature map. The ReLu and Tanh activation functions can increase the nonlinear ability of the model and generate multimodal features. The weight is multiplied by the original feature map to obtain the weighted multimodal feature .

[0025] Optionally, the text-guided visual decoding module consists of three stages, and the input of each stage is the fusion feature output of the previous stage And the multimodal features of the corresponding stages in the visual encoding module ;

[0026] In each stage, the visual decoder first upsamples the feature map of the previous stage through bilinear interpolation, gradually expands the spatial resolution of the feature map, restores the spatial information of the original image, and lays the foundation for subsequent image segmentation;

[0027] The upsampled features and the multimodal features of the corresponding stage of the visual encoding module The addition is performed in the channel dimension to further realize the deep fusion of visual features and text features, make full use of the semantic information in the text features to guide the restoration of visual features, enhance the sensitivity of the segmentation task to details and semantic boundaries, and use the multimodal features Through layer-by-layer fusion and upsampling operations, the features output by the visual decoding module gradually approach the original resolution of the input image, and provide a more delicate and semantically clear feature representation for the subsequent angle transformation module;

[0028] During the entire decoding process, the visual decoding module uses the smallest scale enhanced visual features output by the visual encoding module. As input, after three levels of decoding, three larger-scale feature maps are gradually generated, which together with the initial input feature map constitute four-scale feature maps. Finally, the size of the feature maps of the four scales is consistent with the feature map corresponding to the visual encoding module. Figure 1 It provides structural support for feature fusion and segmentation, and finally outputs multi-scale features. .

[0029] Optionally, the angle transformation module includes three angle transformation layers, three nonlinear projection layers and one linear projection layer, and receives the multi-scale features output by the text-guided visual decoding module. As input, the final mask prediction is output. The whole process is shown in the following formula:

[0030] (8)

[0031] (9)

[0032] (10)

[0033] in, Represents the smallest feature map of the input, represents the input of each angle transformation layer, Represents the concat operation, which realizes the feature fusion in the channel dimension. The scale of feature maps of different scales is unified through bilinear interpolation. The process includes 3×3 convolutional layers, batch normalization layers, and ReLU activation functions, corresponding to nonlinear projection layers. represents the angle transformation layer, represents the final segmentation mask, is the linear projection function, and the feature Mapped to the final segmentation mask.

[0034] Optionally, the angle transformation layer captures angle information from the input features to perform angle transformation, and dynamically re-parameterizes the convolution kernel weights to filter out redundant features through the updated convolution kernel weights. The process is as follows:

[0035] The feature map is fed into a depthwise separable convolutional layer and an average pooling layer to further extract features, and then fed into a linear prediction layer, which includes a fully connected layer and an activation function to predict n angle parameters based on the extracted features. and the corresponding weight parameters , , according to the coordinates of the points on the feature map Perform coordinate transformation to obtain , coordinates after angle transformation The weights will fall to the new position, and the bilinear interpolation will ensure the continuity of weight changes to prevent the loss of weight information due to angle changes. Finally, the updated convolution kernel weights, weight parameters and original visual features are sent to the convolution layer to filter redundant features and generate new features. The formula is as follows:

[0036] (11)

[0037] (12)

[0038] (13)

[0039] in, is the coordinate of the point on the original feature map, is the coordinate after angle transformation, is the inverse matrix of the angular affine transformation rotation matrix, is bilinear interpolation, represents the original static convolution kernel weight, According to the transformed The original weight To update, is the original feature map, is the weighted feature map.

[0040] Optionally, the training process of the auricle representation segmentation model uses instance contrast learning to improve the model's discriminability for different instances and its robustness to different languages ​​describing the same instance;

[0041] The instance contrastive learning samples b text descriptions of an image. These text descriptions may point to the same instance or different instances in the image. The model generates b mask predictions based on the image and b text descriptions, making the masks of sentences describing the same instance similar and suppressing the overlap of masks of different instances. The overlap score is obtained by calculating the overlap between each two masks. The i-th and j-th final masks are and The overlap fraction is calculated by the following formula:

[0042] (14)

[0043] in, , n is the pixel size of feature map S, is the i-th segmentation mask The normalized value at position n is used to calculate the instance contrast loss based on the overlap score. , which is used to enhance the model's ability to distinguish different instances. The formula is as follows:

[0044] (15)

[0045] in, express and Whether they correspond to the same instance, is the i-th mask The intersection-over-union ratio between the actual annotation and the original annotation is used to prevent incorrect mask prediction;

[0046] The loss function of the ear pinna segmentation model uses a joint optimization loss function to combine the binary cross entropy loss, Dice loss and the instance contrast loss Combined with the above, the segmentation accuracy and boundary distinction ability of the model are improved, which can be expressed as:

[0047] (16)

[0048] in, represents the binary cross entropy loss, represents the Dice loss, Denotes the instance contrast loss.

[0049] In another aspect, a system for segmenting auricle reference is provided, the system comprising:

[0050] An acquisition module, used for acquiring a human ear image to be segmented and a corresponding text description;

[0051] A segmentation module, used for inputting the human ear image to be segmented and the corresponding text description into an auricle reference segmentation model based on text guidance and angle transformation, wherein the auricle reference segmentation model includes a text encoding module, a text-guided visual encoding module, a text-guided visual decoding module and an angle transformation module;

[0052] The text encoding module embeds the text description into a high-dimensional word vector to obtain text features ;

[0053] The text-guided visual encoding module realizes the fusion of text features and image features by organizing into a four-stage structure. The four stages are connected in series, and the output of the previous stage is used as the input of the next stage to realize multi-stage feature fusion. Each stage includes a visual encoder, a cross-modal perception module and an attention gating module. The visual encoder of each stage generates visual features. The cross-modal perception module aligns and fuses the visual features across modalities. With text features , get multimodal features , the multimodal features Each element in is weighted by the attention gating module to obtain weighted multimodal features , by element and original visual features Added together, they produce a set of enhanced visual features that embed linguistic information , the smallest scale enhanced visual features produced by the last stage inputting the text-guided visual decoding module;

[0054] The text-guided visual decoding module gradually restores the spatial resolution of the image, and further integrates text features and visual features to provide high-quality feature representation for the final segmentation task and output multi-scale features;

[0055] The angle transformation module performs angle transformation on the multi-scale features and outputs a segmentation mask of an area related to the text description.

[0056] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned auricle reference segmentation method.

[0057] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned auricle reference segmentation method.

[0058] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0059] 1) The present invention achieves a more comprehensive understanding of the ear region by introducing a text-guided mechanism, combining the high-level semantic information contained in the natural language description with the color and texture features of the RGB image. This method is particularly suitable for scenes where ear features are not obvious or the background is complex, and effectively improves the robustness and accuracy of the segmentation model in complex scenes. Specifically, by using a high-resolution imaging device to collect human ear images as input, the collected images are fine-grained mask annotations are performed, and the text description is used to provide high-level semantic information for the ear area and features. The visual encoding module is used to extract visual features from the human ear image, and the text encoding module is used to encode the text description to obtain the corresponding semantic embedding representation. Then, the cross-modal perception module is used to combine the multi-scale image features with the text feature image to obtain multi-modal features, establish the association between pixel-level features and high-level semantic information, and enhance the segmentation ability of the model. Then, the multi-modal features are sent to the attention gating module, the multi-modal features are weighted, and the original visual features are accumulated element by element. Then, the visual decoding module upsamples the features sent to the visual encoding module and gradually increases the scale of the feature map. During the upsampling process, it will be fused with the multi-modal features of the encoding module in the multi-scale channel dimension, achieving a high alignment of text semantic information and visual semantic information, which is helpful for subsequent fine-grained segmentation.

[0060] 2) The angle transformation module designed by the present invention corrects the deformation characteristics of the ear in the input image at different inclinations and postures, thereby enhancing the posture adaptability of the segmentation model. Combining multi-view features with text guidance, the model can more accurately segment diverse auricle areas.

[0061] 3) The present invention designs a text-guided segmentation strategy, performs fine-grained annotation of the ear's fine-grained structures, and introduces an example comparative learning method to improve the model's boundary distinction capability, significantly reducing the mask overlap between dense structures of the ear, thereby achieving accurate segmentation of key parts such as the helix and tragus.

[0062] 4) The present invention designs a text-guided ear image segmentation method, which facilitates human-computer interaction and is more suitable for task scenarios with high requirements for human-computer interaction. For example, in the medical field, the positioning of ear acupoints is a skill that requires experience accumulation, and the text-guided segmentation method provides clear segmentation results and visual semantic annotations, and can dynamically adjust the segmentation target area according to the text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0064] Figure 1 is a flow chart of an auricle reference segmentation method provided by an embodiment of the present invention;

[0065] Figure 2 It is an overall block diagram of a cross-modal outer ear region segmentation and key point positioning method provided by an embodiment of the present invention;

[0066] Figure 3 is a structural block diagram of a cross-modal perception module provided by an embodiment of the present invention;

[0067] Figure 4 is a structural block diagram of an attention gating module provided by an embodiment of the present invention;

[0068] Figure 5 is a structural block diagram of an angle conversion module provided by an embodiment of the present invention;

[0069] Figure 6 is a structural block diagram of an angle conversion layer provided by an embodiment of the present invention;

[0070] Figure 7 is a schematic diagram of example comparative learning provided by an embodiment of the present invention;

[0071] Figure 8 is a block diagram of an auricle reference segmentation system provided by an embodiment of the present invention;

[0072] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0074] An embodiment of the present invention provides an auricle reference segmentation method, which can be implemented by an electronic device, and the electronic device can be a terminal or a server. Figure 1 The flowchart of this method is shown in FIG. Figure 2 The overall block diagram of the method is shown below. The processing flow may include the following steps:

[0075] S1, obtaining the human ear image to be segmented and the corresponding text description;

[0076] The embodiment of the present invention collects the human ear image to be segmented through a high-definition camera and obtains the corresponding text description. The text description of the ear acupoint area carries the corresponding organ attributes, position attributes or shape attributes, etc. For example, the text description of the concha cavity is: hemispherical acupoint related to the heart, acupoint related to the heart, acupoints under the ear nail boat.

[0077] S2, inputting the ear image to be segmented and the corresponding text description into an auricle reference segmentation model based on text guidance and angle transformation, wherein the auricle reference segmentation model includes a text encoding module, a text-guided visual encoding module, a text-guided visual decoding module and an angle transformation module, such as Figure 2 As shown;

[0078] The text encoding module embeds the text description into a high-dimensional word vector to obtain text features ;

[0079] The text-guided visual encoding module realizes the fusion of text features and image features by organizing into a four-stage structure. The four stages are connected in series, and the output of the previous stage is used as the input of the next stage to realize multi-stage feature fusion. Each stage includes a visual encoder, a cross-modal perception module and an attention gating module. The visual encoder of each stage generates visual features. (The four-stage visual encoder generates visual features at four scales ), the cross-modal perception module aligns and fuses the visual features across modalities With text features (To improve the accuracy of the model, the multimodal features are aligned and fused in all four stages so that the text semantic information can effectively supervise the feature extraction process of the visual encoding module) to obtain the multimodal features , the multimodal features Each element in is weighted by the attention gating module to obtain weighted multimodal features , by element and original visual features Added together, they produce a set of enhanced visual features that embed linguistic information , the smallest scale enhanced visual features produced by the last stage inputting the text-guided visual decoding module;

[0080] The text-guided visual decoding module gradually restores the spatial resolution of the image, and further integrates text features and visual features to provide high-quality feature representation for the final segmentation task and output multi-scale features;

[0081] The angle transformation module performs angle transformation on the multi-scale features and outputs a segmentation mask of an area related to the text description.

[0082] Optionally, the text encoding module uses BERT to perform text encoding and embeds the text description into a high-dimensional word vector to obtain the text feature: ,in and N represent the number of word embedding channels and the number of words contained in the text description, respectively.

[0083] Optionally, the visual encoder adopts a Swin-Transformer encoder, and the input feature map is divided into windows of fixed size. The features in each window are modeled through a multi-head self-attention mechanism to capture the relationship between local areas. After the modeling is completed at the window level, the position of the window will be periodically translated to achieve global context association across windows (solving the problem of information isolation in traditional windowing methods). In the specific encoding process, the feature map is first mapped to a high-dimensional space through a linear embedding module, and then feature extraction and nonlinear transformation are performed layer by layer through a sliding window attention module and a multi-layer perceptron to enhance the expression ability of high-level semantic features and output the visual feature map. .

[0084] Alternatively, if Figure 3 As shown, the cross-modal perception module uses the visual features As a query, the text feature As the key and value, we use the scaled dot product attention to calculate the process of supervising visual features with text features, and promote the fusion of multimodal information. After calculating the scaled dot product attention, we will obtain the features With input visual features Perform the projection operation again to obtain and , and then perform a dot product operation to finally obtain the multimodal feature , the process formula is as follows:

[0085] (1)

[0086] (2)

[0087] (3)

[0088] (4)

[0089] (5)

[0090] (6)

[0091] (7)

[0092] in, , , , , is the projection function, Represents the dot product operation, key function Sum value function Is has 1×1 convolution with the number of output channels, query function It is also a 1×1 convolution, and the number of output channels remains the same as the text feature. When the attention is calculated, the two-dimensional visual feature map is cut and arranged into one-dimensional sequence data to adapt to the subsequent attention calculation operation. After the attention calculation is completed, the one-dimensional sequence data is restored to the form of a two-dimensional feature map.

[0093] Alternatively, if Figure 4 As shown, the attention gating module learns a set of multimodal features based on The weight map is used to rescale Each element in , to prevent multimodal features Overwhelming visual features The visual signal in the image is used to generate the image and allow an adaptive amount of textual semantic information to flow to the next stage. The specific process is as follows:

[0094] The multimodal features Perform 1×1 convolution and ReLu activation, and then perform 1×1 convolution and Tanh activation again. In this process, 1×1 convolution adjusts the number of channels to achieve the reorganization and weight distribution of channel information without changing the spatial resolution of the input feature map. The ReLu and Tanh activation functions can increase the nonlinear ability of the model and generate multimodal features. The weight is multiplied by the original feature map to obtain the weighted multimodal feature .

[0095] Alternatively, if Figure 2 As shown in the figure, the text-guided visual decoding module consists of three stages, and the input of each stage is the fusion feature output of the previous stage. And the multimodal features of the corresponding stages in the visual encoding module ;

[0096] In each stage, the visual decoder first upsamples the feature map of the previous stage through bilinear interpolation, gradually expands the spatial resolution of the feature map, restores the spatial information of the original image, and lays the foundation for subsequent image segmentation;

[0097] The upsampled features and the multimodal features of the corresponding stage of the visual encoding module The addition is performed in the channel dimension to further realize the deep fusion of visual features and text features, make full use of the semantic information in the text features to guide the restoration of visual features, enhance the sensitivity of the segmentation task to details and semantic boundaries, and use the multimodal features Through layer-by-layer fusion and upsampling operations, the features output by the visual decoding module gradually approach the original resolution of the input image, and provide a more delicate and semantically clear feature representation for the subsequent angle transformation module;

[0098] During the entire decoding process, the visual decoding module uses the smallest scale enhanced visual features output by the visual encoding module. As input, after three levels of decoding, three larger-scale feature maps are gradually generated, which together with the initial input feature map constitute four-scale feature maps. Finally, the size of the feature maps of the four scales is consistent with the feature map corresponding to the visual encoding module. Figure 1 It provides structural support for feature fusion and segmentation, and finally outputs multi-scale features. .

[0099] Alternatively, if Figure 5 As shown, the angle transformation module includes three angle transformation layers, three nonlinear projection layers and one linear projection layer, receiving the multi-scale features output by the text-guided visual decoding module As input, the final mask prediction is output. The whole process is shown in the following formula:

[0100] (8)

[0101] (9)

[0102] (10)

[0103] in, Represents the smallest feature map of the input, represents the input of each angle transformation layer, Represents the concat operation, which realizes the feature fusion in the channel dimension. The scale of feature maps of different scales is unified through bilinear interpolation. The process includes 3×3 convolutional layers, batch normalization layers, and ReLU activation functions, corresponding to nonlinear projection layers. represents the angle transformation layer, represents the final segmentation mask, is the linear projection function, and the feature Mapped to the final segmentation mask.

[0104] Alternatively, if Figure 6 As shown, the angle transformation layer captures angle information from the input features for angle transformation, and dynamically re-parameterizes the convolution kernel weights, and filters out redundant features through the updated convolution kernel weights. The process is as follows:

[0105] The feature map is fed into a depthwise separable convolutional layer and an average pooling layer to further extract features, and then fed into a linear prediction layer, which includes a fully connected layer and an activation function to predict n angle parameters based on the extracted features. and the corresponding weight parameters , , according to the coordinates of the points on the feature map Perform coordinate transformation to obtain , coordinates after angle transformation The weights will fall to the new position, and the bilinear interpolation will ensure the continuity of weight changes to prevent the loss of weight information due to angle changes. Finally, the updated convolution kernel weights, weight parameters and original visual features are sent to the convolution layer to filter redundant features and generate new features. The formula is as follows:

[0106] (11)

[0107] (12)

[0108] (13)

[0109] in, is the coordinate of the point on the original feature map, is the coordinate after angle transformation, is the inverse matrix of the angular affine transformation rotation matrix, is bilinear interpolation, represents the original static convolution kernel weight, According to the transformed The original weight To update, is the original feature map, is the weighted feature map.

[0110] Optionally, the training process of the auricle representation segmentation model uses instance contrast learning to improve the model's discriminability for different instances and its robustness to different languages ​​describing the same instance;

[0111] like Figure 7 As shown, the instance contrastive learning samples b text descriptions of an image. These text descriptions may point to the same instance or different instances in the image. The model generates b mask predictions based on the image and b text descriptions, making the masks of sentences describing the same instance similar and suppressing the overlap of masks of different instances. The overlap score is obtained by calculating the overlap between each two masks. The i-th and j-th final masks are and The overlap fraction between is calculated by the following formula:

[0112] (14)

[0113] in, , n is the pixel size of feature map S, is the i-th segmentation mask The normalized value at position n is used to calculate the instance contrast loss based on the overlap score. , which is used to enhance the model's ability to distinguish different instances. The formula is as follows:

[0114] (15)

[0115] in, express and Whether they correspond to the same instance, is the i-th mask The intersection-over-union ratio between the actual annotation and the original annotation is used to prevent incorrect mask prediction;

[0116] The loss function of the ear pinna segmentation model uses a joint optimization loss function to combine the binary cross entropy loss, Dice loss and the instance contrast loss Combined with the above, the segmentation accuracy and boundary distinction ability of the model are improved, which can be expressed as:

[0117] (16)

[0118] in, represents the binary cross entropy loss, represents the Dice loss, Denotes the instance contrast loss.

[0119] The binary cross entropy loss of the embodiment of the present invention is used for pixel-level classification to determine whether each pixel belongs to the target area, and to penalize the accuracy of the model's prediction probability, which can provide fine-grained supervision signals and improve the accuracy of segmentation. Dice loss is a loss function based on regional overlap, which is used to evaluate the similarity between the predicted area and the real area. It has stronger sensitivity to small target areas, can effectively alleviate the problem of imbalance in target areas, and significantly improve the model's perception of boundary areas. Instance contrast loss is used to enhance the model's ability to distinguish different instances, prevent the phenomenon that partial mask overlap may exist in different ear areas, and improve the model's boundary distinction ability.

[0120] The training process of the auricle reference segmentation model based on text guidance and angle transformation in the embodiment of the present invention is as follows:

[0121] 1) Data collection.

[0122] A certain number of human ear images are collected by high-definition cameras, the acupuncture points of the human ears are manually marked, and they are associated with text descriptions.

[0123] 2) Dataset division.

[0124] All collected data samples are divided into training set, validation set and test set in a ratio of 8:1:1 to form a complete data set. The training set is used for algorithm training, the validation set is used to adjust parameters and verify the training effect, and the test set is used to evaluate the final performance of the model. Finally, the model with the best performance on the test set is selected as the final model.

[0125] 3) Model training.

[0126] The model is trained on the training set and the model effect is verified on the validation set. The model with the best performance on the validation set is selected as the final model.

[0127] like Figure 8 As shown, an embodiment of the present invention further provides an auricle reference segmentation system, the system comprising:

[0128] An acquisition module 810 is used to acquire a human ear image to be segmented and a corresponding text description;

[0129] A segmentation module 820 is used to input the human ear image to be segmented and the corresponding text description into an auricle reference segmentation model based on text guidance and angle transformation, wherein the auricle reference segmentation model includes a text encoding module, a text-guided visual encoding module, a text-guided visual decoding module and an angle transformation module;

[0130] The text encoding module embeds the text description into a high-dimensional word vector to obtain text features ;

[0131] The text-guided visual encoding module realizes the fusion of text features and image features by organizing into a four-stage structure. The four stages are connected in series, and the output of the previous stage is used as the input of the next stage to realize multi-stage feature fusion. Each stage includes a visual encoder, a cross-modal perception module and an attention gating module. The visual encoder of each stage generates visual features. The cross-modal perception module aligns and fuses the visual features across modalities. With text features , get multimodal features , the multimodal features Each element in is weighted by the attention gating module to obtain weighted multimodal features , by element and original visual features Added together, they produce a set of enhanced visual features that embed linguistic information , the smallest scale enhanced visual features produced by the last stage inputting the text-guided visual decoding module;

[0132] The text-guided visual decoding module gradually restores the spatial resolution of the image, and further integrates text features and visual features to provide high-quality feature representation for the final segmentation task and output multi-scale features;

[0133] The angle transformation module performs angle transformation on the multi-scale features and outputs a segmentation mask of an area related to the text description.

[0134] An auricle reference segmentation system provided in an embodiment of the present invention has a functional structure corresponding to an auricle reference segmentation method provided in an embodiment of the present invention, which will not be described in detail here.

[0135] Fig. 9It is a structural diagram of an electronic device 900 provided in an embodiment of the present invention. The electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 901 and one or more memories 902, wherein the memory 902 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 901 to implement the steps of the above-mentioned auricle reference segmentation method.

[0136] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned auricle reference segmentation method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0137] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for segmenting auricle reference, characterized in that: The method comprises: S1, obtaining the human ear image to be segmented and the corresponding text description; S2, inputting the human ear image to be segmented and the corresponding text description into an auricle reference segmentation model based on text guidance and angle transformation, wherein the auricle reference segmentation model includes a text encoding module, a text-guided visual encoding module, a text-guided visual decoding module and an angle transformation module; The text encoding module embeds the text description into a high-dimensional word vector to obtain text features ; The text-guided visual encoding module realizes the fusion of text features and image features by organizing into a four-stage structure. The four stages are connected in series, and the output of the previous stage is used as the input of the next stage to realize multi-stage feature fusion. Each stage includes a visual encoder, a cross-modal perception module and an attention gating module. The visual encoder of each stage generates visual features. The cross-modal perception module aligns and fuses the visual features across modalities. With text features , get multimodal features , the multimodal features Each element in is weighted by the attention gating module to obtain weighted multimodal features , by element and original visual features Added together, they produce a set of enhanced visual features that embed linguistic information , the smallest scale enhanced visual features produced by the last stage inputting the text-guided visual decoding module; The text-guided visual decoding module gradually restores the spatial resolution of the image, and further integrates text features and visual features to provide high-quality feature representation for the final segmentation task and output multi-scale features; The angle transformation module performs angle transformation on the multi-scale features and outputs a segmentation mask of an area related to the text description.

2. The method according to claim 1, characterized in that: The text encoding module uses BERT to perform text encoding and embeds the text description into a high-dimensional word vector to obtain the text feature: ,in and N represent the number of word embedding channels and the number of words contained in the text description, respectively.

3. The method according to claim 1, characterized in that The visual encoder adopts the Swin-Transformer encoder. The input feature map is divided into windows of fixed size. The features in each window are modeled through a multi-head self-attention mechanism to capture the relationship between local areas. After the modeling is completed at the window level, the position of the window will be periodically translated to achieve global context association across windows. In the specific encoding process, the feature map is first mapped to a high-dimensional space through a linear embedding module, and then feature extraction and nonlinear transformation are performed layer by layer through a sliding window attention module and a multi-layer perceptron to enhance the expression ability of high-level semantic features and output the visual feature. .

4. The method according to claim 1, characterized in that The cross-modal perception module uses the visual features As a query, the text feature As the key and value, we use the scaled dot product attention to calculate the process of supervising visual features with text features, and promote the fusion of multimodal information. After calculating the scaled dot product attention, we will obtain the features With input visual features Perform the projection operation again to obtain and , and then perform a dot product operation to finally obtain the multimodal feature , the process formula is as follows: （1） （2） （3） （4） （5） （6） （7） in, , , , , is the projection function, Represents the dot product operation, key function Sum value function Is has 1×1 convolution with the number of output channels, query function It is also a 1×1 convolution, and the number of output channels remains the same as the text feature. When the attention is calculated, the two-dimensional visual feature map is cut and arranged into one-dimensional sequence data to adapt to the subsequent attention calculation operation. After the attention calculation is completed, the one-dimensional sequence data is restored to the form of a two-dimensional feature map.

5. The method according to claim 1, characterized in that The attention gating module learns a set of The weight map is used to rescale Each element in , to prevent multimodal features Overwhelming visual features The visual signal in the image is used to generate the image and allow an adaptive amount of textual semantic information to flow to the next stage. The specific process is as follows: The multimodal features Perform 1×1 convolution and ReLu activation, and then perform 1×1 convolution and Tanh activation again. In this process, 1×1 convolution adjusts the number of channels to achieve the reorganization and weight distribution of channel information without changing the spatial resolution of the input feature map. The ReLu and Tanh activation functions can increase the nonlinear ability of the model and generate multimodal features. The weight is multiplied by the original feature map to obtain the weighted multimodal feature .

6. The method according to claim 1, characterized in that The text-guided visual decoding module consists of three stages, and the input of each stage is the fusion feature output of the previous stage. And the multimodal features of the corresponding stages in the visual encoding module ; In each stage, the visual decoder first upsamples the feature map of the previous stage through bilinear interpolation, gradually expands the spatial resolution of the feature map, restores the spatial information of the original image, and lays the foundation for subsequent image segmentation; The upsampled features and the multimodal features of the corresponding stage of the visual encoding module The addition is performed in the channel dimension to further realize the deep fusion of visual features and text features, make full use of the semantic information in the text features to guide the restoration of visual features, enhance the sensitivity of the segmentation task to details and semantic boundaries, and use the multimodal features Through layer-by-layer fusion and upsampling operations, the features output by the visual decoding module gradually approach the original resolution of the input image, and provide a more delicate and semantically clear feature representation for the subsequent angle transformation module; During the entire decoding process, the visual decoding module uses the smallest scale enhanced visual features output by the visual encoding module. As input, after three levels of decoding, three larger-scale feature maps are gradually generated, which together with the initial input feature map constitute four-scale feature maps. Finally, the sizes of the feature maps of the four scales are consistent with the feature maps corresponding to the visual encoding module, which provides structural support for feature fusion and segmentation, and finally outputs multi-scale features. .

7. The method according to claim 1, characterized in that The angle transformation module includes three angle transformation layers, three nonlinear projection layers and one linear projection layer, and receives the multi-scale features output by the text-guided visual decoding module. As input, the final mask prediction is output. The whole process is shown in the following formula: （8） （9） （10） in, Represents the input feature map with the smallest scale, represents the input of each angle transformation layer, Represents the concat operation, which realizes the feature fusion in the channel dimension. The scale of feature maps of different scales is unified through bilinear interpolation. The process includes 3×3 convolutional layers, batch normalization layers, and ReLU activation functions, corresponding to nonlinear projection layers. represents the angle transformation layer, represents the final segmentation mask, is the linear projection function, and the feature Mapped to the final segmentation mask.

8. The method according to claim 7, characterized in that The angle transformation layer captures angle information from the input features to transform the angle, and dynamically re-parameterizes the convolution kernel weights to filter out redundant features through the updated convolution kernel weights. The process is as follows: The feature map is fed into a depthwise separable convolutional layer and an average pooling layer to further extract features, and then fed into a linear prediction layer, which includes a fully connected layer and an activation function to predict n angle parameters based on the extracted features. and the corresponding weight parameters , , according to the coordinates of the points on the feature map Perform coordinate transformation to obtain , coordinates after angle transformation The weights will fall to the new position, and the bilinear interpolation will ensure the continuity of weight changes to prevent the loss of weight information due to angle changes. Finally, the updated convolution kernel weights, weight parameters and original visual features are sent to the convolution layer to filter redundant features and generate new features. The formula is as follows: （11） （12） （13） in, is the coordinate of the point on the original feature map, is the coordinate after angle transformation, is the inverse matrix of the angular affine transformation rotation matrix, is bilinear interpolation, represents the original static convolution kernel weight, According to the transformed The original weight To update, is the original feature map, is the weighted feature map.

9. The method according to claim 1, characterized in that: The training process of the auricle representation segmentation model uses instance contrast learning to improve the model's discriminability for different instances and its robustness to different languages ​​describing the same instance; The instance contrastive learning samples b text descriptions of an image. These text descriptions may point to the same instance or different instances in the image. The model generates b mask predictions based on the image and the b text descriptions, making the masks of sentences describing the same instance similar and suppressing the overlap of masks of different instances. The overlap score is obtained by calculating the overlap between each two masks. The i-th and j-th final masks are and The overlap fraction between them is calculated by the following formula: （14） in, , n is the pixel size of feature map S, is the i-th segmentation mask The normalized value at position n is used to calculate the instance contrast loss based on the overlap score. , which is used to enhance the model's ability to distinguish different instances. The formula is as follows: （15） in, express and Whether they correspond to the same instance, is the i-th mask The intersection-over-union ratio between the actual annotation and the original annotation is used to prevent incorrect mask prediction; The loss function of the ear pinna segmentation model uses a joint optimization loss function to combine the binary cross entropy loss, Dice loss and the instance contrast loss Combined with the above, the segmentation accuracy and boundary distinction ability of the model are improved, which can be expressed as: （16） in, represents the binary cross entropy loss, represents the Dice loss, Denotes the instance contrast loss.

10. An auricle reference segmentation system, characterized in that: The system comprises: An acquisition module, used for acquiring a human ear image to be segmented and a corresponding text description; A segmentation module, used for inputting the human ear image to be segmented and the corresponding text description into an auricle reference segmentation model based on text guidance and angle transformation, wherein the auricle reference segmentation model includes a text encoding module, a text-guided visual encoding module, a text-guided visual decoding module and an angle transformation module; The text encoding module embeds the text description into a high-dimensional word vector to obtain text features ; The text-guided visual encoding module realizes the fusion of text features and image features by organizing into a four-stage structure. The four stages are connected in series, and the output of the previous stage is used as the input of the next stage to realize multi-stage feature fusion. Each stage includes a visual encoder, a cross-modal perception module and an attention gating module. The visual encoder of each stage generates visual features. The cross-modal perception module aligns and fuses the visual features across modalities. With text features , get multimodal features , the multimodal features Each element in is weighted by the attention gating module to obtain weighted multimodal features , by element and original visual features Added together, they produce a set of enhanced visual features that embed linguistic information , the smallest scale enhanced visual features produced by the last stage inputting the text-guided visual decoding module; The text-guided visual decoding module gradually restores the spatial resolution of the image, and further integrates text features and visual features to provide high-quality feature representation for the final segmentation task and output multi-scale features; The angle transformation module performs angle transformation on the multi-scale features and outputs a segmentation mask of an area related to the text description.

Images