A text-guided three-dimensional point cloud reference segmentation method and system for human ears

Abstract

The application provides a text-guided human ear three-dimensional point cloud reference segmentation method and system, which comprises the following steps: inputting a human ear three-dimensional point cloud to be segmented and a corresponding text description into a human ear point cloud reference segmentation model, wherein the model comprises a text encoder, a point cloud clustering module, a point cloud encoder and a text query feature decoding module; the text encoder encodes the text description into a text feature vector L; the point cloud clustering module divides the point cloud into different clusters and performs dimensional transformation on the center points of the clusters to obtain point cloud clustering features F d ; the point cloud encoder uses L and F d to guide point cloud feature extraction, and obtains point cloud encoding features F p' ; the text query feature decoding module decodes a mask preliminary vector Y4 through a fusion query decoder, and screens Y4 through a text-guided point cloud mask output module to obtain a mask that is most consistent with the region related to the text description as a final output mask. The application can perform reference segmentation on a human ear three-dimensional point cloud.

Description

Technical Field

[0001] This invention relates to the field of text-guided human ear 3D point cloud reference segmentation technology, and in particular to a text-guided human ear 3D point cloud reference segmentation method and system. Background Technology

[0002] With the widespread application of ear-related fields, the demand for precise segmentation of the ear region is increasing. Wearable devices related to the ear include in-ear headphones, hearing enhancement devices, and auricular acupuncture therapy devices. The wearing comfort, stability, and functionality of these wearable devices are closely related to the geometric features of key ear regions. The auricle has a complex 3D structure, composed of various depressions, protrusions, and cartilage tissue. Obtaining the geometric feature parameters of regions such as the tragus, antitragus, concha, and helix plays a crucial role in personalized device modeling and wearing comfort, thus requiring precise segmentation of key ear regions. Similarly, in ear diagnosis, auxiliary examination is needed based on the location of ear regions exhibiting abnormal textures, depressions, bulges, etc., which also necessitates precise segmentation of each region.

[0003] When analyzing two-dimensional images of the auricle, the images are easily affected by factors such as changes in lighting and angular occlusion, leading to the loss of key texture features and the inability to acquire geometric features. In contrast, three-dimensional point cloud data can provide richer geometric and spatial information. Therefore, point cloud data has become an important foundation for achieving high-precision human ear region segmentation.

[0004] Currently, common deep learning methods for human ear 3D point cloud segmentation, such as PointNet, PointNet++, and Point Transformer, are primarily designed for indoor 3D point cloud data, which is much larger than the human ear. Therefore, these methods exhibit poor accuracy in ear region segmentation. Furthermore, these methods mainly rely on the geometric features of a single modality for semantic learning, lacking effective semantic guidance. They can only rely on fixed label categories for segmentation, and these categories are usually predefined and specific (e.g., "conch," "earlobe," etc.). The limitation of this approach is that it cannot accurately locate the region of interest based on specific task requirements or free-form language instructions. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention provides a text-guided method for segmenting three-dimensional point clouds representing the human ear, the technical solution of which is as follows:

[0006] On the one hand, a text-guided method for segmenting 3D point clouds of the human ear is provided, which includes:

[0007] S1. Obtain the 3D point cloud of the human ear to be segmented and the corresponding text description;

[0008] S2. Input the three-dimensional point cloud of the ear to be segmented and the corresponding text description into the trained end-to-end ear point cloud referencing segmentation model. The ear point cloud referencing segmentation model includes a text encoder, a point cloud clustering module, a point cloud encoder, and a text query feature decoding module.

[0009] The text encoder encodes the text description into high-dimensional word vectors to obtain the text feature vector L;

[0010] The point cloud clustering module analyzes the spatial distribution and density characteristics of each point in the point cloud, divides the point cloud into different clusters, each cluster representing a relatively independent region of the human ear, and performs a dimensional transformation on the center point of each point cloud cluster to obtain the point cloud clustering feature F. d ;

[0011] The point cloud encoder, through a multimodal feature alignment module, uses the text feature vector L and the point cloud clustering feature F. d Guided point cloud feature extraction to obtain point cloud encoded features F p' ;

[0012] The text query feature decoding module includes a fusion query decoder and a text-guided point cloud mask output module. The fusion query decoder decodes the initial mask vector Y4, and the text-guided point cloud mask output module filters the initial mask vector Y4 to obtain the mask that best matches the region related to the text description, which is then used as the final output mask.

[0013] Optionally, the point cloud encoder includes three downsampling feature extraction layers and three upsampling feature decoding layers, and includes three multimodal feature alignment modules. These modules perform preliminary text-point cloud alignment and guide the downsampling of clustering features during the point cloud downsampling feature extraction process. The processing procedure of the point cloud encoder is as follows:

[0014] Point cloud data F p The input is the downsampling feature extraction layer. In each downsampling feature extraction layer, downsampling features are first extracted through convolutional layers and max pooling layers to obtain downsampling features, making their length and width dimensions half of the input. The downsampling features are then combined with the text feature vector L and the point cloud clustering features F. d Input the multimodal feature alignment module to obtain new downsampling features and new point cloud clustering features;

[0015] The downsampled feature F'3 output from the third downsampled feature extraction layer is used as the input to the upsampled feature decoding layer. Each upsampled feature decoding layer is upsampled through a 2×2×2 deconvolution layer with a stride of 2. After three upsampled feature decoding layers, the point cloud encoded feature F is obtained. p' .

[0016] Optionally, the multimodal feature alignment module uses the text feature vector L and the point cloud clustering feature F. d Feature extraction during point cloud downsampling, the point cloud clustering feature F d After passing through a linear layer, the positional features of different subdivided regions of the human ear point cloud can be represented to a certain extent. This is then added to the downsampled features of the human ear point cloud and used in the initial alignment of text features with the downsampled features of the human ear point cloud. This allows the downsampled features of the human ear point cloud to exhibit high feature aggregation within different regions and significant feature differences between different regions, improving the model's ability to distinguish between different regions of the human ear and reducing the size of the point cloud encoded feature F. p' The focus range is adjusted to allow it to pay more attention to the subdivided regions of the human ear described in the text. The processing procedure of the multimodal feature alignment module is as follows:

[0017] First, the input downsampled features F i Clustering features of point clouds Add, F i This represents the downsampled features output by the i-th downsampled feature extraction layer. This represents the point cloud clustering features input to the i-th multimodal feature alignment module, where 1≤i≤3;

[0018] The summation result is mapped through a linear layer and used as the query vector Q. The input text feature vector L is mapped through a linear layer to a key vector K and a value vector V. Cross-attention is calculated on the key vector K, value vector V, and query vector Q. Then, the result is weighted and summed with the input downsampled features and point cloud clustering features to obtain new downsampled features and point cloud clustering features. The process is as follows:

[0019]

[0020] F′ i =F i +αA

[0021] Where Linear() represents a linear operation with a learnable linear layer, d (Linear(L)) F' represents the dimension of (Linear(L)), α and β are hyperparameters, and F' i For the updated downsampled features, This is the updated point cloud clustering feature.

[0022] Optionally, the fusion query decoder consists of four stacked mask cross-attention layers and a shared multilayer perceptron, wherein the shared multilayer perceptron is initialized only once and its parameters are shared by all mask cross-attention layers.

[0023] The mask cross-attention layer introduces a mask vector to guide the update of the mask query vector, making it more focused on the mask region. This allows the model to differentially extract features of the mask region and the background during the decoding process. Each mask cross-attention layer and the shared multilayer perceptron form a fusion query decoding layer. The fusion query decoder has a total of 4 fusion query decoding layers, where in the i-th fusion query decoding layer, 1≤i≤4, the point cloud encoded features F... p' As key vectors and value vectors This represents the mask query vector input to the i-th mask cross-attention layer, where S represents the mask query vector of the output of the fourth mask cross-attention layer. i This represents the mask vector of the input i-th mask cross-attention layer, for S i F p' Perform mask cross-attention operation to focus the generated mask on the target region during decoding, so that the i-th mask vector can guide the generation of the (i+1)-th mask query vector, and output the result as the new query vector. The process is as follows:

[0024] First, regarding S i (x,y) undergoes the following binarization transformation:

[0025]

[0026] Then, the following masked cross-attention operation is performed:

[0027]

[0028] Among them, S i (x,y) represents S i The value of the element with index (x, y) in the middle. F represents p' Dimensions The initial value of S1 is obtained through random initialization, and its initial value is the same as that of F. p' For matrices of the same size containing only 1s, during the masked cross-attention operation, when S before binarization transformation... i When (x,y)=1, it represents the point that is predicted as a mask region in the (i-1)th fusion query decoding layer. The result of the operation remains the matrix product of the corresponding positions of the query vector and the transpose of the key vector in the original attention mechanism: When S before binarization transformation i When (x,y)=0, it represents a point that is predicted as a non-masked region in the (i-1)th fusion query decoding layer. The result of the operation becomes -∞, after which... After the operation, the values ​​at these positions are 0, allowing the new query vector to focus more on S. i The region in the middle is considered the mask, while the non-masked background region is ignored, which improves the model's ability to distinguish between the target mask and its surrounding background region during the segmentation process and reduces over-segmentation or under-segmentation.

[0029] Next, the point cloud is encoded with features F p' The input is a shared multilayer perceptron for feature extraction and dimensionality transformation. The result is then mapped to a range of 0 to 1 using a sigmoid function, and finally compared with... Perform matrix multiplication to obtain the initial mask vector Y i ;

[0030] Next, for Y i The following nonlinear operation is performed to obtain the mask vector S for each point. i+1 :

[0031]

[0032] S i+1 and The vector is fed into the next fusion query decoding layer for further feature decoding. After four fusion query decoding layers, the resulting initial mask vector Y4 and the output mask query vector are then processed. The final output mask is obtained by sending the text-guided point cloud mask output module.

[0033] Optionally, the text-guided point cloud mask output module first performs semantic weighting on the text feature vector L to enable the model to better learn the feature distribution of the text during training, and then performs further text-point cloud feature alignment. The semantic weighting includes:

[0034] For each word in the text description, a learnable parameter vector conforming to a Gaussian distribution is generated. These parameter vectors represent the semantic distribution of each word. These parameter vectors are concatenated to form a semantic distribution matrix P. During model training, the semantic distribution matrix P is continuously updated so that it can learn the semantic information contained in different words in the text description and their contribution to the text description. The text feature vector L is weighted using the semantic distribution matrix P, which represents different semantic information, so that the text features are more biased towards the parts that contribute more to the text description.

[0035] Then, the semantic distribution matrix P is used as the query vector, and the text feature vector L is used as the key vector and value vector. Cross attention operation is performed on P and L, and the result is used to perform residual sum operation with the semantic distribution matrix P to obtain the text-aligned semantic feature P'. Then, P' is multiplied by L to obtain the semantically weighted text feature vector L'.

[0036] For the mask query vector A matrix multiplication operation is performed with the text feature vector L' to obtain the point cloud-text matching matrix G. A softmax function is then applied to G to ensure its value is between 0 and 1. Finally, a matrix multiplication operation is performed between G and the initial mask vector Y4 to obtain the output mask matrix Y. out For the output mask matrix Y out A nonlinear operation is performed to obtain the final output mask.

[0037] Optionally, the loss function for training the human ear point cloud-referencing segmentation model consists of three parts;

[0038] The first part is to convert the output mask matrix Y out Performing a binary cross-entropy operation with the mask truth value yields...

[0039] Part Two: Designing the Contrast Region Regularization Loss Function The quality of the mask can be controlled by the number of point clouds in the output mask, as shown in the following formula:

[0040]

[0041] Where γ is a hyperparameter, σ() represents the sigmoid function, and Ω N |Ω represents the set of all points in the point cloud of the human ear. N | represents the total number of points in the human ear point cloud, Ω M |Ω represents the set of points within the target area of ​​the mask, which is also the set of positive sample points. M | indicates the number of positive sample points. Y represents out The size of the i-th value, Represents vector Y out The size of the j-th value, the first term This represents the proportion of the predicted mask's point cloud size to the total point cloud size in the human ear. During training, it will be optimized towards a smaller and smaller value, minimizing the number of mask point clouds. This avoids excessively large segmented point cloud mask areas and reduces oversegmentation. The second term... This represents the proportion of points within the target area of ​​the mask that are predicted as mask points. During training, it will be optimized towards becoming smaller and smaller. It will be optimized in a larger and larger direction, so that the number of mask point clouds in the target area is optimized to the maximum, avoiding the segmented point cloud mask range being smaller than the target mask range, and reducing the situation of undersegmentation;

[0042] Part Three: Designing the Point-to-Point Comparison Estimation Loss Function To further enhance the model's ability to distinguish between target points and adjacent background points during the optimization process, the formula is as follows:

[0043]

[0044] Among them, Ω W |Ω represents the set of points within the non-masked target area, which is also the set of negative sample points. W | represents the number of negative sample points, cos() represents calculating the cosine similarity of vectors, P i W represents the L2-normalized eigenvector of the i-th positive sample point in Fp'; j F represents p' The L2 normalized feature vector of the j-th negative sample point, P avg The average eigenvector representing the positive sample points is calculated as follows: τ is the smoothing hyperparameter, and the optimization objective of this part of the loss function is to make τ the smoothing hyperparameter. The value should be as large as possible, while making the value of as large as possible. The value of F should be minimized during the optimization process. p' The feature values ​​corresponding to positive sample points become larger and larger, while the feature values ​​corresponding to negative sample points become smaller and smaller. This widens the feature difference between positive and negative sample points, prompting the model to distinguish the described object from the adjacent background points in a fine-grained manner. This further reduces the oversegmentation or undersegmentation of the model and improves the accuracy of edge point segmentation.

[0045] The final loss function is

[0046] On the other hand, a text-guided three-dimensional point cloud referencing and segmentation system for the human ear is provided, the system comprising:

[0047] The acquisition module is used to acquire the 3D point cloud of the human ear to be segmented and the corresponding text description;

[0048] The segmentation module is used to input the three-dimensional point cloud of the ear to be segmented and the corresponding text description into the trained end-to-end ear point cloud referencing segmentation model. The ear point cloud referencing segmentation model includes a text encoder, a point cloud clustering module, a point cloud encoder, and a text query feature decoding module.

[0049] The text encoder encodes the text description into high-dimensional word vectors to obtain the text feature vector L;

[0050] The point cloud clustering module analyzes the spatial distribution and density characteristics of each point in the point cloud, divides the point cloud into different clusters, each cluster representing a relatively independent region of the human ear, and performs a dimensional transformation on the center point of each point cloud cluster to obtain the point cloud clustering feature F. d ;

[0051] The point cloud encoder, through a multimodal feature alignment module, uses the text feature vector L and the point cloud clustering feature F. d Guided point cloud feature extraction to obtain point cloud encoded features F p' ;

[0052] The text query feature decoding module includes a fusion query decoder and a text-guided point cloud mask output module. The fusion query decoder decodes the initial mask vector Y4, and the text-guided point cloud mask output module filters the initial mask vector Y4 to obtain the mask that best matches the region related to the text description, which is then used as the final output mask.

[0053] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described text-guided human ear 3D point cloud referencing segmentation method.

[0054] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described text-guided human ear 3D point cloud referencing segmentation method.

[0055] The beneficial effects of the technical solution provided by this invention include at least the following:

[0056] 1) Strong text interaction capabilities.

[0057] This invention proposes a text-guided three-dimensional point cloud segmentation method for the human ear. By integrating natural language instructions and point cloud data, it achieves accurate localization and segmentation of specific ear regions, significantly enhancing the semantic controllability of the model and making the segmentation process more flexible and personalized.

[0058] 2) High segmentation accuracy.

[0059] This invention designs a multimodal feature alignment module in the point cloud encoding process. By guiding the point cloud encoding stage with point cloud clustering features and text encoding features, and combined with the text query feature decoding module, it effectively improves the model's ability to identify complex ear structures. Through the multimodal feature alignment mechanism, the model can more accurately understand natural language instructions and locate the corresponding ear region, significantly reducing the probability of incorrect segmentation and thus improving segmentation accuracy.

[0060] 3) High segmentation accuracy.

[0061] This invention improves the accuracy of fine-grained text-point cloud feature matching for each point by designing a fusion query decoder and a text-guided point cloud mask output module, thus enhancing the fine-grained segmentation capability during the decoding process. Simultaneously, it introduces a contrast region regularization loss function and a point-to-point contrast estimation loss function to optimize the model from two aspects: the number of points in the output mask and the ability to distinguish target points from background points. This effectively improves the segmentation accuracy. This method can handle the problems of blurred boundaries and complex structures in human ear point cloud data, achieving high-precision fine-grained segmentation and meeting the application needs of high-precision fields such as ear medical auxiliary diagnosis and ear wearable device design. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of a text-guided three-dimensional point cloud referencing segmentation method for the human ear provided in an embodiment of the present invention;

[0064] Figure 2 This is a general block diagram of a cross-modal external ear region segmentation and key point localization method provided in an embodiment of the present invention;

[0065] Figure 3 This is a block diagram of the point cloud encoder structure provided in an embodiment of the present invention;

[0066] Figure 4 This is a structural block diagram of the text query feature decoding module provided in an embodiment of the present invention;

[0067] Figure 5 This is a block diagram of a text-guided three-dimensional point cloud referential segmentation system for the human ear provided in an embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0070] This invention provides a text-guided method for segmenting three-dimensional point clouds of the human ear. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps:

[0071] S1. Obtain the 3D point cloud of the human ear to be segmented and the corresponding text description;

[0072] Textual descriptions of embodiments of the present invention, such as "segmentation of the concha cavity related to headphones".

[0073] S2. Input the three-dimensional point cloud of the ear to be segmented and the corresponding text description into the trained end-to-end ear point cloud referencing segmentation model. The ear point cloud referencing segmentation model includes a text encoder, a point cloud clustering module, a point cloud encoder, and a text query feature decoding module.

[0074] The text encoder encodes the text description into high-dimensional word vectors to obtain the text feature vector L;

[0075] This invention employs the BERT natural language processing algorithm to encode the input text. It converts words, sentences, and other information in the text into high-dimensional text feature vectors L, which capture the semantic information of the text.

[0076] The point cloud clustering module analyzes the spatial distribution and density characteristics of each point in the point cloud, dividing the point cloud into different clusters (in this embodiment, a density-based clustering method (DBSCAN) is used, where each cluster represents a relatively independent region of the human ear), and the point cloud clustering feature F is obtained by performing a dimensional transformation on the center point of each point cloud cluster. d ;

[0077] The point cloud encoder, through a multimodal feature alignment module, uses the text feature vector L and the point cloud clustering feature F. d Guided point cloud feature extraction to obtain point cloud encoded features F p' ;

[0078] The text query feature decoding module includes a fusion query decoder and a text-guided point cloud mask output module. The fusion query decoder decodes the initial mask vector Y4, and the text-guided point cloud mask output module filters the initial mask vector Y4 to obtain the mask that best matches the region related to the text description, which is then used as the final output mask.

[0079] Optionally, such as Figure 3 As shown, the point cloud encoder includes three downsampling feature extraction layers and three upsampling feature decoding layers, and also includes three multimodal feature alignment modules. During the point cloud downsampling feature extraction process, it performs preliminary text-point cloud alignment and guides the downsampling of clustering features. The processing procedure of the point cloud encoder is as follows:

[0080] Point cloud data F p The input is the downsampling feature extraction layer. In each downsampling feature extraction layer, downsampling features are first extracted through convolutional layers and max pooling layers to obtain downsampling features, making their length and width dimensions half of the input. The downsampling features are then combined with the text feature vector L and the point cloud clustering features F. d Input the multimodal feature alignment module to obtain new downsampling features and new point cloud clustering features;

[0081] The downsampled feature F'3 output from the third downsampled feature extraction layer is used as the input to the upsampled feature decoding layer. Each upsampled feature decoding layer is upsampled through a 2×2×2 deconvolution layer with a stride of 2. After three upsampled feature decoding layers, the point cloud encoded feature F is obtained. p' (Size is N×C, where N is the number of human ear point clouds and C is the number of channels).

[0082] Optionally, such as Figure 3 As shown, the multimodal feature alignment module uses the text feature vector L and the point cloud clustering feature F. d Feature extraction during point cloud downsampling, the point cloud clustering feature F d After passing through a linear layer, the positional features of different subdivided regions of the human ear point cloud can be represented to a certain extent. This is then added to the downsampled features of the human ear point cloud and used in the initial alignment of text features with the downsampled features of the human ear point cloud. This allows the downsampled features of the human ear point cloud to exhibit high feature aggregation within different regions and significant feature differences between different regions, improving the model's ability to distinguish between different regions of the human ear and reducing the size of the point cloud encoded feature F. p' The focus range is adjusted to allow it to pay more attention to the subdivided regions of the human ear described in the text. The processing procedure of the multimodal feature alignment module is as follows:

[0083] First, the input downsampled features F i Clustering features of point clouds Add, F i This represents the downsampled features output by the i-th downsampled feature extraction layer. This represents the point cloud clustering features input to the i-th multimodal feature alignment module, where 1≤i≤3;

[0084] The summation result is mapped through a linear layer and used as the query vector Q. The input text feature vector L is mapped through a linear layer to a key vector K and a value vector V. Cross-attention is calculated on the key vector K, value vector V, and query vector Q. Then, the result is weighted and summed with the input downsampled features and point cloud clustering features to obtain new downsampled features and point cloud clustering features. The process is as follows:

[0085]

[0086] F′ i =F i +αA

[0087] Where Linear() represents a linear operation with a learnable linear layer, d (Linear(L)) The dimension of (Linear(L)) is represented by α and β, which are hyperparameters used to control the guiding strength of point cloud clustering features and text features on the extraction of point cloud downsampling features, while constraining the influence of text features on point cloud clustering features. The impact of the update involves setting hyperparameters α and β (in this embodiment of the invention, α is set to 0.8 and β to 0.5), F' i For the updated downsampled features, This is the updated point cloud clustering feature.

[0088] Optionally, such as Figure 4 As shown, the fusion query decoder consists of four stacked mask cross-attention layers and a shared multilayer perceptron, wherein the shared multilayer perceptron is initialized only once and its parameters are shared by all mask cross-attention layers.

[0089] Compared to traditional cross-attention mechanisms, the mask cross-attention layer introduces a mask vector to guide the update of the mask query vector, making it more focused on the mask region. This allows the model to differentially extract features of the mask region and background during the decoding process. Each mask cross-attention layer and a shared multilayer perceptron form a fusion query decoding layer. The fusion query decoder has a total of four fusion query decoding layers, where in the i-th fusion query decoding layer, 1≤i≤4, the point cloud encoded features F... p' As key vectors and value vectors This represents the mask query vector input to the i-th mask cross-attention layer, where S represents the mask query vector of the output of the fourth mask cross-attention layer. i This represents the mask vector of the input i-th mask cross-attention layer, for S i F p' Perform mask cross-attention operation to focus the generated mask on the target region during decoding, so that the i-th mask vector can guide the generation of the (i+1)-th mask query vector, and output the result as the new query vector. The process is as follows:

[0090] First, regarding S i (x,y) undergoes the following binarization transformation:

[0091]

[0092] Then, the following masked cross-attention operation is performed:

[0093]

[0094] Among them, S i (x,y) represents S i The value of the element with index (x, y) in the middle. F represents p' Dimensions The initial value of S1 is obtained through random initialization, and its initial value is the same as that of F. p' For matrices of the same size containing only 1s, during the masked cross-attention operation, when S before binarization transformation... i When (x,y)=1, it represents the point that is predicted as a mask region in the (i-1)th fusion query decoding layer. The result of the operation remains the matrix product of the corresponding positions of the query vector and the transpose of the key vector in the original attention mechanism: When S before binarization transformation i When (x,y)=0, it represents a point that is predicted as a non-masked region in the (i-1)th fusion query decoding layer. The result of the operation becomes -∞, after which... After the operation, the values ​​at these positions are 0, allowing the new query vector to focus more on S. i The region in the middle is considered the mask, while the non-masked background region is ignored, which improves the model's ability to distinguish between the target mask and its surrounding background region during the segmentation process and reduces over-segmentation or under-segmentation.

[0095] Next, the point cloud is encoded with features F p' The input is a shared multilayer perceptron for feature extraction and dimensionality transformation. The result is then mapped to a range of 0 to 1 using a sigmoid function, and finally compared with... Perform matrix multiplication to obtain the initial mask vector Y i ;

[0096] Next, for Y i The following nonlinear operation is performed to obtain the mask vector S for each point. i+1 :

[0097]

[0098] S i+1 and The vector is fed into the next fusion query decoding layer for further feature decoding. After four fusion query decoding layers, the resulting initial mask vector Y4 and the output mask query vector are then processed. The final output mask is obtained by sending the text-guided point cloud mask output module.

[0099] Optionally, such as Figure 4 As shown, the text-guided point cloud mask output module first performs semantic weighting on the text feature vector L to enable the model to better learn the feature distribution of the text during training, and then performs further text-point cloud feature alignment. The semantic weighting includes:

[0100] For each word in the text description, a learnable parameter vector conforming to a Gaussian distribution is generated. These parameter vectors represent the semantic distribution of each word. These parameter vectors are concatenated to form a semantic distribution matrix P. During model training, the semantic distribution matrix P is continuously updated so that it can learn the semantic information contained in different words in the text description and their contribution to the text description (e.g., noun subjects such as "ear concha" and "acupoint" represent the target to be segmented and contribute the most to the text description; adjective phrases such as "related to headphones" and "related to digestive function" represent descriptions of the segmentation target and contribute significantly to the text description). The text feature vector L is weighted using the semantic distribution matrix P, which represents different semantic information, so that the text features are more biased towards the parts that contribute the most to the text description.

[0101] Then, the semantic distribution matrix P is used as the query vector, and the text feature vector L is used as the key vector and value vector. Cross attention operation is performed on P and L, and the result is used to perform residual sum operation with the semantic distribution matrix P to obtain the text-aligned semantic feature P'. Then, P' is multiplied by L to obtain the semantically weighted text feature vector L'.

[0102] For the mask query vector A matrix multiplication operation is performed with the text feature vector L' to obtain the point cloud-text matching matrix G. A softmax function is then applied to G to ensure its value is between 0 and 1. Finally, a matrix multiplication operation is performed between G and the initial mask vector Y4 to obtain the output mask matrix Y. out For the output mask matrix Y out A nonlinear operation is performed to obtain the final output mask (since the values ​​of the elements in G and Y4 are all between 0 and 1, a point will only be classified as a point in the output mask if it has a large value in both the point cloud-text matching matrix G and the initial mask vector Y4).

[0103] Optionally, the loss function for training the human ear point cloud-referencing segmentation model consists of three parts;

[0104] The first part is to convert the output mask matrix Y out Performing a binary cross-entropy operation with the mask truth value yields...

[0105] Part Two: Designing the Contrast Region Regularization Loss Function (To address the issue of blurred boundaries between the target and background in human ear point clouds), the mask quality is controlled by the number of point clouds in the output mask, as shown in the following formula:

[0106]

[0107] Where γ is a hyperparameter, σ() represents the sigmoid function, and Ω N |Ω represents the set of all points in the point cloud of the human ear. N | represents the total number of points in the human ear point cloud, Ω M |Ω represents the set of points within the target area of ​​the mask, which is also the set of positive sample points. M | indicates the number of positive sample points. Y represents out The size of the i-th value, Represents vector Y out The size of the j-th value, the first term This represents the proportion of the predicted mask's point cloud size to the total point cloud size in the human ear. During training, it will be optimized towards a smaller and smaller value, minimizing the number of mask point clouds. This avoids excessively large segmented point cloud mask areas and reduces oversegmentation. The second term... This represents the proportion of points within the target area of ​​the mask that are predicted as mask points. During training, it will be optimized towards becoming smaller and smaller. It will be optimized in a larger and larger direction, so that the number of mask point clouds in the target area is optimized to the maximum, avoiding the segmented point cloud mask range being smaller than the target mask range, and reducing the situation of undersegmentation;

[0108] Part Three: Designing the Point-to-Point Comparison Estimation Loss Function To further improve the model's ability to distinguish between target points and adjacent background points during the optimization process (since the number of targets in the ear region may reach dozens, and the point cloud counts in each region differ significantly and are closely adjacent, accurate segmentation is quite difficult), the formula is as follows:

[0109]

[0110] Among them, Ω W |Ω represents the set of points within the non-masked target area, which is also the set of negative sample points. W | represents the number of negative sample points, cos() represents calculating the cosine similarity of vectors, P i W represents the L2 normalized feature vector (of size 1×C) of the i-th positive sample point in Fp';j F represents p' The L2 normalized feature vector (of size 1×C) of the j-th negative sample point, P avg The average eigenvector representing the positive sample points is calculated as follows: τ is a smoothing hyperparameter (set to 0.07 in this embodiment), and the optimization objective of this part of the loss function is to make... The value should be as large as possible, while making the value of as large as possible. The value of F should be minimized during the optimization process. p' The feature values ​​corresponding to positive sample points become larger and larger, while the feature values ​​corresponding to negative sample points become smaller and smaller. This widens the feature difference between positive and negative sample points, prompting the model to distinguish the described object from the adjacent background points in a fine-grained manner. This further reduces the oversegmentation or undersegmentation of the model and improves the accuracy of edge point segmentation.

[0111] The final loss function is

[0112] The training process of the human ear point cloud referencing segmentation model in this embodiment of the invention is as follows:

[0113] 1) Data collection.

[0114] Human ear point cloud data was collected using a 3D scanning device. Software such as CloudCompare was used to segment and annotate the ear region, obtaining a point cloud mask set composed of ground truth masks for subdivided ear regions. A large language model (such as ChatGPT4) was used to generate text descriptions for segmenting specific regions of the ear, such as "segmenting the concha related to headphones." The text generated by the large language model was manually checked and corrected. The segmentation task text descriptions were paired with the point cloud mask set, ensuring that each text description had its corresponding point cloud mask and ear point cloud data. This resulted in a human ear point cloud referential segmentation dataset composed of several point cloud-text pairs.

[0115] 2) Data set partitioning.

[0116] All collected data samples are divided into training, validation, and test sets in an 8:1:1 ratio to form a complete dataset. The training set is used for training the algorithm, the validation set is used for parameter tuning and verifying the training effect, and the test set is used to evaluate the final performance of the model. Finally, the model that performs best on the test set is selected as the final model.

[0117] 3) Model training.

[0118] The model is trained on the training set and its performance is validated on the validation set. The model that performs best on the validation set is selected as the final model.

[0119] like Figure 5As shown, this embodiment of the invention also provides a text-guided three-dimensional point cloud representation segmentation system for the human ear, the system comprising:

[0120] The acquisition module 510 is used to acquire the three-dimensional point cloud of the human ear to be segmented and the corresponding text description;

[0121] The segmentation module 520 is used to input the three-dimensional point cloud of the ear to be segmented and the corresponding text description into the trained end-to-end ear point cloud referencing segmentation model. The ear point cloud referencing segmentation model includes a text encoder, a point cloud clustering module, a point cloud encoder and a text query feature decoding module.

[0122] The text encoder encodes the text description into high-dimensional word vectors to obtain the text feature vector L;

[0123] The point cloud clustering module analyzes the spatial distribution and density characteristics of each point in the point cloud, divides the point cloud into different clusters, each cluster representing a relatively independent region of the human ear, and performs a dimensional transformation on the center point of each point cloud cluster to obtain the point cloud clustering feature F. d ;

[0124] The point cloud encoder, through a multimodal feature alignment module, uses the text feature vector L and the point cloud clustering feature F. d Guided point cloud feature extraction to obtain point cloud encoded features F p' ;

[0125] The text query feature decoding module includes a fusion query decoder and a text-guided point cloud mask output module. The fusion query decoder decodes the initial mask vector Y4, and the text-guided point cloud mask output module filters the initial mask vector Y4 to obtain the mask that best matches the region related to the text description, which is then used as the final output mask.

[0126] The text-guided three-dimensional point cloud referencing system for the human ear provided in this embodiment of the invention has a functional structure that corresponds to the text-guided three-dimensional point cloud referencing method for the human ear provided in this embodiment of the invention, and will not be described again here.

[0127] Figure 6 This is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 601 and one or more memories 602. The memory 602 stores at least one instruction, which is loaded and executed by the processor 601 to implement the steps of the above-described text-guided three-dimensional point cloud representation segmentation method for the human ear.

[0128] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the text-guided three-dimensional point cloud representation and segmentation method for the human ear. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0129] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text-guided method for segmenting three-dimensional point clouds representing the human ear, characterized in that, The method includes: S1. Obtain the 3D point cloud of the human ear to be segmented and the corresponding text description; S2. Input the three-dimensional point cloud of the ear to be segmented and the corresponding text description into the trained end-to-end ear point cloud referencing segmentation model. The ear point cloud referencing segmentation model includes a text encoder, a point cloud clustering module, a point cloud encoder, and a text query feature decoding module. The text encoder encodes the text description into high-dimensional word vectors to obtain the text feature vector L; The point cloud clustering module analyzes the spatial distribution and density characteristics of each point in the point cloud, divides the point cloud into different clusters, each cluster representing a relatively independent region of the human ear, and performs a dimensional transformation on the center point of each point cloud cluster to obtain the point cloud clustering feature F. d ; The point cloud encoder, through a multimodal feature alignment module, uses the text feature vector L and the point cloud clustering feature F. d Guided point cloud feature extraction to obtain point cloud encoded features F p' ; The text query feature decoding module includes a fusion query decoder and a text-guided point cloud mask output module. The fusion query decoder decodes the initial mask vector Y4, and the text-guided point cloud mask output module filters the initial mask vector Y4 to obtain the mask that best matches the region related to the text description, which is then used as the final output mask.

2. The method according to claim 1, characterized in that, The point cloud encoder includes three downsampling feature extraction layers and three upsampling feature decoding layers, and also includes three multimodal feature alignment modules. During the point cloud downsampling feature extraction process, it performs preliminary text-point cloud alignment and guides the downsampling of clustering features. The processing procedure of the point cloud encoder is as follows: Point cloud data F p The input is the downsampling feature extraction layer. In each downsampling feature extraction layer, downsampling features are first extracted through convolutional layers and max pooling layers to obtain downsampling features, making their length and width dimensions half of the input. The downsampling features are then combined with the text feature vector L and the point cloud clustering features F. d Input the multimodal feature alignment module to obtain new downsampling features and new point cloud clustering features; The downsampled feature F output by the third downsampled feature extraction layer ' 3 serves as the input to the upsampled feature decoding layer. Each upsampled feature decoding layer is upsampled through a 2×2×2 deconvolution layer with a stride of 2. After three upsampled feature decoding layers, the point cloud encoded feature F is obtained. p' .

3. The method according to claim 2, characterized in that, The multimodal feature alignment module uses the text feature vector L and the point cloud clustering feature F. d Feature extraction during point cloud downsampling, the point cloud clustering feature F d After passing through a linear layer, the positional features of different subdivided regions of the human ear point cloud can be represented to a certain extent. This is then added to the downsampled features of the human ear point cloud and used in the initial alignment of text features with the downsampled features of the human ear point cloud. This allows the downsampled features of the human ear point cloud to exhibit high feature aggregation within different regions and significant feature differences between different regions, improving the model's ability to distinguish between different regions of the human ear and reducing the size of the point cloud encoded feature F. p' The focus range is adjusted to allow it to pay more attention to the subdivided regions of the human ear described in the text. The processing procedure of the multimodal feature alignment module is as follows: First, the input downsampled features F i Clustering features of point clouds Add, F i This represents the downsampled features output by the i-th downsampled feature extraction layer. This represents the point cloud clustering features input to the i-th multimodal feature alignment module, where The summation result is mapped through a linear layer and used as the query vector Q. The input text feature vector L is mapped through a linear layer to a key vector K and a value vector V. Cross-attention is calculated on the key vector K, value vector V, and query vector Q. Then, the result is weighted and summed with the input downsampled features and point cloud clustering features to obtain new downsampled features and point cloud clustering features. The process is as follows: Where Linear() represents a linear operation with a learnable linear layer, d (Linear(L)) Let F represent the dimension of (Linear(L)), α and β are hyperparameters, and F ' i For the updated downsampled features, This is the updated point cloud clustering feature.

4. The method according to claim 1, characterized in that, The fusion query decoder consists of four stacked mask cross-attention layers and a shared multilayer perceptron, wherein the shared multilayer perceptron is initialized only once and its parameters are shared by all mask cross-attention layers. The mask cross-attention layer introduces a mask vector to guide the update of the mask query vector, making it more focused on the mask region. This allows the model to differentially extract features of the mask region and the background during the decoding process. Each mask cross-attention layer and the shared multilayer perceptron form a fusion query decoding layer. The fusion query decoder has a total of 4 fusion query decoding layers, where in the i-th fusion query decoding layer, 1≤i≤4, the point cloud encoded features F... p' As key vectors and value vectors This represents the mask query vector input to the i-th mask cross-attention layer, where S represents the mask query vector of the output of the fourth mask cross-attention layer. i This represents the mask vector of the input i-th mask cross-attention layer, for S i F p' Perform mask cross-attention operation to focus the generated mask on the target region during decoding, so that the i-th mask vector can guide the generation of the (i+1)-th mask query vector, and output the result as the new query vector. The process is as follows: First, regarding S i (x,y) undergoes the following binarization transformation: Then, the following masked cross-attention operation is performed: Among them, S i (x,y) represents S i The value of the element with index (x, y) in the middle. F represents p' Dimensions The initial value of S1 is obtained through random initialization, and its initial value is the same as that of F. p' For matrices of the same size containing only 1s, during the masked cross-attention operation, when S before binarization transformation... i When (x,y)=1, it represents the point that is predicted as a mask region in the (i-1)th fusion query decoding layer. The result of the operation remains the matrix product of the corresponding positions of the query vector and the transpose of the key vector in the original attention mechanism: When S before binarization transformation i When (x,y)=0, it represents the point that is predicted as a non-masked region in the (i-1)th fusion query decoding layer, S i + The result of the operation becomes -∞, after which... After the operation, the values ​​at these positions are 0, allowing the new query vector to focus more on S. i The region in the middle is considered the mask, while the non-masked background region is ignored, which improves the model's ability to distinguish between the target mask and its surrounding background region during the segmentation process and reduces over-segmentation or under-segmentation. Next, the point cloud is encoded with features F p' The input is a shared multilayer perceptron for feature extraction and dimensionality transformation. The result is then mapped to a range of 0 to 1 using a sigmoid function, and finally compared with... Perform matrix multiplication to obtain the initial mask vector Y i ; Next, for Y i The following nonlinear operation is performed to obtain the mask vector S for each point. i+1 : S i+1 and The vector is fed into the next fusion query decoding layer for further feature decoding. After four fusion query decoding layers, the resulting initial mask vector Y4 and the output mask query vector are then processed. The final output mask is obtained by sending the text-guided point cloud mask output module.

5. The method according to claim 4, characterized in that, The text-guided point cloud mask output module first performs semantic weighting on the text feature vector L to enable the model to better learn the feature distribution of the text during training, and then performs further text-point cloud feature alignment. The semantic weighting includes: For each word in the text description, a learnable parameter vector conforming to a Gaussian distribution is generated. These parameter vectors represent the semantic distribution of each word. These parameter vectors are concatenated to form a semantic distribution matrix P. During model training, the semantic distribution matrix P is continuously updated so that it can learn the semantic information contained in different words in the text description and their contribution to the text description. The text feature vector L is weighted using the semantic distribution matrix P, which represents different semantic information, so that the text features are more biased towards the parts that contribute more to the text description. Then, the semantic distribution matrix P is used as the query vector, and the text feature vector L is used as the key vector and value vector. Cross-attention operation is performed on P and L, and the result is summed with the semantic distribution matrix P to obtain the semantic feature P of text alignment. ' Then P ' Multiplying by L yields the semantically weighted text feature vector L. ' ; For the mask query vector With text feature vector L ' Matrix multiplication is performed to obtain the point cloud-text matching matrix G. A softmax function is applied to G to ensure its value is between 0 and 1. Then, matrix multiplication is performed between G and the initial mask vector Y4 to obtain the output mask matrix Y. out For the output mask matrix Y out A nonlinear operation is performed to obtain the final output mask.

6. The method according to claim 5, characterized in that, The loss function for training the human ear point cloud-based segmentation model consists of three parts; The first part is to convert the output mask matrix Y out Performing a binary cross-entropy operation with the mask truth value yields... Part Two: Designing the Contrast Region Regularization Loss Function The quality of the mask can be controlled by the number of point clouds in the output mask, as shown in the following formula: Where γ is a hyperparameter, σ() represents the sigmoid function, and Ω N |Ω represents the set of all points in the point cloud of the human ear. N | represents the total number of points in the human ear point cloud, Ω M |Ω represents the set of points within the target area of ​​the mask, which is also the set of positive sample points. M | indicates the number of positive sample points. Y represents out The size of the i-th value, Represents vector Y out The size of the j-th value, the first term This represents the proportion of the predicted mask's point cloud size to the total point cloud size in the human ear. During training, it will be optimized towards a smaller and smaller value, minimizing the number of mask point clouds. This avoids excessively large segmented point cloud mask areas and reduces oversegmentation. The second term... This represents the proportion of points within the target area of ​​the mask that are predicted as mask points. During training, it will be optimized towards becoming smaller and smaller. It will be optimized in a larger and larger direction, so that the number of mask point clouds in the target area is optimized to the maximum, avoiding the segmented point cloud mask range being smaller than the target mask range, and reducing the situation of undersegmentation; Part Three: Designing the Point-to-Point Comparison Estimation Loss Function To further enhance the model's ability to distinguish between target points and adjacent background points during the optimization process, the formula is as follows: Among them, Ω W |Ω represents the set of points within the non-masked target area, which is also the set of negative sample points. W | represents the number of negative sample points, cos() represents calculating the cosine similarity of vectors, P i F represents p' The L2 normalized feature vector of the i-th positive sample point; W j F represents p' The L2 normalized feature vector of the j-th negative sample point, P avg The average eigenvector representing the positive sample points is calculated as follows: τ is the smoothing hyperparameter, and the optimization objective of this part of the loss function is to make τ the smoothing hyperparameter. The value should be as large as possible, while making the value of as large as possible. The value of F should be minimized during the optimization process. p' The feature values ​​corresponding to positive sample points become larger and larger, while the feature values ​​corresponding to negative sample points become smaller and smaller. This widens the feature difference between positive and negative sample points, prompting the model to distinguish the described object from the adjacent background points in a fine-grained manner. This further reduces the oversegmentation or undersegmentation of the model and improves the accuracy of edge point segmentation. The final loss function is 7. A text-guided three-dimensional point cloud representation and segmentation system for the human ear, characterized in that, The system includes: The acquisition module is used to acquire the 3D point cloud of the human ear to be segmented and the corresponding text description; The segmentation module is used to input the three-dimensional point cloud of the ear to be segmented and the corresponding text description into the trained end-to-end ear point cloud referencing segmentation model. The ear point cloud referencing segmentation model includes a text encoder, a point cloud clustering module, a point cloud encoder, and a text query feature decoding module. The text encoder encodes the text description into high-dimensional word vectors to obtain the text feature vector L; The point cloud clustering module analyzes the spatial distribution and density characteristics of each point in the point cloud, divides the point cloud into different clusters, each cluster representing a relatively independent region of the human ear, and performs a dimensional transformation on the center point of each point cloud cluster to obtain the point cloud clustering feature F. d ; The point cloud encoder, through a multimodal feature alignment module, uses the text feature vector L and the point cloud clustering feature F. d Guided point cloud feature extraction to obtain point cloud encoded features F p' ; The text query feature decoding module includes a fusion query decoder and a text-guided point cloud mask output module. The fusion query decoder decodes the initial mask vector Y4, and the text-guided point cloud mask output module filters the initial mask vector Y4 to obtain the mask that best matches the region related to the text description, which is then used as the final output mask.

8. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, characterized in that, The at least one instruction is loaded and executed by the processor to implement the text-guided human ear 3D point cloud referencing segmentation method as described in any one of claims 1-6.

9. A computer-readable storage medium storing at least one instruction, characterized in that, The at least one instruction is loaded and executed by the processor to implement the text-guided human ear 3D point cloud referencing segmentation method as described in any one of claims 1-6.

Images