Cross-modal external ear region segmentation and key point positioning method and system

Abstract

The invention provides a cross-modal external ear region segmentation and key point positioning method and system, and the method comprises the steps: obtaining multi-modal external ear data to be segmented and positioned, and inputting the data into a multi-modal external ear region segmentation and key point positioning multi-task end-to-end model, the model comprises a modal data feature extraction module, a modulo convergence module, a vision field communication module, a region segmentation head and a key point positioning head. Each modal data feature extraction module is used for extracting each modal feature; the modulo convergence module is used for enhancing voxel features and depth map features by using image features; the vision field communication module is used for performing three-mode aggregation on the three-dimensional point cloud features, the image enhanced voxel view features and the image enhanced depth image features to obtain aggregated vision field communication features; the region segmentation head is used for carrying out region segmentation on the aggregated vision field communication features; and the key point positioning head is used for carrying out key point positioning on voxel features in the aggregated vision field communication features. According to the invention, region segmentation and key point positioning can be carried out on the external ear.

Description

Technical Field

[0001] The present invention relates to the technical field of outer ear region segmentation and key point positioning, and in particular to a cross-modal outer ear region segmentation and key point positioning method and system. Background Art

[0002] In the field of wearable ear device research and development, the segmentation of the outer ear region and the positioning of key points based on 3D data are important foundations for size and appearance design. The outer ear area is relatively small, but it has multiple closely adjacent areas and scattered key points. The ear areas closely related to wearable ear devices include the helix, concha, etc., and the key points include the key points on the intersection line of the helix edge, the concha and the anti-helix, the tragus, and the anti-tragus. Since 3D point cloud data has accurate three-dimensional measurement capabilities, the accuracy of region segmentation and key point positioning based on 3D ear data plays a vital role in the wearability of the device.

[0003] Although each person's auricle has these key areas and key points, each individual has differences due to the following reasons. First, when looking down from the top of the head, the angle between the external ear and the skull, i.e., the cranial ear angle, is different for different individuals (the two extreme cases are protruding ears and close-to-the-head ears); second, when observing from the front of the external ear, the rotation angle of the external ear on the side of the skull, i.e., the "auricle inclination", varies from 0 to 70 degrees; third, the geometric structure of the ear varies from individual to individual, and the degree of surface undulation varies. These differences pose great challenges to the segmentation of each small area and the positioning of key points. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a cross-modal outer ear region segmentation and key point positioning method, and the technical solution is as follows:

[0005] On the one hand, a cross-modal outer ear region segmentation and key point localization method is provided, the method comprising:

[0006] S1. Acquire multimodal outer ear data to be segmented and located, wherein the multimodal outer ear data includes an outer ear RGB image, a depth map, and a 3D point cloud;

[0007] S2, inputting the multimodal outer ear data into a multimodal outer ear region segmentation and key point positioning multi-task end-to-end model, wherein the end-to-end model includes a feature extraction module for each modal data, a modality convergence module, a visual field connection module, a region segmentation head and a key point positioning head;

[0008] The feature extraction modules of each modality data extract features of each modality, including a depth map feature extraction module, an RGB image feature extraction module, a point cloud feature extraction module and a voxel feature extraction module, respectively extracting depth map features, RGB image features, point cloud features and voxel features;

[0009] The model convergence module uses image features to enhance voxel features and depth map features, including a voxel and RGB image feature fusion module and a depth map and RGB image feature fusion module, to obtain image enhanced voxel view features and image enhanced depth map features respectively;

[0010] The visual field connection module performs trimodal aggregation on the three-dimensional point cloud features, the image-enhanced voxel view features, and the image-enhanced depth map features to obtain aggregated visual field connection features;

[0011] The region segmentation head performs region segmentation on the aggregated visual field connectivity features to obtain an outer ear region result;

[0012] The key point positioning head performs key point positioning on the voxel features in the aggregated visual field connectivity features to obtain the 3D coordinate position of each key point.

[0013] Optionally, the depth map feature extraction module extracts the input depth map ,in , , are the height, width, and number of channels of the depth map, respectively. Convolution, normalization, and nonlinear activation are performed first, and then local feature extraction and context association are performed through a multi-layer stacked encoder layer. The multi-layer stacking is used to capture complex patterns and structures. The processing process of the encoder layer is:

[0014] The input features are first subjected to convolution, batch normalization and non-linear activation, and then input into two modules containing convolution, batch normalization and non-linear activation. The output of this module is residually concatenated with the input.

[0015] Add a dropout after each encoder layer to alleviate the gradient vanishing problem in deep networks and accelerate the convergence of the model;

[0016] Then the features are upsampled by multi-layer stacked decoders to gradually restore the feature resolution, and the output of each decoding layer is added to the dropout output of the corresponding encoding layer to effectively extract the spatial structure information in the depth map. The processing process of the decoder layer is:

[0017] The input features are first upsampled, batch normalized, and nonlinearly activated, and then input into two modules containing convolution, batch normalization, and nonlinear activation. The output of this module is residually concatenated with the input.

[0018] A dropout is added after the last decoder layer to enhance the stability of the model and prevent overfitting. After dropout, the final depth map features are obtained. ,in are the height, width and number of channels of the feature map respectively.

[0019] Optionally, the RGB image feature extraction module extracts the input RGB image ,in , , They are the height, width, and number of channels of the RGB image, respectively. They are first convolved, batch normalized, and nonlinearly activated to obtain the initial feature map, and then sequentially input into encoders 1, 2, 3, and 4 for 3, 4, 6, and 3 downsampling, respectively, to reduce the feature size to increase the receptive field. The features output by encoders 2, 3, and 4 are upsampled to keep the feature size consistent with the size of the initial feature map. Finally, the three upsampling results, the initial feature map, and the output of encoder 1 are concatenated on the feature channel, and convolution is performed to extract the RGB image features. With the depth map feature The number of channels is consistent, which is convenient for subsequent feature fusion. are the height, width and number of channels of the feature map respectively.

[0020] Optionally, the point cloud feature extraction module extracts 3D point cloud features by a downsampling method based on geometric neighborhood. , where N is the number of points, is the number of channels of point cloud features, and the extraction process is:

[0021] Define multiple local center points and their local neighborhoods;

[0022] Calculate the local geometric features of each point within the defined neighborhood, wherein the local geometric features include the position offset vector of each point in the neighborhood relative to the center point, the distance distribution of each point in the neighborhood to the center point, and the normal direction;

[0023] Then, through clustering, multiple neighborhoods are divided into multiple clusters, and then a representative point is selected for each cluster, and the local geometric features of the representative point are used as the extracted point cloud features. The dimensions of the point cloud features are compressed, but the spatial structure information is retained, so that important geometric information is not lost.

[0024] Optionally, the voxel feature extraction module maintains the general spatial structure of the point cloud while reducing the computational complexity. The extraction process includes two stages: grouping and sampling. In the grouping stage, each point is assigned to a voxel according to its spatial coordinates to generate a buffer. ,in is the maximum number of voxels, T is the maximum number of points per voxel, is the feature dimension of the point. If the number of points in a voxel exceeds T, it is downsampled to T points through the sampling stage. When the number of points and voxels does not exceed the capacity, the unused part of the buffer is filled with 0; then the non-empty voxels are averaged and pooled, and the feature mean of all points in the voxel is calculated through aggregation operation, and the features of multiple points are integrated into a single representation to obtain the voxel feature ,in is the number of channels of voxel features, is the number of non-empty voxels.

[0025] Optionally, the voxel and RGB image feature fusion module uses image features to enhance voxel features, and the process is:

[0026] The camera matrix is ​​used to establish the correspondence between three-dimensional space points and two-dimensional image pixels. The 3D point ( ) and RGB image pixels ( ) are shown in the following formula:

[0027] (1)

[0028] in, is the camera extrinsic matrix composed of the rotation matrix and the translation matrix, is the camera intrinsic parameter matrix. Through the above transformation, the center point coordinates of the three-dimensional voxel are projected to the image plane, and the corresponding pixel position is determined , the corresponding RGB image feature is expressed as ;

[0029] Since there may be errors in the camera calibration process, resulting in deviations in the projection position, the pixel offset is introduced For pixel position Dynamically adjust to more accurately align 3D voxels with 2D image features, and sample the first Image features Expressed as ,in Represents the pixel position The estimated offset, , the total number of offsets is , pixel offset The estimation is automatically learned by optimizing the loss function of region segmentation and key point positioning during network training, compensating for the pixel offset problem caused by camera calibration error. The loss function uses a multi-head cross-attention mechanism to learn the pixel offset, so that the pixel offset is automatically aligned with the optimal pixel offset. The process is as follows:

[0030] For Voxel characteristics , after the linear transformation matrix , as the query vector Query, for the Image features After the linear transformation matrix and , as the key vector Key and the value vector Value, the formula is as follows:

[0031] in, , , There are three different learnable linear transformation weight matrices responsible for converting input into Query, Key, and Value;

[0032] The correlation between the query vector Query and the key vector Key is calculated through the cross attention mechanism, which reflects the degree of match between each query voxel feature and the image pixel feature, and generates a weight distribution matrix , the formula is as follows:

[0033]

[0034] in: is the query vector of voxel features, is the key vector of image pixels, is the dimension of the key vector, used to normalize the dot product value, is the weight distribution matrix, indicating Voxel characteristics and The correlation of pixel features;

[0035] Use the above formula to generate the weight distribution matrix , and normalized by a softmax operation to ensure that the sum of all weights is 1, which allows each voxel to focus on the image pixels most relevant to it according to the weight distribution;

[0036] Using the weight distribution matrix Perform weighted summation on all values ​​to generate the final attention output representation:

[0037] The weight distribution matrix Used to guide the network to dynamically learn a more accurate pixel offset corresponding to each voxel point in the image The network continuously adjusts by minimizing the loss of region segmentation and key point localization , the query vector Query is guided to the corresponding pixel in the image The key vectors and value vectors of these regions will help the model obtain more accurate ear image feature information to achieve precise region segmentation and key point positioning;

[0038] M multi-head cross attention mechanisms are used to process each voxel to better capture multimodal information. The M multi-head cross attention mechanisms perform multiple sets of linear transformations on queries, keys, and values, respectively, and independently calculate weight distribution and enhanced features between multiple heads. The results of the M multi-head cross attention mechanisms and the first Voxel characteristics The image is spliced ​​on the feature channel and passes through a multi-layer perceptron, residual splicing and a multi-layer perceptron to obtain the first Voxel characteristics Then, the image enhancement Voxel characteristics The original voxel features after the multi-layer perceptron are concatenated and the voxel features after the nonlinear transformation of the multi-layer perceptron are obtained to obtain the final image enhanced voxel features. ,in is the number of channels of voxel features;

[0039] The depth map and RGB image feature fusion module uses image features to enhance depth map features. The process is as follows:

[0040] Automatically learn pixel offsets by optimizing the loss function for region segmentation and key point localization during network training , sampled after offset correction image features, The image features are represented as ,in Depth map The pixel point corresponds to the pixel position of the RGB image. Represents the pixel position The estimated offset, , the total number of offsets is The loss function uses a multi-head cross-attention mechanism to learn pixel offsets, allowing the pixel offsets to automatically align with the optimal pixel offsets to obtain the final image enhanced deep view features .

[0041] Optionally, the view connection module first applies a depth-to-point positive transformation and a voxel-to-point positive transformation to map the depth and voxel view features to the point view space respectively, wherein:

[0042] In the process of forward transformation from depth to point view space, for each pixel in the original depth map, its coordinates in the camera coordinate system are calculated according to its depth value and camera intrinsic parameters; then, according to the camera extrinsic parameters, the coordinates in the camera coordinate system are converted to the coordinates in the world coordinate system to obtain the point cloud coordinates corresponding to each pixel, and the coordinates in the depth feature map are obtained. The bilinear interpolation resampling is used to obtain the feature information corresponding to each point, and finally the point coordinates and feature information are transformed into the depth map feature at the point perspective. Among them is the number of points, is the final number of feature channels;

[0043] In the process of forward transformation from voxel to point view space, the voxel grid is traversed, and for each non-empty voxel, its position in three-dimensional space is calculated, and the calculated valid point position is stored in the point cloud data structure. The corresponding voxel features are reversely transformed into point features, and then bilinear interpolation is used to generate interpolated voxel features according to the number and weight of points in the 3D point cloud. Finally, the position and features are transformed through dimension and stored in the voxel features under point view. middle;

[0044] For the initial features of the point cloud , and further transformed by multi-layer perceptron to obtain ;

[0045] After these transformations, the voxel features , depth map features and point cloud features Splicing together on the feature channel to generate the visual connectivity feature ;

[0046] Then is weighted by the learnable parameter matrix Wv, where Wv consists of The third part is used to dynamically adjust the importance of each view feature and learn by optimizing the loss function;

[0047] Weighted The features are further enhanced by multi-layer perceptron and global average pooling to obtain global enhanced features. ;

[0048] Then, from the global enhancement features Regenerate adaptive features for each view and connect them through residual connection Original feature added to three different views middle, Include , , and obtain cross-view enhanced features , Include , ,This form can further promote cross-view feature interaction;

[0049] Then the cross-view enhancement feature The point cloud is transformed into the voxel space through the inverse transformation and the depth map space through the inverse transformation, and the transformation is projected back to the original voxel and depth map space, respectively, where:

[0050] In the process of inverse transformation from point cloud to voxel space, the point view features are mapped back to voxel space. The process is as follows:

[0051] First, the three-dimensional coordinates of each point are traversed to determine the voxel grid position to which it belongs. According to the characteristics of the point and its position in the voxel, the trilinear interpolation method is used to calculate the weight of the point on the voxel feature. The features of all points are weighted and accumulated to generate voxel features. This weighting is based on the interpolation weight of each point to ensure the smooth distribution of the features in the voxel. Finally, the generated voxel features are stored as ,in is the number of non-empty voxels;

[0052] In the process of inverse transformation from point cloud to depth map space, the process of projecting point view features back to depth map space is as follows:

[0053] First, according to the camera's external and internal parameters, the three-dimensional coordinates of each point are projected to the two-dimensional depth map pixel position, and the bilinear interpolation method is used to map the feature value of the point to its corresponding depth map pixel. Finally, the generated depth map feature is stored as a dimension of ,in and W are the height and width of the depth map;

[0054] The process of "transformation-cross-view enhancement-inverse transformation" in the above-mentioned visual connectivity module will go through N cycles to improve the accuracy of regional segmentation and obtain better visual connectivity features. , each cycle will get a fusion feature , N times of looping to get N They are spliced ​​together on the feature channels to obtain the aggregated visual field connectivity features.

[0055] Optionally, the region segmentation head concatenates N feature channel dimensions , and input into the fully connected layer, softmax and conditional random field modules for region segmentation tasks, and finally obtain multiple 3D mask results, each of which corresponds to an ear region.

[0056] Optionally, the key point localization head estimates the voxel likelihood value of each key point using a voxel encoder and a voxel decoder for the voxel features in the input aggregated visual connectivity features, generates a 3D heat map for each key point, and merges the 3D heat maps of all key points to obtain the 3D coordinate position of each key point, the process is as follows:

[0057] The voxel encoder uses two downsampling blocks and a 3D residual block stack to further extract the input voxel features and map the high-dimensional voxel data to a low-dimensional space;

[0058] The voxel decoder uses two upsampling blocks and a 3D residual block stack to expand the spatial size of the feature map and reduce the spacing between voxels, which helps the network locate key points more densely while reducing the number of channels to compress the extracted features;

[0059] In addition, the voxel encoder and voxel decoder implement voxel-by-voxel addition at each corresponding scale, which enhances the stability of the decoder to the feature mapping. The voxel features in the output of the voxel decoder and the aggregated visual field connectivity features of the input are spliced ​​through a 3D residual block, and then dimension reduction and probability estimation are performed through the 3D residual block and the softmax activation function. The probability estimate of each key point is described by a 3D heat map, where the larger the element value, the higher the probability of the key point at that position, thereby completing the position prediction of the key point.

[0060] On the other hand, a cross-modal outer ear region segmentation and key point localization system is provided, the system comprising:

[0061] An acquisition module is used to acquire multimodal outer ear data to be segmented and located, wherein the multimodal outer ear data includes an outer ear RGB image, a depth map and a 3D point cloud;

[0062] A segmentation and positioning module, used for inputting the multimodal outer ear data into a multimodal outer ear region segmentation and key point positioning multi-task end-to-end model, wherein the end-to-end model includes a feature extraction module for each modal data, a modality convergence module, a visual field connection module, a region segmentation head and a key point positioning head;

[0063] The feature extraction modules of each modality data extract features of each modality, including a depth map feature extraction module, an RGB image feature extraction module, a point cloud feature extraction module and a voxel feature extraction module, respectively extracting depth map features, RGB image features, point cloud features and voxel features;

[0064] The model convergence module uses image features to enhance voxel features and depth map features, including a voxel and RGB image feature fusion module and a depth map and RGB image feature fusion module, to obtain image enhanced voxel view features and image enhanced depth map features respectively;

[0065] The visual field connection module performs trimodal aggregation on the three-dimensional point cloud features, the image-enhanced voxel view features, and the image-enhanced depth map features to obtain aggregated visual field connection features;

[0066] The region segmentation head performs region segmentation on the aggregated visual field connectivity features to obtain an outer ear region result;

[0067] The key point positioning head performs key point positioning on the voxel features in the aggregated visual field connectivity features to obtain the 3D coordinate position of each key point.

[0068] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned cross-modal external ear area segmentation and key point positioning method.

[0069] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned cross-modal external ear area segmentation and key point positioning method.

[0070] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0071] 1. The present invention is based on three modal information of the outer ear: RGB image, depth image, and 3D point cloud. The voxel features are extracted according to the 3D point cloud, and then the rich semantic information of the image is fully utilized to enhance the features of the voxel view and depth map features with the image features at the pixel level, thereby obtaining a more comprehensive and accurate feature representation, and realizing the refined segmentation and positioning of details such as the outer ear contour, adjacent areas, and key points.

[0072] 2. The present invention designs a learnable modality convergence module and a visual field interconnection module, which are used to interact and learn the features of the three modalities and perform enhanced representation between different modalities, thereby improving the accuracy of segmentation and point positioning.

[0073] 3. In the analog convergence module, the present invention proposes a feature sampling method based on pixel compensation, so that each voxel feature can find the most suitable image feature, and effectively combine the multimodal information of voxels and images through a multi-head cross-attention mechanism, and use image features to enhance the expression ability of voxel features, thereby improving the fusion accuracy and matching degree of multimodal features.

[0074] 4. The present invention designs an end-to-end multi-task model of region segmentation and key point localization based on cross-modal data fusion, which improves the accuracy and robustness of region segmentation and key point localization through joint modeling and collaborative optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 It is a flow chart of a cross-modal outer ear region segmentation and key point positioning method provided by an embodiment of the present invention;

[0077] Figure 2 It is an overall block diagram of a cross-modal outer ear region segmentation and key point positioning method provided by an embodiment of the present invention;

[0078] Figure 3 is a structural block diagram of a depth map feature extraction module provided by an embodiment of the present invention;

[0079] Figure 4 is a block diagram of the encoder layer structure provided by an embodiment of the present invention;

[0080] Figure 5 is a block diagram of a decoder layer structure provided by an embodiment of the present invention;

[0081] Figure 6 It is a structural block diagram of an RGB image feature extraction module provided by an embodiment of the present invention;

[0082] Figure 7 is a structural block diagram of a point cloud feature extraction module provided by an embodiment of the present invention;

[0083] Figure 8 is a structural block diagram of a voxel feature extraction module provided by an embodiment of the present invention;

[0084] Fig. 9 is a structural block diagram of a voxel and RGB image feature fusion module provided by an embodiment of the present invention;

[0085] Fig.10 is a structural block diagram of a visual connection module provided in an embodiment of the present invention;

[0086] Fig.11 is a block diagram of a region segmentation header structure provided by an embodiment of the present invention;

[0087] Fig.12is a structural block diagram of a key point positioning head provided by an embodiment of the present invention;

[0088] Fig.13 It is a block diagram of a cross-modal outer ear region segmentation and key point positioning system provided by an embodiment of the present invention;

[0089] Fig.14 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0091] An embodiment of the present invention provides a cross-modal outer ear region segmentation and key point positioning method, which can be implemented by an electronic device, and the electronic device can be a terminal or a server. Figure 1 The flowchart of this method is shown in FIG. Figure 2 The overall block diagram of the method is shown below. The processing flow may include the following steps:

[0092] S1. Acquire multimodal outer ear data to be segmented and located, wherein the multimodal outer ear data includes an outer ear RGB image, a depth map, and a 3D point cloud;

[0093] The embodiment of the present invention uses Intel RealSense D455 to collect the outer ear RGB image to be segmented and located (including the full image of the ear) ), depth map (full image of the ear ) and 3D point cloud (683 points in the ear).

[0094] S2, inputting the multimodal outer ear data into a multimodal outer ear region segmentation and key point positioning multi-task end-to-end model, wherein the end-to-end model includes a feature extraction module for each modal data, a modality convergence module, a visual field connection module, a region segmentation head and a key point positioning head;

[0095] The feature extraction modules of each modality data extract features of each modality, including a depth map feature extraction module, an RGB image feature extraction module, a point cloud feature extraction module and a voxel feature extraction module, respectively extracting depth map features, RGB image features, point cloud features and voxel features;

[0096] The model convergence module uses image features to enhance voxel features and depth map features, including a voxel and RGB image feature fusion module and a depth map and RGB image feature fusion module, to obtain image enhanced voxel view features and image enhanced depth map features respectively;

[0097] The visual field connection module performs trimodal aggregation on the three-dimensional point cloud features, the image-enhanced voxel view features, and the image-enhanced depth map features to obtain aggregated visual field connection features;

[0098] The region segmentation head performs region segmentation on the aggregated visual field connectivity features to obtain an outer ear region result;

[0099] The key point positioning head performs key point positioning on the voxel features in the aggregated visual field connectivity features to obtain the 3D coordinate position of each key point.

[0100] Alternatively, if Figure 3 As shown, the depth map feature extraction module extracts the input depth map ,in , , are the height, width, and number of channels of the depth map, respectively. Convolution, normalization, and nonlinear activation are performed first, and then local feature extraction and context association are performed through a multi-layer stacked encoder layer. The multi-layer stacking is used to capture complex patterns and structures, such as Figure 4 As shown, the processing process of the encoder layer is:

[0101] The input features are first subjected to convolution, batch normalization and non-linear activation, and then input into two modules containing convolution, batch normalization and non-linear activation. The output of this module is residually concatenated with the input.

[0102] Add a dropout after each encoder layer to alleviate the gradient vanishing problem in deep networks and accelerate the convergence of the model;

[0103] The features are then upsampled by multi-layer decoders to gradually restore the feature resolution, and the output of each decoding layer is added to the dropout output of the corresponding encoding layer to effectively extract the spatial structure information in the depth map, such as Figure 5 As shown, the processing process of the decoder layer is:

[0104] The input features are first upsampled, batch normalized, and nonlinearly activated, and then input into two modules containing convolution, batch normalization, and nonlinear activation. The output of this module is residually concatenated with the input.

[0105] A dropout is added after the last decoder layer to enhance the stability of the model and prevent overfitting. After dropout, the final depth map features are obtained. ,in are the height, width and number of channels of the feature map respectively.

[0106] Alternatively, if Figure 6As shown, the RGB image feature extraction module is used to extract the input RGB image ,in , , They are the height, width, and number of channels of the RGB image, respectively. They are first convolved, batch normalized, and nonlinearly activated to obtain the initial feature map, and then input into encoders 1, 2, 3, and 4 for 3, 4, 6, and 3 downsampling respectively, reducing the feature size to increase the receptive field. The features output by encoders 2, 3, and 4 are upsampled to keep the feature size consistent with the size of the initial feature map. Finally, the three upsampling results, the initial feature map, and the output of encoder 1 are spliced ​​on the feature channel (splicing on the feature channel is to merge the feature channels, increase the number of channels, improve the feature expression ability of the network, and the model can express information more richly), and convolution is performed to extract the RGB image features. With the depth map feature The number of channels is consistent, which is convenient for subsequent feature fusion. are the height, width and number of channels of the feature map respectively.

[0107] The RGB image feature extraction module allows direct addition of input and output by extracting image features layer by layer and performing upsampling, thereby enhancing the learning ability of the network and reducing the gradient vanishing problem in deep network training. The model can combine multi-level features to enhance its understanding and processing capabilities of images.

[0108] Alternatively, if Figure 7 As shown, the point cloud feature extraction module extracts 3D point cloud features through a downsampling method based on geometric neighborhood , where N is the number of points, is the number of channels of point cloud features, and the extraction process is:

[0109] Define multiple local center points and their local neighborhoods;

[0110] Calculate the local geometric features of each point within the defined neighborhood, wherein the local geometric features include the position offset vector of each point in the neighborhood relative to the center point, the distance distribution of each point in the neighborhood to the center point, and the normal direction;

[0111] Then, through clustering, multiple neighborhoods are divided into multiple clusters, and then a representative point is selected for each cluster, and the local geometric features of the representative point are used as the extracted point cloud features. The dimensions of the point cloud features are compressed, but the spatial structure information is retained, so that important geometric information is not lost.

[0112] The key advantage of the point cloud feature extraction module is that it can reduce the redundancy of point cloud data while maintaining key spatial position information in the point cloud, laying the foundation for subsequent multimodal data fusion.

[0113] Alternatively, if Figure 8 As shown, the voxel feature extraction module maintains the general spatial structure of the point cloud while reducing the computational complexity (the downsampling method based on geometric neighborhood is more suitable for applications that require fine preservation of local geometric information, but is more computationally complex and suitable for fine feature extraction and modeling tasks. The two methods have their own advantages. The embodiment of the present invention combines the two methods). The extraction process includes two stages: grouping and sampling. In the grouping stage, each point is assigned to a voxel according to its spatial coordinates to generate a buffer. ,in is the maximum number of voxels, T is the maximum number of points per voxel, is the feature dimension of the point. If the number of points in a voxel exceeds T, it is downsampled to T points through the sampling stage. When the number of points and voxels does not exceed the capacity, the unused part of the buffer is filled with 0; then the non-empty voxels are averaged and pooled, and the feature mean of all points in the voxel is calculated through aggregation operation, and the features of multiple points are integrated into a single representation to obtain the voxel feature ,in is the number of channels of voxel features, is the number of non-empty voxels.

[0114] Alternatively, if Fig. 9 As shown, the voxel and RGB image feature fusion module uses image features to enhance voxel features. The process is:

[0115] The camera matrix is ​​used to establish the correspondence between three-dimensional space points and two-dimensional image pixels. The 3D point ( ) and RGB image pixels ( ) are shown in the following formula:

[0116] (1)

[0117] in, is the camera extrinsic matrix composed of the rotation matrix and the translation matrix, is the camera intrinsic parameter matrix. Through the above transformation, the center point coordinates of the three-dimensional voxel are projected to the image plane, and the corresponding pixel position is determined , the corresponding RGB image feature is expressed as ;

[0118] Since there may be errors in the camera calibration process, resulting in deviations in the projection position, the pixel offset is introduced For pixel position Dynamically adjust to more accurately align 3D voxels with 2D image features, and sample the first Image features Expressed as ,in Represents the pixel position The estimated offset, , the total number of offsets is , pixel offset The estimation is automatically learned by optimizing the loss function of region segmentation and key point positioning during network training, compensating for the pixel offset problem caused by camera calibration error. The loss function uses a multi-head cross-attention mechanism to learn the pixel offset, so that the pixel offset is automatically aligned with the optimal pixel offset. The process is as follows:

[0119] For Voxel characteristics , after the linear transformation matrix , as the query vector Query, for the Image features After the linear transformation matrix and , as the key vector Key and the value vector Value, the formula is as follows:

[0120] in, , , There are three different learnable linear transformation weight matrices responsible for converting input into Query, Key, and Value;

[0121] The correlation between the query vector Query and the key vector Key is calculated through the cross attention mechanism, which reflects the degree of match between each query voxel feature and the image pixel feature, and generates a weight distribution matrix , the formula is as follows:

[0122]

[0123] in: is the query vector of voxel features, is the key vector of the image pixels, is the dimension of the key vector, used to normalize the dot product value, is the weight distribution matrix, indicating Voxel characteristics and The correlation of pixel features;

[0124] Use the above formula to generate the weight distribution matrix , and normalized by a softmax operation to ensure that the sum of all weights is 1, which allows each voxel to focus on the image pixels most relevant to it according to the weight distribution;

[0125] Using the weight distribution matrix Perform weighted summation on all values ​​to generate the final attention output representation:

[0126] The weight distribution matrix Used to guide the network to dynamically learn a more accurate pixel offset corresponding to each voxel point in the image The network continuously adjusts by minimizing the loss of region segmentation and key point localization , the query vector Query is guided to the corresponding pixel in the image The key vectors and value vectors of these regions will help the model obtain more accurate ear image feature information to achieve precise region segmentation and key point positioning;

[0127] M multi-head cross attention mechanisms are used to process each voxel to better capture multimodal information. The M multi-head cross attention mechanisms perform multiple sets of linear transformations on queries, keys, and values, respectively, and independently calculate weight distribution and enhanced features between multiple heads. The results of the M multi-head cross attention mechanisms and the first Voxel characteristics The image is spliced ​​on the feature channel and passes through a multi-layer perceptron, residual splicing and a multi-layer perceptron to obtain the first Voxel characteristics Then, the image enhancement Voxel characteristics The original voxel features after the multi-layer perceptron are concatenated and the voxel features after the nonlinear transformation of the multi-layer perceptron are obtained to obtain the final image enhanced voxel features. (If some voxels have no corresponding image features, a zero vector is added to keep the feature dimension consistent), where is the number of channels of voxel features;

[0128] The depth map and RGB image feature fusion module uses image features to enhance depth map features. The process is as follows:

[0129] Automatically learn pixel offsets by optimizing the loss function for region segmentation and key point localization during network training , sampled after offset correction image features, Image features Expressed as ,in Depth map The pixel point corresponds to the pixel position of the RGB image. Represents the pixel position The estimated offset, , the total number of offsets is The loss function uses a multi-head cross-attention mechanism to learn pixel offsets, allowing the pixel offsets to automatically align with the optimal pixel offsets to obtain the final image enhanced deep view features .

[0130] The specific process of the fusion of the depth map and the RGB image feature fusion module is similar to the specific process of the fusion of the voxel and the RGB image feature fusion module, which will not be repeated here.

[0131] Alternatively, if Fig.10 As shown, the visual field connection module first applies the depth-to-point positive transformation and the voxel-to-point positive transformation to map the depth and voxel view features to the point view space respectively, wherein:

[0132] In the process of forward transformation from depth to point view space, for each pixel in the original depth map, its coordinates in the camera coordinate system are calculated according to its depth value and camera intrinsic parameters; then, according to the camera extrinsic parameters, the coordinates in the camera coordinate system are converted to the coordinates in the world coordinate system to obtain the point cloud coordinates corresponding to each pixel, and the coordinates in the depth feature map are obtained. The bilinear interpolation resampling is used to obtain the feature information corresponding to each point, and finally the point coordinates and feature information are transformed into the depth map feature at the point perspective. Among them is the number of points, is the final number of feature channels (the main purpose of bilinear interpolation is to obtain the eigenvalues ​​corresponding to each point cloud coordinate from the depth feature map and generate the corresponding insertion points, so that the mapping features have higher accuracy and better continuity. Since the pixels in the depth map are discretely distributed, and the coordinates of the point cloud are usually continuous, such coordinates may not necessarily correspond directly to the pixel positions on the depth map. Therefore, bilinear interpolation is an effective means to obtain the corresponding feature information based on the continuous coordinates of the point cloud on the discrete depth feature map);

[0133] In the process of forward transformation from voxel to point view space, the voxel grid is traversed, and for each non-empty voxel, its position in three-dimensional space is calculated, and the calculated valid point position is stored in the point cloud data structure. The corresponding voxel features are reversely transformed into point features, and then bilinear interpolation is used to generate interpolated voxel features according to the number and weight of points in the 3D point cloud. Finally, the position and features are transformed through dimension and stored in the voxel features under point view. middle;

[0134] For the initial features of the point cloud , and further transformed by multi-layer perceptron to obtain ;

[0135] After these transformations, the voxel features , depth map features

[0136] and point cloud features Splicing together on the feature channel produces the visual field connection feature ;

[0137] Then is weighted by the learnable parameter matrix Wv, where Wv consists of The third part is used to dynamically adjust the importance of each view feature and learn by optimizing the loss function;

[0138] Weighted The features are further enhanced by multi-layer perceptron and global average pooling to obtain global enhanced features. ;

[0139] Then, from the global enhancement features Regenerate adaptive features for each view and connect them through residual connection Original feature added to three different views middle, Include , , and obtain cross-view enhanced features , Include , ,This form can further promote cross-view feature interaction;

[0140] Then the cross-view enhancement feature The point cloud is transformed into the voxel space through the inverse transformation and the depth map space through the inverse transformation, and the transformation is projected back to the original voxel and depth map space, respectively, where:

[0141] In the process of inverse transformation from point cloud to voxel space, the point view features are mapped back to voxel space. The process is as follows:

[0142] First, the three-dimensional coordinates of each point are traversed to determine the voxel grid position to which it belongs. According to the characteristics of the point and its position in the voxel, the trilinear interpolation method is used to calculate the weight of the point on the voxel feature. The features of all points are weighted and accumulated to generate voxel features. This weighting is based on the interpolation weight of each point to ensure the smooth distribution of the features in the voxel. Finally, the generated voxel features are stored as ,in is the number of non-empty voxels;

[0143] In the process of inverse transformation from point cloud to depth map space, the process of projecting point view features back to depth map space is as follows:

[0144] First, according to the camera's external and internal parameters, the three-dimensional coordinates of each point are projected to the two-dimensional depth map pixel position, and the bilinear interpolation method is used to map the feature value of the point to its corresponding depth map pixel. Finally, the generated depth map feature is stored as a dimension of ,in and W are the height and width of the depth map;

[0145] The process of "transformation-cross-view enhancement-inverse transformation" in the above-mentioned visual connectivity module will go through N cycles to improve the accuracy of regional segmentation and obtain better visual connectivity features. , each cycle will get a fusion feature , N times of looping to get N ( , ) are spliced ​​together on the feature channel to obtain the aggregated view connectivity feature.

[0146] Alternatively, if Fig.11 As shown, the region segmentation head concatenates N feature channel dimensions , and input into the fully connected layer, softmax and conditional random field module (the conditional random field module can reduce errors caused by noise or misclassification, enhance the coherence of the point cloud area, and effectively suppress isolated points or unreasonable segmentation, thereby improving the final segmentation accuracy) to perform the regional segmentation task, and finally obtain multiple 3D mask results, each of which corresponds to an ear area.

[0147] Alternatively, if Fig.12 As shown, the key point localization head estimates the voxel likelihood value of each key point using the voxel encoder and the voxel decoder for the voxel features in the input aggregated visual field connectivity features, generates a 3D heat map for each key point, and merges the 3D heat maps of all key points to obtain the 3D coordinate position of each key point. The process is as follows:

[0148] The voxel encoder uses two downsampling blocks and a 3D residual block stack to further extract the input voxel features and map the high-dimensional voxel data to a low-dimensional space;

[0149] The voxel decoder uses two upsampling blocks and a 3D residual block stack to expand the spatial size of the feature map and reduce the spacing between voxels, which helps the network locate key points more densely while reducing the number of channels to compress the extracted features;

[0150] In addition, the voxel encoder and voxel decoder implement voxel-by-voxel addition at each corresponding scale, which enhances the stability of the decoder to the feature mapping. The voxel features in the output of the voxel decoder and the aggregated visual field connectivity features of the input are spliced ​​through a 3D residual block, and then dimension reduction and probability estimation are performed through the 3D residual block and the softmax activation function. The probability estimate of each key point is described by a 3D heat map, where the larger the element value, the higher the probability of the key point at that position, thereby completing the position prediction of the key point.

[0151] The training process of the multi-modal outer ear region segmentation and key point localization multi-task end-to-end model is as follows:

[0152] 1. Multimodal outer ear data collection and annotation.

[0153] The embodiment of the present invention collects RGB images of the outer ears of multiple people (including the full image of the ear) through Intel RealSense D455 ), depth map (full image of the ear ) and 3D point cloud (683 points in the ear). Cloudcompare and Labelimg software were used to annotate multiple areas and key points of the outer ear.

[0154] 2. Data set division.

[0155] The outer ear depth map, RGB image and point cloud of the same person obtained in step 1 are regarded as one sample. All collected and constructed sample sets are divided into training set, validation set and test set in a ratio of 8:1:1.

[0156] 2. Model training.

[0157] The model is trained on the training set and the model effect is verified on the validation set. The model with the best performance on the validation set is selected as the final model.

[0158] like Fig.13 As shown, an embodiment of the present invention further provides a cross-modal outer ear region segmentation and key point positioning system, the system comprising:

[0159] An acquisition module 1310 is used to acquire multimodal outer ear data to be segmented and located, wherein the multimodal outer ear data includes an outer ear RGB image, a depth map, and a 3D point cloud;

[0160] The segmentation and positioning module 1320 is used to input the multimodal outer ear data into a multimodal outer ear region segmentation and key point positioning multi-task end-to-end model, wherein the end-to-end model includes a feature extraction module for each modal data, a modality convergence module, a visual field connection module, a region segmentation head, and a key point positioning head;

[0161] The feature extraction modules of each modality data extract features of each modality, including a depth map feature extraction module, an RGB image feature extraction module, a point cloud feature extraction module and a voxel feature extraction module, respectively extracting depth map features, RGB image features, point cloud features and voxel features;

[0162] The model convergence module uses image features to enhance voxel features and depth map features, including a voxel and RGB image feature fusion module and a depth map and RGB image feature fusion module, to obtain image enhanced voxel view features and image enhanced depth map features respectively;

[0163] The visual field connection module performs trimodal aggregation on the three-dimensional point cloud features, the image-enhanced voxel view features, and the image-enhanced depth map features to obtain aggregated visual field connection features;

[0164] The region segmentation head performs region segmentation on the aggregated visual field connectivity features to obtain an outer ear region result;

[0165] The key point positioning head performs key point positioning on the voxel features in the aggregated visual field connectivity features to obtain the 3D coordinate position of each key point.

[0166] The cross-modal external ear region segmentation and key point positioning system provided in an embodiment of the present invention has a functional structure corresponding to the cross-modal external ear region segmentation and key point positioning method provided in an embodiment of the present invention, which will not be described in detail here.

[0167] Fig.14 It is a structural diagram of an electronic device 1400 provided in an embodiment of the present invention. The electronic device 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1401 and one or more memories 1402, wherein at least one instruction is stored in the memory 1402, and the at least one instruction is loaded and executed by the processor 1401 to implement the steps of the above-mentioned cross-modal external ear area segmentation and key point positioning method.

[0168] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned cross-modal outer ear region segmentation and key point positioning method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-modal outer ear region segmentation and key point location method, characterized in that: The method comprises: S1. Acquire multimodal outer ear data to be segmented and located, wherein the multimodal outer ear data includes an outer ear RGB image, a depth map, and a 3D point cloud; S2, inputting the multimodal outer ear data into a multimodal outer ear region segmentation and key point positioning multi-task end-to-end model, wherein the end-to-end model includes a feature extraction module for each modal data, a modality convergence module, a visual field connection module, a region segmentation head and a key point positioning head; The feature extraction modules of each modality data extract features of each modality, including a depth map feature extraction module, an RGB image feature extraction module, a point cloud feature extraction module and a voxel feature extraction module, respectively extracting depth map features, RGB image features, point cloud features and voxel features; The model convergence module uses image features to enhance voxel features and depth map features, including a voxel and RGB image feature fusion module and a depth map and RGB image feature fusion module, to obtain image enhanced voxel view features and image enhanced depth map features respectively; The visual field connection module performs trimodal aggregation on the three-dimensional point cloud features, the image-enhanced voxel view features, and the image-enhanced depth map features to obtain aggregated visual field connection features; The region segmentation head performs region segmentation on the aggregated visual field connectivity features to obtain an outer ear region result; The key point positioning head performs key point positioning on the voxel features in the aggregated visual field connectivity features to obtain the 3D coordinate position of each key point.

2. The method according to claim 1, characterized in that: The depth map feature extraction module extracts the input depth map ,in , , are the height, width, and number of channels of the depth map, respectively. Convolution, normalization, and nonlinear activation are performed first, and then local feature extraction and context association are performed through a multi-layer stacked encoder layer. The multi-layer stacking is used to capture complex patterns and structures. The processing process of the encoder layer is: The input features are first subjected to convolution, batch normalization and non-linear activation, and then input into two modules containing convolution, batch normalization and non-linear activation. The output of this module is residually concatenated with the input. Add a dropout after each encoder layer to alleviate the gradient vanishing problem in deep networks and accelerate the convergence of the model; Then the features are upsampled by multi-layer stacked decoders to gradually restore the feature resolution, and the output of each decoding layer is added to the dropout output of the corresponding encoding layer to effectively extract the spatial structure information in the depth map. The processing process of the decoder layer is: The input features are first upsampled, batch normalized, and nonlinearly activated, and then input into two modules containing convolution, batch normalization, and nonlinear activation. The output of this module is residually concatenated with the input. A dropout is added after the last decoder layer to enhance the stability of the model and prevent overfitting. After dropout, the final depth map features are obtained. ,in are the height, width and number of channels of the feature map respectively.

3. The method according to claim 2, characterized in that The RGB image feature extraction module is used to extract the input RGB image ,in , , They are the height, width, and number of channels of the RGB image, respectively. They are first convolved, batch normalized, and nonlinearly activated to obtain the initial feature map, and then sequentially input into encoders 1, 2, 3, and 4 for 3, 4, 6, and 3 downsampling, respectively, to reduce the feature size to increase the receptive field. The features output by encoders 2, 3, and 4 are upsampled to keep the feature size consistent with the size of the initial feature map. Finally, the three upsampling results, the initial feature map, and the output of encoder 1 are concatenated on the feature channel, and convolution is performed to extract the RGB image features. With the depth map feature The number of channels is consistent, which is convenient for subsequent feature fusion. are the height, width and number of channels of the feature map respectively.

4. The method according to claim 1, characterized in that: The point cloud feature extraction module extracts 3D point cloud features through a downsampling method based on geometric neighborhood , where N is the number of points, is the number of channels of point cloud features, and the extraction process is: Define multiple local center points and their local neighborhoods; Calculate the local geometric features of each point within the defined neighborhood, wherein the local geometric features include the position offset vector of each point in the neighborhood relative to the center point, the distance distribution of each point in the neighborhood to the center point, and the normal direction; Then, through clustering, multiple neighborhoods are divided into multiple clusters, and then a representative point is selected for each cluster, and the local geometric features of the representative point are used as the extracted point cloud features. The dimensions of the point cloud features are compressed, but the spatial structure information is retained, so that important geometric information is not lost.

5. The method according to claim 1, characterized in that The voxel feature extraction module maintains the general spatial structure of the point cloud while reducing the computational complexity. The extraction process includes two stages: grouping and sampling. In the grouping stage, each point is assigned to a voxel according to its spatial coordinates to generate a buffer. ,in is the maximum number of voxels, T is the maximum number of points per voxel, is the feature dimension of the point. If the number of points in a voxel exceeds T, it is downsampled to T points through the sampling stage. When the number of points and voxels does not exceed the capacity, the unused part of the buffer is filled with 0; then the non-empty voxels are averaged and pooled, and the feature mean of all points in the voxel is calculated through aggregation operation, and the features of multiple points are integrated into a single representation to obtain the voxel feature ,in is the number of channels of voxel features, is the number of non-empty voxels.

6. The method according to claim 1, characterized in that The voxel and RGB image feature fusion module uses image features to enhance voxel features. The process is as follows: The camera matrix is ​​used to establish the correspondence between three-dimensional space points and two-dimensional image pixels. The 3D point ( ) and RGB image pixels ( ) are shown in the following formula: （1） in, is the camera extrinsic matrix composed of the rotation matrix and the translation matrix, is the camera intrinsic parameter matrix. Through the above transformation, the center point coordinates of the three-dimensional voxel are projected to the image plane, and the corresponding pixel position is determined , the corresponding RGB image feature is expressed as ; Since there may be errors in the camera calibration process, resulting in deviations in the projection position, the pixel offset is introduced For pixel position Dynamically adjust to more accurately align 3D voxels with 2D image features, and sample the first Image features Expressed as ,in Represents the pixel position The estimated offset, , the total number of offsets is , pixel offset The estimation is automatically learned by optimizing the loss function of region segmentation and key point positioning during network training, compensating for the pixel offset problem caused by camera calibration error. The loss function uses a multi-head cross-attention mechanism to learn the pixel offset, so that the pixel offset is automatically aligned with the optimal pixel offset. The process is as follows: For Voxel characteristics , after the linear transformation matrix , as the query vector Query, for the Image features After the linear transformation matrix and , as the key vector Key and the value vector Value, the formula is as follows: ; in, , , There are three different learnable linear transformation weight matrices responsible for converting input into Query, Key, and Value; The correlation between the query vector Query and the key vector Key is calculated through the cross attention mechanism, which reflects the degree of match between each query voxel feature and the image pixel feature, and generates a weight distribution matrix , the formula is as follows: ； in: is the query vector of voxel features, is the key vector of image pixels, is the dimension of the key vector, used to normalize the dot product value, is the weight distribution matrix, indicating Voxel characteristics and The correlation of pixel features; Use the above formula to generate the weight distribution matrix , and normalized by a softmax operation to ensure that the sum of all weights is 1, which allows each voxel to focus on the image pixels most relevant to it according to the weight distribution; Using the weight distribution matrix Perform weighted summation on all values ​​to generate the final attention output representation: ; The weight distribution matrix Used to guide the network to dynamically learn a more accurate pixel offset corresponding to each voxel point in the image The network continuously adjusts by minimizing the loss of region segmentation and key point localization , the query vector Query is guided to the corresponding pixel in the image The key vectors and value vectors of these regions will help the model obtain more accurate ear image feature information to achieve precise region segmentation and key point positioning; M multi-head cross attention mechanisms are used to process each voxel to better capture multimodal information. The M multi-head cross attention mechanisms perform multiple sets of linear transformations on queries, keys, and values, respectively, and independently calculate weight distribution and enhanced features between multiple heads. The results of the M multi-head cross attention mechanisms and the first Voxel characteristics The image is spliced ​​on the feature channel and passes through a multi-layer perceptron, residual splicing and a multi-layer perceptron to obtain the first Voxel characteristics Then, the image enhancement Voxel characteristics The original voxel features after the multi-layer perceptron are concatenated and the voxel features after the nonlinear transformation of the multi-layer perceptron are obtained to obtain the final image enhanced voxel features. ,in is the number of channels of voxel features; The depth map and RGB image feature fusion module uses image features to enhance depth map features. The process is as follows: Automatically learn pixel offsets by optimizing the loss function for region segmentation and key point localization during network training , sampled after offset correction image features, Image features Expressed as ,in Depth map The pixel point corresponds to the pixel position of the RGB image. Represents the pixel position The estimated offset, , the total number of offsets is The loss function uses a multi-head cross-attention mechanism to learn pixel offsets, allowing the pixel offsets to automatically align with the optimal pixel offsets to obtain the final image enhanced deep view features .

7. The method according to claim 1, characterized in that The view connection module first applies a depth-to-point positive transformation and a voxel-to-point positive transformation to map the depth and voxel view features to the point view space respectively, wherein: In the process of forward transformation from depth to point view space, for each pixel in the original depth map, its coordinates in the camera coordinate system are calculated according to its depth value and camera intrinsic parameters; then, according to the camera extrinsic parameters, the coordinates in the camera coordinate system are converted to the coordinates in the world coordinate system to obtain the point cloud coordinates corresponding to each pixel, and the coordinates in the depth feature map are obtained. The bilinear interpolation resampling is used to obtain the feature information corresponding to each point, and finally the point coordinates and feature information are transformed into the depth map feature at the point perspective. Among them is the number of points, is the final number of feature channels; In the process of forward transformation from voxel to point view space, the voxel grid is traversed, and for each non-empty voxel, its position in three-dimensional space is calculated, and the calculated valid point position is stored in the point cloud data structure. The corresponding voxel features are reversely transformed into point features, and then bilinear interpolation is used to generate interpolated voxel features according to the number and weight of points in the 3D point cloud. Finally, the position and features are transformed through dimension and stored in the voxel features under point view. middle; For the initial features of the point cloud , and further transformed by multi-layer perceptron to obtain ; After these transformations, the voxel features , depth map features and point cloud features Splicing together on the feature channel to generate the visual connectivity feature ; Then is weighted by the learnable parameter matrix Wv, where Wv consists of The third part is used to dynamically adjust the importance of each view feature and learn by optimizing the loss function; Weighted The features are further enhanced by multi-layer perceptron and global average pooling to obtain global enhanced features. ; Then, from the global enhancement features Regenerate adaptive features for each view and connect them through residual connection Original feature added to three different views middle, Include , , and obtain cross-view enhanced features , Include , ,This form can further promote cross-view feature interaction; Then the cross-view enhancement feature The point cloud is transformed into the voxel space through the inverse transformation and the depth map space through the inverse transformation, and the transformation is projected back to the original voxel and depth map space, respectively, where: In the process of inverse transformation from point cloud to voxel space, the point view features are mapped back to voxel space. The process is as follows: First, the three-dimensional coordinates of each point are traversed to determine the voxel grid position to which it belongs. According to the characteristics of the point and its position in the voxel, the trilinear interpolation method is used to calculate the weight of the point on the voxel feature. The features of all points are weighted and accumulated to generate voxel features. This weighting is based on the interpolation weight of each point to ensure the smooth distribution of the features in the voxel. Finally, the generated voxel features are stored as ,in is the number of non-empty voxels; In the process of inverse transformation from point cloud to depth map space, the process of projecting point view features back to depth map space is as follows: First, according to the camera's external and internal parameters, the three-dimensional coordinates of each point are projected to the two-dimensional depth map pixel position, and the bilinear interpolation method is used to map the feature value of the point to its corresponding depth map pixel. Finally, the generated depth map feature is stored as a dimension of ,in and W are the height and width of the depth map; The process of "transformation-cross-view enhancement-inverse transformation" in the above-mentioned visual connectivity module will go through N cycles to improve the accuracy of regional segmentation and obtain better visual connectivity features. , each cycle will get a fusion feature , N times of looping to get N They are spliced ​​together on the feature channels to obtain the aggregated visual field connectivity features.

8. The method according to claim 1, characterized in that The region segmentation head concatenates the N feature channel dimensions , and input into the fully connected layer, softmax and conditional random field modules for region segmentation tasks, and finally obtain multiple 3D mask results, each of which corresponds to an ear region.

9. The method according to claim 1, characterized in that: The key point localization head estimates the voxel likelihood value of each key point using the voxel encoder and the voxel decoder for the voxel features in the input aggregated visual connectivity features, generates a 3D heat map for each key point, and merges the 3D heat maps of all key points to obtain the 3D coordinate position of each key point. The process is as follows: The voxel encoder uses two downsampling blocks and a 3D residual block stack to further extract the input voxel features and map the high-dimensional voxel data to a low-dimensional space; The voxel decoder uses two upsampling blocks and a 3D residual block stack to expand the spatial size of the feature map and reduce the spacing between voxels, which helps the network locate key points more densely while reducing the number of channels to compress the extracted features; In addition, the voxel encoder and voxel decoder implement voxel-by-voxel addition at each corresponding scale, which enhances the stability of the decoder to the feature mapping. The voxel features in the output of the voxel decoder and the aggregated visual field connectivity features of the input are spliced ​​through a 3D residual block, and then dimension reduction and probability estimation are performed through the 3D residual block and the softmax activation function. The probability estimate of each key point is described by a 3D heat map, where the larger the element value, the higher the probability of the key point at that position, thereby completing the position prediction of the key point.

10. A cross-modal outer ear region segmentation and key point location system, characterized in that: The system comprises: An acquisition module is used to acquire multimodal outer ear data to be segmented and located, wherein the multimodal outer ear data includes an outer ear RGB image, a depth map and a 3D point cloud; A segmentation and positioning module, used for inputting the multimodal outer ear data into a multimodal outer ear region segmentation and key point positioning multi-task end-to-end model, wherein the end-to-end model includes a feature extraction module for each modal data, a modality convergence module, a visual field connection module, a region segmentation head and a key point positioning head; The feature extraction modules of each modality data extract features of each modality, including a depth map feature extraction module, an RGB image feature extraction module, a point cloud feature extraction module and a voxel feature extraction module, respectively extracting depth map features, RGB image features, point cloud features and voxel features; The model convergence module uses image features to enhance voxel features and depth map features, including a voxel and RGB image feature fusion module and a depth map and RGB image feature fusion module, to obtain image enhanced voxel view features and image enhanced depth map features respectively; The visual field connection module performs trimodal aggregation on the three-dimensional point cloud features, the image-enhanced voxel view features, and the image-enhanced depth map features to obtain aggregated visual field connection features; The region segmentation head performs region segmentation on the aggregated visual field connectivity features to obtain an outer ear region result; The key point positioning head performs key point positioning on the voxel features in the aggregated visual field connectivity features to obtain the 3D coordinate position of each key point.

Images