Real-time auricle identification and auricular point positioning method and system for multi-task feature sharing

Abstract

The invention discloses a real-time auricle identification and auricular point positioning method and system for multi-task feature sharing. An auricle identification and auricular point positioning combined model used in the method comprises an encoder and a decoder. The encoder includes a backbone network and a neck network. The decoder comprises a plurality of detection heads. A plurality of detection heads receive outputs of different layers of the neck network. The detection head comprises a target regression branch, a target classification branch and a feature point regression branch which are parallel. A plurality of detection heads share output features of the same encoder, and parallel target regression branches, target classification branches and feature point regression branches in the detection heads are respectively used for calculating loss functions of different tasks; therefore, auricle identification, frame selection and auricular point feature point marking tasks can be efficiently completed; on the basis, high-frame-rate real-time prediction is realized in the field of auricular point positioning for the first time, and the limitation of the prior art is broken through.

Description

Technical Field

[0001] The present invention belongs to the technical field of ear acupoint positioning, and in particular relates to a real-time auricle recognition and ear acupoint positioning method and system with multi-task shared features. Background Art

[0002] Auricular acupuncture therapy is a widely used traditional Chinese medicine therapy. By pressing and stimulating specific acupuncture points on the ear, it can achieve effects such as regulating insomnia, treating obesity, and relieving pain. However, the location of auricular acupuncture points on the auricle lacks intuitive visual markings, and the complexity of the auricle shape and the distribution of many acupuncture points in an area of ​​only 25 square centimeters make it a challenge to quickly and accurately locate these acupuncture points. Beginners face great difficulties in determining the location of auricular acupuncture points. This may not only lead to a decrease in treatment effectiveness, but also hinder the promotion and popularization of auricular acupuncture therapy.

[0003] The auricle structure contains abundant protrusions and depressions, and is considered to be a special and quantifiable biological feature. According to the Chinese national standard "GB / T13734-2008", there are 91 acupuncture points on the auricle that are closely related to the ear structure. The core task of accurately locating these acupuncture points is to align the auricle structure, so as to determine their specific positions based on the relative positions of the acupuncture points. However, due to the diversity and complexity of the shape of the ear, traditional machine learning methods such as the AAM model or acupuncture point positioning technology based on the two-stage algorithm have problems such as low efficiency and excessive resource consumption, and it is difficult to meet real-time requirements.

[0004] The development of augmented reality technology provides an intuitive guidance tool for auricular acupoint learning and treatment, especially solutions based on head-mounted display devices or mobile devices, which can display acupoints in real time according to the acquired images to help beginners quickly master auricular acupoint positioning. However, existing auricular acupoint positioning algorithms usually adopt a two-stage method, first identifying the auricular area and then detecting feature points. This method can achieve good performance under specific acquisition conditions, but the algorithm performance is often limited in scenarios where the auricle is partially blocked or the acquisition angle is not ideal. In addition, when the two-stage method is deployed on resource-constrained mobile terminal devices, it is difficult to meet the requirements of high frame rate real-time performance due to its large amount of computation.

[0005] In existing research, there is a lack of high-quality annotated datasets in the field of auricle research, and most existing auricle feature point positioning research is mainly based on traditional two-dimensional images. In order to cope with the diversity of subject detection head directions and postures, the ability to observe the locations of required acupoints in real time has become an urgent need. There is still much room for improvement in performance under non-ideal acquisition scenarios. Summary of the invention

[0006] The present invention aims to solve the problem of auricular acupoint positioning and realize the combination of real-time positioning and high precision. A real-time auricular acupoint positioning method and system based on multi-task learning are proposed. The method can efficiently identify acupoints on the auricle under non-ideal acquisition conditions, and extract key point features in real time, providing technical support for the intelligentization of traditional Chinese medicine ear diagnosis.

[0007] In a first aspect, the present invention provides a method for real-time auricle recognition and ear acupoint location based on multi-task shared features, which comprises the following steps:

[0008] A joint model for auricle recognition and auricle acupoint location is constructed. The joint model for auricle recognition and auricle acupoint location includes an encoder and a decoder. The encoder includes a backbone network and a neck network. The decoder includes multiple detection heads. The multiple detection heads receive outputs of different layers of the neck network. The detection heads include parallel target regression branches, target classification branches, and feature point regression branches.

[0009] The image under test collected in real time is input into the joint model of auricle recognition and auricle acupoint location. The encoder extracts features from the image under test. The target regression branch in the detection head outputs the position of the auricle in the image under test, and the target classification branch identifies whether the auricle exists in the image under test. The feature point regression branch identifies the position of the auricle on the auricle. The target regression branch and the target classification branch jointly complete the task of auricle recognition and selection; the feature point regression branch completes the task of auricle location.

[0010] The decoder with decoupling design can independently handle the classification, regression and feature point regression subtasks in the detection task. The integration of task decoupling and feature sharing enhances the adaptability of the network, enabling it to handle diverse scales and semantic complexity. The dynamic architecture enhances the scalability and versatility of the model and enriches its multi-task capabilities.

[0011] Preferably, the backbone network includes a plurality of deformable convolutional deep feature extraction layers. The deformable convolutional deep feature extraction layer includes a deformable convolution block, a separation operation, a bottleneck structure and a splicing layer. The deformable convolution block dynamically adjusts the sampling position according to the offset. The convolution layer is generated by an additional convolution layer. The intermediate features output by the deformable convolution block are divided into two parts through a separation operation; the first part is directly input to the splicing layer; the second part is input to the splicing layer after passing through the bottleneck structure; the two parts of the features input to the splicing layer are spliced ​​and then passed through a deformable convolution to obtain the output features of the deformable convolutional deep feature extraction layer.

[0012] Preferably, the bottleneck structure comprises deformable bottleneck blocks connected in sequence, and the convolution in each deformable bottleneck block uses a deformable convolution.

[0013] The deformable convolution has high adaptability and can dynamically adjust the position and shape of the target. The deformable convolution module in the network is designed in this way. By combining the deformable convolution and feature fusion strategy, the deformable convolution deep feature extraction layer aims to divide the input feature map along the channel dimension and integrate high-level and low-level features. The deformable convolution block is used to extract low-level features, and the bottleneck structure extracts high-level features through multiple deformable bottleneck blocks. The final output is fused through the splicing layer to achieve feature separation and fusion.

[0014] Preferably, the backbone network comprises a plurality of standard convolution blocks in series; a deformable convolutional deep feature extraction layer is arranged after each of the last plurality of standard convolution blocks.

[0015] Preferably, the neck network adopts a path aggregation network structure; a pooling local aggregation module is provided in the upsampling part of the neck network. The pooling local aggregation module includes a convolution layer and multiple maximum pooling layers connected in sequence. The output features of the convolution layer and the multiple maximum pooling layers are spliced ​​and then output through convolution.

[0016] Preferably, the specific structure of the neck network includes an upsampling part and a downsampling part connected in sequence. The upsampling part includes a plurality of feature extraction blocks connected in sequence by upsampling. The downsampling part includes a plurality of feature extraction blocks connected in sequence by downsampling. The feature extraction blocks located in the same layer in the upsampling part and the downsampling part are jump-connected. The first feature extraction block in the upsampling part is a pooling local aggregation module. The outputs of the plurality of deformable convolutional deep feature extraction layers in the backbone network are respectively connected to different feature extraction blocks in the upsampling part.

[0017] Preferably, the target regression branch includes serial standard convolution, 2D convolution and bounding box loss function (BboxLoss). The target classification branch includes serial standard convolution, 2D convolution and classification loss function (Cls Loss). The feature point regression branch includes serial standard convolution, 2D convolution and key point loss function (Landmark loss).

[0018] The classification loss function is constructed based on cross entropy loss; the bounding box loss function is constructed based on distribution focus loss and complete intersection-over-union loss; and the key point loss function is constructed based on key point similarity.

[0019] In a second aspect, the present invention provides a real-time auricular acupoint locating system with multi-task shared features, which is used to execute the aforementioned real-time auricular acupoint recognition and locating method with multi-task shared features; the real-time auricular acupoint locating system comprises an image acquisition module, an auricular acupoint recognition and locating module, and a display output module. The image acquisition module is used to acquire the measured image in real time. The auricular acupoint recognition and locating module is used to detect the auricular position in the measured image and multiple auricular acupoint feature points on the auricular acupoint through the auricular acupoint recognition and locating joint model. The display output module is used to display the measured image with the auricular acupoint feature points marked in real time.

[0020] Preferably, the display output module adopts a head-mounted display device.

[0021] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; the processor executes the aforementioned real-time auricle recognition and ear acupoint positioning method with multi-task shared features.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The multiple detection heads in the present invention share the output features of the same encoder, and the parallel target regression branch, target classification branch, and feature point regression branch in the detection head are used to calculate the loss functions of different tasks respectively; thus, the tasks of auricle recognition, frame selection, and auricular feature point marking can be completed efficiently; based on this, the present invention realizes high frame rate real-time prediction in the field of auricular point positioning for the first time, breaking through the limitations of the existing technology. Experimental results show that the algorithm has shown significant advantages on various types of data sets, and can still effectively maintain excellent performance even in non-ideal acquisition scenarios such as partial occlusion of the ear.

[0024] 2. The present invention organically integrates multiple tasks into a single model, optimizes the use of computing resources by sharing a backbone network, significantly shortens the inference time, and realizes the organic combination of real-time performance and high-precision auricular acupoint positioning, which is suitable for application in head-mounted display devices, thereby marking the auricular acupoint positions on the auricle of the object for the user in real time. In addition, the auricle recognition and auricular acupoint positioning joint model provided by the present invention enhances the ability to capture multi-scale features and the ability to adapt to complex scenes, while ensuring high robustness and high precision of positioning by expanding the receptive field and enriching the global context information.

[0025] 3. The backbone network in the present invention introduces a deformable convolution mechanism in multiple convolution modules, thereby extracting richer feature information from the input image, thereby improving the model's ability to express complex and detailed features; the present invention enables real-time and accurate detection of ear points in complex environments such as low light, occlusion or deformation, which can effectively avoid missed detection and significantly improve the performance and robustness of the model in complex scenarios.

[0026] 4. The pooling local aggregation module in the present invention utilizes a sequential structure to promote the gradient flow of the entire network, which not only solves the problem of redundant feature extraction, but also reduces the computational cost while maintaining a consistent receptive field, thereby improving the processing speed of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of an embodiment of the present invention.

[0028] Figure 2 Schematic diagram of an auricle image dataset used in an embodiment of the present invention.

[0029] Figure 3 A reference diagram for marking the positions of ear acupoint feature points used in an embodiment of the present invention.

[0030] Figure 4 This is a model architecture diagram of the auricle recognition and ear acupoint positioning joint model adopted in an embodiment of the present invention.

[0031] Figure 5 Schematic diagram of the structure of a deformable convolutional deep feature extraction layer (C2F-DCN) in an embodiment of the present invention.

[0032] Figure 6 Schematic diagram of the structure of the pooling local aggregation module (SPPELAN) in an embodiment of the present invention.

[0033] Figure 7 This is a diagram showing the positioning result of ear acupoint feature points on a clear auricle image according to an embodiment of the present invention.

[0034] Figure 8 Schematic diagram of the positioning result of ear acupoint feature points under non-ideal conditions according to an embodiment of the present invention.

[0035] Fig. 9 This is a diagram showing the robustness effect of real-time positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] Example 1

[0039] like Figure 1 As shown, a real-time auricle recognition and ear point positioning method with multi-task shared features includes the following steps:

[0040] Step 1: Collect ear image data sets and perform data preprocessing; data preprocessing includes data enhancement and normalization of the data sets.

[0041] In this embodiment, RGB images of the left and right ears of 432 volunteers were collected and captured using a professional auricle image acquisition camera. Figure 2 As shown in the figure, the subject stood in front of the camera, with the auricle fully exposed, facing the camera at a 90° angle, occupying 80% of the screen area, and the background was simple and clean. According to the requirements of the Chinese national standard "GB / T 13734-2008", 91 ear acupoint feature points closely related to the auricle structure were marked, such as Figure 3 As shown in the figure, it is used to capture the features of the auricle and align the auricle structure. All the annotation work was completed by professional Chinese medicine ear doctors, and finally a high-quality ear acupoint feature point dataset (EID) was established, providing reliable data support for subsequent research. In order to verify the model's ability to locate auricles under non-ideal conditions, the ear database in the wild environment (ITWE) was used, which covers auricle images collected in non-ideal and unconstrained environments. At the same time, the EID database was used to evaluate the accuracy of the model in the task of locating auricle feature points, while the ITWE database was used to test the performance of the model in complex non-ideal environments.

[0042] The data enhancement process is as follows: To improve the diversity of the dataset and reduce the asymmetry of left and right ear characteristics, the EID dataset uses a mirror flip operation to convert the right ear image to the left ear and vice versa. Subsequently, the number of training samples in the dataset is expanded by rotating it every 15° within the range of ±90°.

[0043] The dataset is divided into training and test sets in a ratio of 8:2. The images in the ITWE dataset are from online resources. After cropping, the ear area is extracted and the size is adjusted. At the same time, the coordinates of the 55 annotation points are transformed synchronously to maintain the relative position relationship. The cropped images are uniformly normalized to ensure data consistency and enhance the generalization ability of the model. This preprocessing strategy significantly optimizes the training effect.

[0044] Step 2: Design and build a joint model of auricle recognition and ear acupoint positioning based on deep learning to extract key feature points in auricle images.

[0045] like Figure 4 As shown in the figure, the joint model of auricle recognition and auricular point location adopts an end-to-end encoder-decoder single-stage network design, including an encoder and a decoder. The encoder includes a backbone network and a neck network. The decoder includes multiple detection heads (heads); multiple detection heads of the model share the same encoder, so that they can share features. The backbone network is responsible for feature extraction and is passed to the neck network as the input of the feature pyramid network (FPN). The backbone network introduces a deformable convolution mechanism through multiple convolution modules to extract richer feature information from the input image. The neck network is designed as a bidirectional network to fuse features from different levels and promote cross-scale feature fusion. The decoder adopts a multi-task learning strategy to design the detection head structure and uses multiple detection heads for target detection and feature point location tasks. The detection head includes multiple parallel convolution modules; these convolution modules receive tensors from the neck network and are used to calculate the loss functions of different tasks.

[0046] The backbone network consists of a series of convolutional layers for feature extraction. In the ear feature point recognition task, the complex structure of the ear and the unique shape of each person's ear are considered. For this reason, a deformable convolution (DCN) is introduced into the backbone network based on CSP-Darknet to dynamically adjust the position of the convolution kernel to obtain a deformable convolution deep feature extraction layer (C2F-DCN).

[0047] The backbone network includes five standard convolution blocks (CBS) connected in sequence, a deep feature extraction layer arranged after the second standard convolution block, and three deformable convolution deep feature extraction layers (C2F-DCN) arranged after the third, fourth, and fifth standard convolution blocks respectively.

[0048] like Figure 5 As shown in the figure, the deformable convolutional deep feature extraction layer (C2F-DCN) includes four submodules: a deformable convolutional block (DBS), a separation operation (Split), a bottleneck structure, and a concatenation layer (Concat) connected in sequence. The goal of the deformable convolutional deep feature extraction layer (C2F-DCN) is to divide the input feature map along the channel dimension and fuse high-level and low-level features.

[0049] like Figure 5 As shown in the figure, the deformable convolution block (DBS) includes deformable convolution, batch normalization and SiLU activation function connected in sequence to extract low-level features. Deformable convolution introduces offset and modulation factor on the basis of traditional convolution. By introducing offset, the convolution kernel can dynamically adjust its sampling position, breaking through the limitation of fixed sampling position of traditional convolution.

[0050]

[0051] Among them, the value of the output feature map z at position q; the weight of the convolution kernel at the sampling point m; is the learned offset, which represents the dynamic offset of the mth sampling point of the convolution kernel; is the learned modulation factor, which represents the weighting coefficient of the mth sampling point; As the dynamically adjusted sampling position, since the sampling position may be a non-integer, a bilinear calculation is usually used.

[0052] The offset is generated by an additional convolutional layer, and the output of the convolutional layer is split into three parameters, corresponding to the offset of the convolution kernel in the x and y directions. Finally, these offsets are concatenated in the channel dimension to form a complete offset to guide the dynamic sampling of the convolution kernel. The modulation factor is responsible for controlling the weighting coefficient of each position of the convolution kernel. Its function is to assign different weights to each sampling point according to its importance, thereby improving the model's ability to express complex and detailed features. As a result, especially in complex environments, such as low-light and occluded auricle detection, missed detection can be effectively avoided, significantly improving the performance and robustness of the model in complex scenarios.

[0053] The input feature map is passed through a deformable convolution block (DBS) to extract basic features and generate an intermediate feature map. Subsequently, the intermediate feature map is split into two parts, one part is directly passed to the final concatenation layer (Concat), and the other part is input to the bottleneck structure for further processing. The bottleneck structure includes multiple sequentially connected layers for extracting high-level features. The final output is merged to promote the separation and fusion of features.

[0054] In the deformable bottleneck block (Bottleneck), a deformable convolution layer is introduced to replace the traditional standard convolution block to significantly improve the flexibility and adaptability of the convolution kernel, so as to more accurately capture the fine-grained feature information of the target and improve the perception of the model separation operation type to complex morphology and detail changes. After processing, this part of the feature map is concatenated with the intermediate feature map directly passed to the concatenation layer (Concat) to fuse multi-level information. The concatenated feature map is further processed by a final deformable convolution block to generate the final output feature map. By dynamically adjusting the position and configuration of the receptive field, this method ensures that the model can capture the details of the ear, reduce information loss and blur, and maintain high performance under complex ear shapes.

[0055] The neck network plays a key role in integrating the features extracted by the backbone network, maximizing parameter efficiency and computational effect while maintaining high accuracy. The neck network adopts a lightweight framework designed to accelerate inference speed without compromising accuracy, thereby improving the utility of the computing module in various application scenarios.

[0056] The neck network receives input from the three layers of features of the backbone network. A path aggregation network (PAN) structure is adopted, which combines the top-down and bottom-up feature pyramid networks (FPN). This structure combines low-level details with high-level semantic features to improve the overall feature representation capability. Its advantage is that it can effectively extract and synthesize multi-scale features without losing key information, so that the model can detect multi-scale targets more accurately and quickly.

[0057] The specific structure of the neck network includes an upsampling part and a downsampling part. The upsampling part includes three feature extraction blocks connected by upsampling in sequence. The downsampling part includes two feature extraction blocks connected by downsampling in sequence. The upsampling part includes three feature extraction blocks connected by upsampling in sequence, which respectively receive the output features of three deformable convolutional deep feature extraction layers (C2F-DCN) in the backbone network. The feature extraction blocks in the same layer in the upsampling part and the downsampling part are jump-connected.

[0058] The first feature extraction block of the upsampling part is the pooling local aggregation module (SPPELAN). The second and third feature extraction blocks of the upsampling part are the same as the two feature extraction blocks of the downsampling part, both of which include a deep feature extraction layer (C2F) and a concatenation layer (Concat). The neck network establishes connections between different feature extraction blocks, enriches the gradient flow, and enhances the model's ability to represent key features while ensuring that the model remains lightweight.

[0059] like Figure 6 As shown in Figure 1, the pooling local aggregation module (SPPELAN) includes a standard convolution block (CBS), three sequentially arranged maximum pooling layers (MaxPool2d), and a concatenation layer (Concat). After the initial CBS unit reduces the number of input feature map channels, preliminary features are extracted. The concatenation layer merges the output features of the standard convolution block (CBS) with each maximum pooling layer, and finally outputs the result through a standard convolution block (CBS) and converts it to the required number of output channels.

[0060] The traditional SPP module performs pooling operations with different kernel sizes in parallel to extract information at different scales. The result is concatenated with the original feature map along the channel dimension, solving the multi-scale problem. In contrast, the pooling local aggregation module (SPPELAN) utilizes a sequential structure to facilitate the gradient flow of the entire network. By applying convolution kernels of the same size to the downsampled input feature maps, it effectively maintains spatial information and captures features of different scales. Since each maximum pooling layer (MaxPool2d) can be regarded as an independent layer, the gradient can be back-propagated through multiple paths, thereby alleviating the problem of gradient vanishing or exploding. Therefore, the pooling local aggregation module (SPPELAN) not only solves the problem of redundant feature extraction, but also reduces the computational cost while maintaining a consistent receptive field, thereby improving the processing speed of the network.

[0061] The output features of the last upsampling feature extraction module and the two downsampling feature extraction modules are connected to the three detection heads of the decoder respectively. The decoder is responsible for parsing and processing the data from the neck feature layer and generating prediction results for each task. The three detection heads share one encoder, which can effectively share and utilize the relevant information between different tasks during the learning process. This design not only reduces the complexity and redundancy between different encoders, but also improves the overall learning and prediction performance.

[0062] In order to achieve accurate positioning of ear point feature points, the model adopts a multi-task learning method and is modeled through a unified network framework with feature sharing. A dedicated feature point decoder is added on the basis of a single decoder to handle feature point positioning tasks.

[0063] In addition, the decoder adopts a task decoupling design, which independently processes classification, regression, and feature point regression subtasks. By combining task decoupling with feature sharing, the network's adaptability is significantly enhanced, and it can better cope with multi-scale and complex semantic scenes. This dynamic architecture not only improves the scalability and versatility of the model, but also gives it stronger multi-task processing capabilities. In addition, the traditional anchor-based structure is optimized into an anchor-free structure. Instead of predefined anchors, the center point, width, and height of the target are directly regressed on feature maps of different scales, thereby effectively reducing computational costs and time overhead, and further improving the efficiency of the model.

[0064] The specific structure of the detection head includes three detection branches, namely, target regression branch, target classification branch and feature point regression branch. The target regression branch includes two layers of standard convolution blocks (CBS), 2D convolution layer (Conv2d) and bounding box loss function (Bbox Loss) connected in sequence. The target classification branch includes two layers of standard convolution blocks (CBS), 2D convolution layer (Conv2d) and classification loss function (Cls Loss) connected in sequence. The feature point regression branch includes two layers of standard convolution blocks (CBS), 2D convolution layer (Conv2d) and key point loss function (Landmark loss) connected in sequence.

[0065] During the training phase, the feature point regression task and the target detection task (including bounding box prediction, classification label and confidence score) are jointly optimized; during the reasoning and prediction phase, the efficient processing of the overall task is achieved through the joint regression mechanism.

[0066] Step 3: Set appropriate training strategies and loss functions to fully train the constructed model.

[0067] In the training process of this embodiment, the model is trained from scratch, the learning rate is set to 1e-2, the optimizer uses stochastic gradient descent (SGD), and the batch size is 16. This embodiment adopts a multi-task learning strategy, and designs corresponding loss functions according to different tasks. In the design of the loss function, multiple tasks such as target classification, bounding box regression, and feature point regression are taken into account. The classification task uses binary cross entropy loss (BCE) to evaluate the deviation between the prediction and the actual label; the bounding box regression uses distribution focus loss (DFL) and complete intersection over union loss (CIoU) for optimization to improve the prediction accuracy of the target position and size; the feature point regression task uses the key point similarity (OKS) metric to evaluate the positioning accuracy of the key points; the existence of the key points is trained using binary cross entropy loss (BCE). Finally, the total loss function is the weighted sum of all task losses. By adjusting the weight coefficients of each loss function, the influence between different tasks is balanced, thereby improving the overall performance and generalization ability of the model.

[0068] Classification loss function: Binary cross entropy loss (BCE) is used to evaluate the deviation between the predicted probability and the actual label. The specific loss function is defined as follows:

[0069]

[0070]

[0071] in, The loss weight of each batch element is used to adjust the contribution of the sample to the overall loss. is the actual label of the nth sample, is the predicted probability of the nth sample after being processed by the Sigmoid function.

[0072] Bounding box loss function: For the bounding box prediction task, a fused distribution focus loss (DFL) and complete intersection over union loss (CIoU) are used to optimize the position and size prediction of the target. The DFL loss captures the position of the bounding box through a generalized distribution, and the CIoU loss comprehensively considers the overlap, centroid distance, and aspect ratio consistency of the target bounding box to improve the accuracy of target detection. The loss function is defined as follows:

[0073]

[0074] Among them, the formula Describes a weighted cross entropy loss for optimizing the prediction accuracy of target locations. represents the probability distribution predicted by the model, is the target position, which is a normalized floating point number; and are the weights of the left and right boundaries of the target, respectively, controlling the contribution of the boundaries to the loss. The Sigmoid function is used to map the predicted value to a probability range (0 to 1). and Represent the model's predicted probabilities for the left and right boundary locations, respectively.

[0075] CIoU not only focuses on the intersection over union (IoU) of the predicted box and the true box, but also improves the accuracy of the detection box position by minimizing the Euclidean distance between the center points of the predicted box and the true box. At the same time, it introduces aspect ratio consistency constraints to make the shape and proportion of the predicted box closer to the true box. This multi-dimensional optimization method can help the model learn the shape, size, and orientation characteristics of the target more accurately, thereby achieving higher accuracy and robustness in target detection tasks. For a target at coordinate And the scale is The grid unit , the CIoU loss between the predicted box and the true box is defined as:

[0076]

[0077] in, represents the real detection box, Represents the detection box predicted by the model. The loss is obtained by subtracting CIoU to get a non-negative value. The closer the CIoU value is to 1, the more consistent the predicted box is with the true box, and the smaller the loss is. By optimizing this loss function, the model can learn the position, size, and shape characteristics of the detection box more accurately.

[0078] Key point loss function: In the auricular feature point recognition task, the key point similarity index (OKS) is used to evaluate the performance of the key point regression task. The OKS metric ranges from 0 to 1, and the higher the value, the higher the accuracy of the key point position. The key point loss function is defined as follows:

[0079]

[0080] Keypoint objectness loss: To train the existence of each keypoint, a binary cross entropy loss is used to measure the error between the confidence of the prediction and the ground truth label. .in, Represents the number of key points, It is used to measure the accuracy of the predicted position and the true position of the nth key point. In addition, the confidence loss Used to quantify the uncertainty of whether each key point exists. This is achieved by using binary cross entropy, BCE is used to measure the difference between the predicted confidence of the nth key point and its true label. The loss function is defined as follows:

[0081]

[0082]

[0083] For the outputs of multiple detection heads, the bounding box and category predictions are filtered by non-maximum suppression (NMS). In some other embodiments, the prediction results of the bounding box and the key point are combined by a weighted average method.

[0084] Step 4: Model verification and optimization: Test the trained model on the verification set, evaluate its performance, and fine-tune and optimize the model based on the verification results.

[0085] The trained model is tested on the test set to evaluate its performance, and the model is fine-tuned and optimized based on the test results. First, the model is comprehensively analyzed using multi-task evaluation indicators. For the ear recognition task, mean average precision (mAP) is used as the core evaluation indicator. mAP comprehensively measures the detection performance by calculating the accuracy score of the model at different recall levels. In particular, the mAP50-95 indicator combines the evaluation results of multiple IoU thresholds to fully reflect the performance of the model in different detection difficulty scenarios, ensuring that the evaluation results are more comprehensive and representative.

[0086]

[0087] in, is the number of categories, yes The average precision of the class is the precision value calculated by averaging under multiple recall rates. The AP of each category is obtained by calculating the precision under different recall rates and then averaging these precision values.

[0088] For the auricular point localization task, the normalized mean error (NME) was used as the evaluation criterion. The ratio of the Euclidean distance of each key point to the width of the ear was calculated to evaluate the degree of match between the predicted position and the actual position.

[0089]

[0090] in, Represents the number of ear point feature points in the image, and Respectively represent The actual coordinates of the feature points and the coordinates predicted by the model, is the normalization factor.

[0091] Step 5: Real-time auricular acupoint positioning. The optimized auricular acupoint recognition and auricular acupoint positioning joint model is used to realize real-time recognition of auricular acupoint feature points in auricular images under any shooting conditions and output the corresponding positioning coordinates.

[0092] This embodiment illustrates the effect achieved by the present invention in combination with the accompanying drawings and experimental data. To further verify the performance of the model, the contribution of each component in the model, including the C2F-DCN module and the SPPELAN structure, is evaluated through ablation experiments, and the results are shown in Table 1. First, to evaluate the impact of the C2F-DCN module, we compared the results of the complete model containing the module with the results of omitting the module. The C2F-DCN module significantly improves the feature extraction capability by dynamically adapting to the shape and position of the ear features. The experimental results on the EID dataset show that the model integrating DCN into the backbone network achieves an mAP value of 0.994 in the auricle detection task and significantly optimizes the NME index in the feature point detection task. This shows that compared with standard convolution, deformable convolution can not only significantly improve the detection accuracy, but also effectively reduce the computational cost. Secondly, the pooled local aggregation module (SPPELAN) combines spatial pyramid pooling (SPP) with the ELAN structure, which significantly reduces the computational overhead while improving the multi-scale feature expression capability. After applying the pooling local aggregation module (SPPELAN) to the neck layer of the model, it shows better performance in both mAP and NME indicators compared with the model using the SPPF module. In addition, the pooling local aggregation module (SPPELAN) successfully reduces the number of model parameters while improving computational efficiency. By integrating C2F-DCN and the pooling local aggregation module (SPPELAN) in the model, better performance was achieved in both auricle detection and feature point detection tasks. The model not only significantly improves the detection accuracy, but also effectively reduces the computational complexity and accelerates the inference speed, further verifying the superiority of the proposed method.

[0093] Table 1. Comparison of ablation experiments

[0094]

[0095] To demonstrate the effectiveness and performance of the proposed real-time location model for ear acupoint feature points, it was compared with several state-of-the-art methods in an experimental environment. As shown in Table 2, two sets of experiments were conducted on the test subset of the EID dataset to verify the performance of the model. The first set of experiments evaluated the proposed model for the ear detection task and compared it with the current mainstream object detection methods (including YOLOv5, YOLOv8, and the latest YOLOv9), using mAP50-95(%) as the accuracy evaluation indicator. The second set of experiments focused on the ear acupoint detection task and evaluated the performance of the model with other competing models, including the multi-task model TCDCN for face alignment and the ear acupoint detection cascade method proposed by Sun et al. These experiments are designed to comprehensively evaluate the performance of the model in different tasks and ensure its feasibility in practical applications. In the first set of experiments, the accuracy of the model of this embodiment was improved by 12.1% compared with YOLOv5, demonstrating the best auricle detection performance. In the second set of experiments, the model of this embodiment showed excellent adaptability in detecting complex shapes and small ear feature areas, which was improved by 75.9% compared with the face alignment algorithm. Compared with traditional auricular point detection methods, the detection accuracy of the model is improved by 66.5%, achieving the highest positioning accuracy in this field. This significant improvement is partly attributed to feature sharing within the multi-task model and module optimization designed for the complex morphology of the auricle. To evaluate the performance of the model under challenging and non-ideal conditions, the experimental results are detailed in Table 3. The experiments were conducted on the ITWE dataset and the results were compared with the current advanced methods. Zhou et al. conducted positioning experiments using multiple methods on the ITWE dataset, while Hrovatič et al. used a double-stacked hourglass model for prediction. In contrast, our method achieved the best NME result on this dataset, with a score of 0.0448, which is 8% higher than the second-ranked method. These experimental results verify the excellent performance of the model in multiple scenarios, and further demonstrate its adaptability and robustness in complex shooting scenarios.

[0096] Table 2 Comparison results of EID dataset

[0097]

[0098] Table 3 Comparison results of ITWE dataset

[0099]

[0100] The present invention proposes a method for real-time auricular acupoint localization based on multi-task learning, which significantly improves computational efficiency and real-time performance through an end-to-end framework design. The method optimizes parameter utilization through information sharing between tasks, reduces redundant calculations, and achieves fast reasoning. It takes only 0.0161 seconds to complete the localization of a single image, which is 28 times faster than the traditional cascade method and has a high frame rate of 60FPS. In addition, in order to demonstrate the effect achieved by the invention, Figure 7 As shown, the model provided in this embodiment can accurately locate the ear acupoint feature point information on the auricle. Figure 8 As shown in Figure 2, even under non-ideal conditions such as low resolution, illumination changes, and occlusion, accurate detection and positioning can still be achieved. Fig. 9 As shown, this embodiment has strong adaptability to different viewing angles and can operate stably at an inclined angle (30°), showing robustness and high efficiency in complex scenes.

[0101] Compared with the traditional auricular feature point positioning method, the optimized model has obvious advantages in processing speed and accuracy. Especially in complex and dynamic actual scenes, it can quickly respond and output positioning results, providing strong support for clinical applications and intelligent diagnosis and treatment. This real-time positioning capability will greatly improve the efficiency of auricular point recognition and help play an important role in the intelligent development of traditional Chinese medicine ear diagnosis and other fields.

[0102] Example 2

[0103] A real-time ear acupoint positioning system with multi-task shared features is used for the real-time auricle recognition ear acupoint positioning method described in Example 1; the real-time ear acupoint positioning system comprises an image acquisition module, an auricle recognition ear acupoint positioning module and a display output module; the image acquisition module is used to acquire the measured image in real time; the auricle recognition ear acupoint positioning module is used to detect the auricle position in the measured image and multiple ear acupoint feature points on the auricle through an auricle recognition ear acupoint positioning joint model; the display output module is used to display the measured image with the ear acupoint feature points marked in real time.

[0104] In some preferred embodiments, the display output module adopts a head-mounted display device.

[0105] In the drawings of the embodiments disclosed in the present invention, only the structures involved in the embodiments disclosed in the present invention are involved, but the above is only a preferred embodiment of the present invention, which can be fully applied to various fields suitable for the present invention. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrations shown and described here.

Claims

1. A real-time auricle recognition and ear point positioning method with multi-task shared features, characterized by: The following steps are involved: Constructing a joint model for auricle recognition and ear acupoint location; the joint model for auricle recognition and ear acupoint location includes an encoder and a decoder; the encoder includes a backbone network and a neck network; the decoder includes multiple detection heads; the multiple detection heads receive outputs of different layers of the neck network; the detection head includes a parallel target classification branch, a target regression branch and a feature point regression branch; The image under test is input into the joint model of auricle recognition and auricular point positioning; the encoder extracts features from the image under test; the target classification branch in the detection head identifies whether there is an auricle in the image under test; the target regression branch outputs the position of the auricle in the image under test; The feature point regression branch identifies the locations of the auricular points on the auricle.

2. The method for real-time auricle recognition and ear point location based on multi-task shared features according to claim 1, characterized in that: The backbone network comprises a plurality of deformable convolutional deep feature extraction layers; the deformable convolutional deep feature extraction layer comprises a deformable convolutional block, a separation operation, a bottleneck structure and a splicing layer; the deformable convolutional block dynamically adjusts the sampling position according to the offset; the offset is generated by the convolutional layer; the intermediate features output by the deformable convolutional block are divided into two parts through a separation operation; the first part is directly input into the splicing layer; the second part is input into the splicing layer after passing through the bottleneck structure; the two parts of the features input into the splicing layer are spliced ​​and then passed through the deformable convolution to obtain the output features of the deformable convolutional deep feature extraction layer.

3. The method for real-time auricle recognition and ear point location based on multi-task shared features according to claim 2, characterized in that: The bottleneck structure includes deformable bottleneck blocks connected in sequence; the convolution in each deformable bottleneck block uses deformable convolution.

4. The method for real-time auricle recognition and ear point location based on multi-task shared features according to claim 2, characterized in that: The backbone network includes a plurality of standard convolution blocks in series; a deformable convolution depth feature extraction layer is arranged after each of the last plurality of standard convolution blocks.

5. The method for real-time auricle recognition and ear point location based on multi-task shared features according to claim 2, characterized in that: The neck network adopts a path aggregation network structure; a pooling local aggregation module is provided in the upsampling part of the neck network; the pooling local aggregation module includes a convolution layer and multiple maximum pooling layers connected in sequence; the output features of the convolution layer and the multiple maximum pooling layers are spliced ​​and then output through convolution.

6. The method for real-time auricle recognition and ear point location based on multi-task shared features according to claim 5, characterized in that: The specific structure of the neck network includes an upsampling part and a downsampling part connected in sequence; the upsampling part includes a plurality of feature extraction blocks connected in sequence by upsampling; the downsampling part includes a plurality of feature extraction blocks connected in sequence by downsampling; the feature extraction blocks located in the same layer in the upsampling part and the downsampling part are jump-connected; the first feature extraction block in the upsampling part is a pooling local aggregation module; the outputs of the plurality of deformable convolutional deep feature extraction layers in the backbone network are respectively connected to different feature extraction blocks in the upsampling part.

7. The method for real-time auricle recognition and ear point location based on multi-task shared features according to claim 1, characterized in that: The target regression branch includes serial standard convolution, 2D convolution and bounding box loss function; the target classification branch includes serial standard convolution, 2D convolution and classification loss function; the feature point regression branch includes serial standard convolution, 2D convolution and key point loss function.

8. A real-time ear acupoint location system with multi-task shared features, characterized in that: A real-time auricle recognition and ear point positioning method for executing a multi-task shared feature as described in any one of claims 1 to 7; the real-time ear point positioning system comprises an image acquisition module, an auricle recognition and ear point positioning module and a display output module; the image acquisition module is used to acquire the measured image in real time; the auricle recognition and ear point positioning module is used to detect the auricle position in the measured image and multiple ear point feature points on the auricle through an auricle recognition and ear point positioning joint model; the display output module is used to display the measured image with the ear point feature points marked in real time.

9. A real-time ear acupoint location system with multi-task shared features according to claim 8, characterized in that: The display output module adopts a head-mounted display device.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory stores computer programs; the processor executes a real-time auricle recognition and ear acupoint positioning method with multi-task shared features as described in any one of claims 1-7.

Images