A tongue localization method based on improved YOLOv5s

Abstract

This invention discloses a tongue localization method based on an improved YOLOv5s, belonging to the field of target detection technology. Based on the YOLOv5s network architecture, this method introduces a coordinate attention module into the backbone network to enhance the saliency of tongue region features and suppress background interference. In the upsampling path of the neck network, bilinear interpolation is used instead of nearest-neighbor interpolation, and a 3×3 convolutional layer is added after each upsampling for detail compensation to improve the jagged edges of the tongue and restore high-frequency detail information. EIoU Loss is used instead of CIoU Loss as the bounding box regression loss, and Focal Loss is used instead of binary cross-entropy loss as the classification loss to improve bounding box localization accuracy and alleviate the problem of positive and negative sample imbalance. This invention can achieve precise tongue localization, effectively improving the accuracy and robustness of tongue detection.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent tongue positioning in Traditional Chinese Medicine, and more specifically, to a tongue positioning method based on an improved YOLOv5s. Background Technology

[0002] Tongue diagnosis, as an important component of inspection in Traditional Chinese Medicine (TCM)'s "Four Diagnostic Methods," allows for the assessment of the state of Qi and blood, the depth of disease location, and the prognosis of illness by observing changes in the tongue's appearance, thus providing a basis for constitution identification. However, the traditional TCM tongue diagnosis process is easily influenced by factors such as the doctor's knowledge level, diagnostic techniques, and the natural environment, exhibiting a strong degree of subjectivity and hindering the objectification and standardization of diagnosis. Therefore, studying the principles of tongue diagnosis using modern scientific methods has significant theoretical and practical value in addressing fundamental scientific issues that restrict the full realization of TCM's unique advantages.

[0003] Tongue localization is a crucial and indispensable preprocessing step in the entire automated tongue diagnosis process. Its core objective is to automatically and accurately segment the tongue region from a single oral cavity image. Traditional tongue localization methods primarily rely on digital image processing and computer vision technologies, and can be broadly categorized into three types:

[0004] The first category is color space-based methods. These methods convert the image from the RGB color space to a space more suitable for color segmentation (such as HSV or Lab), and separate the tongue pixels from the background pixels by setting an appropriate color threshold. However, changes in lighting conditions can alter the tongue's color representation, causing the fixed threshold to fail and easily resulting in incorrect segmentation.

[0005] The second type of method is based on edge detection. This type of method uses edge detection operators to detect edges in the image and uses contour lookup to determine the most likely closed region of the tongue contour. However, there may be other irrelevant edges in the tongue image, which can interfere with the correct selection of the tongue contour, causing the extracted contour to be broken and unable to form a closed region.

[0006] The third type of method is based on active contour models. This type of method initializes a closed curved contour near the tongue body. Under the combined action of internal and external energy, this curve actively deforms and eventually conforms to the actual boundary of the tongue body. However, deep cracks or depressions that may exist at the leading edge of the tongue body make the contour very prone to converging to erroneous local minima.

[0007] In summary, all three traditional methods mentioned above rely on manually designed low-level visual features, making them ill-equipped to handle interference from factors such as changes in lighting, diverse tongue morphology, and image noise. Therefore, improving the accuracy and robustness of tongue localization has become a pressing technical problem to be solved in this field. Summary of the Invention

[0008] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0009] Therefore, the purpose of this invention is to propose a tongue localization method based on an improved YOLOv5s.

[0010] To achieve the above objectives, the present invention provides a tongue localization method based on an improved YOLOv5s. This tongue localization method based on an improved YOLOv5s includes: Step S1: acquiring a patient tongue image dataset; annotating each tongue image in the dataset with polygonal regions of the tongue body to obtain annotation information for the tongue body regions; extracting the polygon vertex coordinate sequence of each tongue image based on the annotation information, and generating a YOLO format label file corresponding to each tongue image based on the polygon vertex coordinate sequence; Step S2: grouping the tongue image dataset by source based on the patient's medical record number, with all tongue images corresponding to the same medical record number belonging to the same source group; using the source group as the smallest dividing unit, randomly allocating each source group to the training set according to a preset ratio; The validation set and test set are respectively composed of source groups assigned to each subset, forming the training set, validation set, and test set; Step S3: Construct a standardized dataset storage directory; the storage directory includes a parallel image storage main directory and a label storage main directory, each containing three subdirectories corresponding one-to-one with the training set, validation set, and test set; store the image files and their corresponding label files from the training set, validation set, and test set into their respective subdirectories; Step S4: Perform preprocessing and data augmentation on the sample images in the training set, and perform preprocessing on the sample images in the validation set and test set without data augmentation, so as to respectively Generate corresponding standardized input data; Step S5: Build a tongue localization model; The tongue localization model is an improvement on the YOLOv5s model, including an input terminal, a backbone network, a neck network, and a detection head network connected in sequence; The backbone network includes a downsampling module, a feature extraction module, an attention module, and a pooling module arranged in sequence along the downsampling direction; The downsampling module is used to perform spatial resolution downsampling and channel expansion on the input feature map; The feature extraction module is used to perform layer-by-layer convolution operations on the downsampled feature map to extract the edge contour features of the tongue region; The attention module is used to perform spatial position attention weighting on the input feature map to enhance the features of the tongue target region. The activation value is used to suppress the feature response of the background interference region; the pooling module is used to perform multi-scale spatial pooling on the attention-weighted feature map to increase the receptive field of the network and capture the contextual semantic information of the tongue at different scales; the neck network adopts a bidirectional fusion structure combining the feature pyramid structure and the path aggregation network to perform feature fusion on the multi-scale feature map generated by the backbone network; the detection head network is used to perform target detection on the multi-scale feature map after fusion by the neck network, and output the position coordinates of the tongue bounding box and its corresponding category confidence; Step S6: Based on the preset total loss function, the tongue localization model is iteratively trained using the training set until the model converges;The total loss function includes bounding box regression loss, classification loss, and confidence loss; Step S7: During the training process in Step S6, after each preset round of iterative training, the validation set is input into the current model for forward propagation, the loss value on the validation set is calculated, and the training strategy is adjusted according to the changing trend of the loss value, and the model weight with the smallest validation set loss is selected as the training result.

[0011] Step S8: Input the test set into the trained tongue localization model for forward propagation, obtain the predicted tongue bounding box coordinates and confidence scores, and calculate the evaluation index based on the prediction results on the test set and the ground truth bounding boxes to verify the generalization performance of the tongue localization model.

[0012] Preferably, the attention module is a coordinate attention module, which is used to: perform global average pooling in the horizontal direction and global average pooling in the vertical direction on the input feature map to generate a horizontal feature tensor and a vertical feature tensor; concatenate the horizontal and vertical feature tensors along the spatial dimension, and perform channel compression and non-linear activation through a 1×1 convolutional layer to generate an intermediate feature tensor; split the intermediate feature tensor into a horizontal sub-tensor and a vertical sub-tensor along the spatial dimension, and perform channel recovery and sigmoid activation through a 1×1 convolutional layer on each sub-tensor to generate a horizontal attention weight tensor and a vertical attention weight tensor; multiply the input feature map element-wise with the horizontal and vertical attention weight tensors to output a weighted feature map.

[0013] Preferably, the total loss function is composed of the bounding box regression loss, classification loss, and confidence loss, each multiplied by its respective preset weight coefficient and then summed; wherein the bounding box regression loss adopts EIoU Loss, the classification loss adopts Focal Loss, and the confidence loss adopts binary cross-entropy loss.

[0014] Preferably, the preprocessing is a standardized process to ensure that all input images are in the same format, including resizing, pixel normalization and label synchronization; the data augmentation is an operation applied only to the training set to increase sample diversity, including geometric transformation operations and noise generation operations.

[0015] This invention proposes an improved tongue localization method based on the YOLOv5s network architecture. Compared with existing technologies, this invention has the following advantages:

[0016] (1) Based on the data augmentation methods preset in the YOLOv5s network architecture (including but not limited to Mosaic, MixUp, HSV enhancement, and random flipping), a geometric transformation enhancement strategy for tongue morphology variations (such as differences in tongue extension angle and mouth opening amplitude) and a noise generation enhancement strategy for tongue image acquisition environment noise were further designed.

[0017] The geometric transformation involves rotating, translating, scaling, and shearing the tongue image, enabling the tongue positioning model to learn the tongue features at different tongue extension angles and improving its robustness to changes in tongue morphology.

[0018] The noise generation is achieved by injecting random Gaussian noise into the tongue image to simulate the lighting fluctuations and equipment noise during the real tongue image acquisition process, thereby enhancing the tongue localization model's ability to suppress noise interference and improving the stability of tongue localization in complex acquisition environments.

[0019] (2) A coordinate attention module is added between the last C3 module and the SPPF module in the YOLOv5s backbone network. The coordinate attention module generates direction-aware attention weights by performing one-dimensional global pooling on the input feature map along the horizontal and vertical spatial directions. This enables the tongue localization model to retain the spatial location information and channel semantic information of the target during the feature extraction process. Thus, even when the contrast between the tongue and the background is low or the background is complex, it can still accurately focus on the tongue area, effectively suppress background interference, and improve the tongue localization accuracy in complex scenes.

[0020] (3) Optimize the upsampling method in the neck network of YOLOv5s: Replace the nearest neighbor interpolation method used in the original YOLOv5s with the bilinear interpolation method to alleviate the jagged effect and coarse feature map problem during the upsampling process; at the same time, add a 3×3 convolutional layer at the output of the bilinear interpolation upsampling, and use the learnable parameters of the convolutional layer to perform detail compensation on the local texture of the interpolated feature map, effectively restoring the high-frequency edge information lost due to the smoothing of bilinear interpolation;

[0021] Through the above optimizations, the tongue localization model retains the fine structure of the tongue edge while suppressing upsampling artifacts, significantly improving the tongue localization accuracy and the fit between the predicted bounding box and the real tongue edge.

[0022] (4) Combinatorial optimization of the loss function of the YOLOv5s network architecture:

[0023] By replacing CIoU Loss with EIoU Loss as the bounding box regression loss, the problem of unclear gradient direction caused by aspect ratio coupling in CIoU Loss is effectively solved, significantly accelerating the convergence speed of bounding box regression and improving the tongue localization accuracy.

[0024] Using Focal Loss as the classification loss, the weight of easily classified samples is dynamically reduced and the weight of difficult-to-classify samples is increased by introducing a modulation factor, which effectively alleviates the problem of severe imbalance between positive and negative samples in tongue detection, and makes the tongue localization model more focused on tongue targets that are difficult to distinguish during the training process.

[0025] The binary cross-entropy loss is retained as the confidence loss, so that the tongue localization model continuously optimizes the prediction accuracy of the confidence score during the training process, and can effectively distinguish between valid and invalid prediction boxes during the inference stage, thus ensuring the reliability of the output results of the tongue localization model.

[0026] Therefore, the EIoU Loss, Focal Loss, and binary cross-entropy loss optimize the training process of the tongue localization model from three dimensions: boundary regression accuracy, hard sample focusing, and confidence assessment, respectively, so that the tongue localization model has higher localization accuracy and stronger anti-interference ability in the tongue detection task.

[0027] Additional aspects and advantages of the invention will become apparent from the description which follows, or may be learned by practice of the invention. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall process of a tongue localization method based on an improved YOLOv5s in one embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the original YOLOv5s network architecture in one embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the backbone network structure in the Tongue_Yolo model based on the improved YOLOv5s in one embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the coordinate attention module structure introduced in the Tongue_Yolo model in one embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the neck network structure of the Tongue_Yolo model in one embodiment of the present invention;

[0033] Figure 6This is a graph showing the loss and evaluation metrics of the Tongue_Yolo model during training, according to one embodiment of the present invention.

[0034] Figure 7 This is an example diagram showing the tongue detection performance of the Tongue_Yolo model on the validation set in one embodiment of the present invention;

[0035] Figure 8 This is a comparison of the tongue localization results of the Tongue_Yolo model and the baseline model SSD in one embodiment of the present invention, wherein (a) is the detection result of Tongue_Yolo and (b) is the detection result of SSD. Detailed Implementation

[0036] To better understand the above-mentioned objects, features, and advantages of the present invention, such as Figures 1 to 8 As shown in the accompanying drawings and specific embodiments, the present invention will be further described in detail below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0037] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0038] Figure 1 The overall flow of a tongue localization method based on an improved YOLOv5s according to an embodiment of the present invention is shown. Figure 1 As shown, this tongue localization method based on the improved YOLOv5s includes:

[0039] Step S1: Acquiring and standardizing tongue images. Specifically, under standardized lighting conditions, a professional tongue imager is used to acquire images of the patient's tongue, and the images are standardized in terms of format and resolution to construct an initial tongue image dataset;

[0040] Step S2: Polygon annotation and label file formation. Specifically, each image in the tongue image dataset is named according to "medical record number_serial number", and the contour is annotated along the edge of the tongue using the polygon plotting method; after expert review and Kappa consistency test (κ≥0.80), a JSON annotation file is generated; the JSON annotation file is parsed, a mapping table between category names and integer numbers is established, the polygon vertex coordinate sequence is extracted, the minimum bounding rectangle is determined through extreme value operation, and after normalizing the coordinates, a TXT label file with the same name as the image main file name is generated;

[0041] Step S3: Dataset partitioning and storage directory creation. Specifically, under the root directory of the dataset, create two main directories at the same level, "images" and "labels," according to the YOLO standard directory structure. Within each main directory, create three subdirectories: "train," "val," and "test," to store the images and label files for the training, validation, and test sets, respectively. For example, the directory structure of the dataset is as follows:

[0042] dataset /

[0043] ├── images /

[0044] │ ├── train /

[0045] │ ├── val /

[0046] │ └── test /

[0047] └── labels /

[0048] ├── train /

[0049] ├── val /

[0050] └── test /

[0051] Based on the medical record number, all tongue images of the same patient are merged into the same source group, and the source group is used as the smallest division unit. The images are randomly assigned to the training set, validation set and test set in a ratio of 7:2:1. The images of each subset and the corresponding TXT label files are copied to the train, val and test subdirectories under images and labels respectively to complete the standardized organization and storage of the dataset.

[0052] Step S4: Image Preprocessing and Data Augmentation. Specifically, uniform preprocessing is performed on each image in the training set, validation set, and test set: scaling to 640×640 pixels, normalizing pixel values ​​to the [0,1] range, and converting to tensor format; based on this, basic data augmentation (color adjustment and Mosaic multi-image stitching) and advanced data augmentation (geometric transformation and noise generation) are additionally performed only on the training set to expand the diversity of training samples;

[0053] Step S5: Build the Tongue_Yolo tongue localization model; wherein, the tongue localization model is a deep learning object detection network obtained by improving the original YOLOv5s network architecture, and includes the following components in sequence along the data flow direction: input end, backbone network, neck network and detection head network;

[0054] The tongue localization model, based on the original YOLOv5s network architecture, includes the following improvements:

[0055] A coordinate attention module is introduced into the backbone network;

[0056] In the upsampling path of the neck network, bilinear interpolation is used instead of nearest neighbor interpolation, and a 3×3 convolutional layer is added after each upsampling for detail compensation.

[0057] In the detection head network, the loss function is optimized, and the specific optimization method will be explained in detail in step S6;

[0058] Step S6: Iteratively train the model using the training set until it converges. Specifically, based on the preset total loss function, iteratively train the tongue localization model using the training set until the model converges; wherein, the total loss function is composed of a weighted sum of three parts: bounding box regression loss, classification loss, and confidence loss. The bounding box regression loss uses EIoU Loss, the classification loss uses Focal Loss, and the confidence loss uses binary cross-entropy loss.

[0059] Step S7: Validation set evaluation and hyperparameter tuning. Specifically, the validation set is input into the trained tongue localization model for forward propagation to obtain the predicted output of the tongue localization model on each validation sample;

[0060] Based on the predicted output and the corresponding ground truth labels, calculate the bounding box regression loss, classification loss, confidence loss, mean precision (mAP), recall, and precision on the validation set.

[0061] Based on the loss trend and accuracy fluctuation on the validation set, the hyperparameters of the tongue localization model (including but not limited to initial learning rate, batch size, anchor frame size and weight decay coefficient) are optimized and adjusted until the accuracy of the validation set no longer improves, thereby completing the final optimization of the tongue localization model.

[0062] Step S8: Evaluate the generalization performance of the model using a test set. Specifically, the test set is input into the finally trained tongue localization model. The tongue localization model sequentially extracts multi-scale features through the backbone network, performs bidirectional feature fusion through the neck network, and performs target prediction through the detection head network along the data flow direction. It outputs the bounding box position coordinates and class confidence scores corresponding to each tongue image in the test set, thereby evaluating the final generalization performance of the tongue localization model.

[0063] This embodiment proposes an improved tongue localization method based on the YOLOv5s network architecture. The YOLOv5s network architecture is lightweight, offers good real-time detection performance, and boasts high detection accuracy, making it suitable for rapid and accurate tongue localization. This embodiment is implemented through the following steps:

[0064] 1) Step 1: Use a professional tongue imager to acquire images of the patient's tongue under standardized lighting conditions; during acquisition, keep the patient's head in a neutral position and naturally extend the tongue to fully expose the tongue body; control the lens plane of the acquisition device to be parallel to the plane of the tongue body, and take pictures under uniform lighting conditions with a color temperature of 5000K~6000K to ensure accurate color reproduction of the image; save the acquired images in JPEG format with a resolution of no less than 1920×1080 pixels to construct the initial tongue image dataset;

[0065] 2) Step 2: Using an open-source image annotation tool (such as LabelMe), load each tongue image from the tongue image dataset into the annotation interface; Annotators who have received standardized training will perform polygonal point annotation along the visible edge contour of the tongue. The annotation rules are as follows: the annotation points must closely fit the edge of the tongue, and the Euclidean distance between two adjacent annotation points must not exceed a preset pixel threshold, which is no more than 10 pixels, preferably 5 pixels. Through the above constraints, it is ensured that the outlined polygon can accurately cover the entire tongue area, while strictly excluding non-tongue interference areas such as the lips, teeth, and mandible. In the annotation interface, the annotator sets the target category of the drawn polygonal region to "tongue". After annotation is completed, the annotation tool automatically generates a JSON-formatted annotation file for each image. This file contains the category label and the coordinate information of each polygon point. The JSON file is stored in the same folder as the original image and is mapped with the same main filename as the original image (for example, the original image subj_001.jpg corresponds to the annotation file subj_001.json). This naming and storage rule facilitates automatic indexing and loading of the corresponding annotation data based on the image filename.

[0066] The system reads all JSON-formatted annotation files generated from the initial annotations via a data loading interface, extracts the polygon point sets, and has two experts with backgrounds in Traditional Chinese Medicine diagnostics independently review them. Abnormal annotation points that deviate from the actual edges of the tongue are removed or corrected, and a consistency check is performed. The consistency check quantifies the consistency of the annotation quality by calculating the Cohen's Kappa coefficient of the two experts' judgments. The Cohen's Kappa coefficient is calculated using the following formula:

[0067] (1)

[0068] In the formula, P o P represents the actual agreement rate between the two experts' judgments. e The expected consistency rate of the two experts' judgments; if κ≥0.8, then the accuracy of the annotation meets the requirements;

[0069] 3) Step 3: After completing expert review and consistency verification, batch convert the qualified JSON format annotation files into YOLO format tag files, which can be achieved in the following way:

[0070] According to the JSON data format specification, the semantic category information in each annotation file is read and parsed, mapped to integer indices starting from 0, and a mapping table between category names and integer numbers is established; preferably, in this embodiment, the category "tongue" corresponds to the integer number 0;

[0071] According to the JSON data format specification, extract the width img_w and height img_h of the image from the annotation file;

[0072] Following the JSON data format specification, extract the vertex sequence of the polygon outline from the annotation file, denoted as {(x i ,y i ) | i = 1, 2, ..., n}, where n is the number of vertices of the polygon, (x i , y i Let x be the absolute pixel coordinate of the i-th vertex; iterate through all vertices, taking the minimum and maximum values ​​of their x and y coordinates respectively, and determine the minimum bounding rectangle parameters of the polygon accordingly; specifically, let x... min =min(x i ), x max =max(x i ), y min =min(y i ), y max =max(y i If the coordinates of the top-left corner of the smallest bounding rectangle are (x, y), then the coordinates of the top-left corner of the smallest bounding rectangle are (x, y). min , y min Width w = x max - x min Height h = y max - y min Therefore, the minimum bounding rectangle of the polygon is obtained;

[0073] Based on the coordinates of the top-left corner and the width and height of the minimum bounding rectangle, calculate the normalized relative coordinates of its bounding box center point and width and height in YOLO format:

[0074] (2)

[0075] In the formula, x c The x-coordinate of the center point of the bounding box;

[0076] (3)

[0077] In the formula, y c The ordinate of the center point of the bounding box;

[0078] (4)

[0079] In the formula, width is the width of the bounding box;

[0080] (5)

[0081] In the formula, height is the height of the bounding box;

[0082] For each image, automatically create a tag file with the same name as its main filename and the extension .txt, and assign the category number and the normalized relative coordinates according to... <class_id><x c > <y c > <width> <height>Write the coordinate values ​​into the file in the format specified in the file, with spaces separating the coordinate values, and each line corresponding to a label target;

[0083] Create a standard directory structure for the YOLO model; specifically, create two main directories at the same level, images and labels, under the root directory of the dataset; create subdirectories, train, val, and test, under the images directory to store the training, validation, and test images after partitioning; create subdirectories, train, val, and test, under the labels directory to store the .txt label files with the same names as the images in each subset.

[0084] 4) Step 4: Using the medical record number corresponding to each patient image in Step 1 as a unique identifier, merge all images in the tongue image dataset according to the medical record number; wherein, multiple tongue images corresponding to the same medical record number are named in the format of "medical record number_serial number" to uniquely identify images of the same patient collected at different time points.

[0085] All tongue images belonging to the same medical record number are grouped into the same source group, and the source group is named after that medical record number. Using the entire source group as the smallest unit of division, all source groups are randomly divided into three subsets in a ratio of training set:validation set:test set = 7:2:1, ensuring that each source group is included in exactly one subset, and that images of the same patient are not distributed across subsets. All images in the source group corresponding to each subset and their corresponding .txt label files are copied to the images / train, images / val, images / test and labels / train, labels / val, labels / test subdirectories, respectively, ensuring that the images and label files maintain a one-to-one correspondence within each subset. At this point, the tongue image dataset is organized in the YOLO standard format, forming non-overlapping training, validation, and test sets.

[0086] 5) Step 5: For each sample image in the training set, validation set, and test set, perform the following preprocessing operations in sequence: scale the original image to a preset input size of 640×640 pixels; normalize the image pixel values ​​from the [0,255] interval to the [0,1] interval; convert the normalized image data into tensor format.

[0087] After completing the unified preprocessing, basic data augmentation operations are performed only on the images in the training set to expand the diversity of training samples. The basic data augmentation operations include two types: color adjustment and Mosaic multi-image stitching. Specifically, the color adjustment includes performing random hue adjustment, random saturation adjustment, and random brightness adjustment on the images in the HSV color space. The hue adjustment range is [0.01, 0.02], the saturation adjustment range is [0.5, 0.8], and the brightness adjustment range is [0.3, 0.5]. The Mosaic multi-image stitching involves randomly selecting four training images, placing them in the upper left, upper right, lower left, and lower right quadrants of the synthesized image, and synchronously mapping the coordinates of the original bounding boxes in each image to the coordinate system of the synthesized image to form complete annotation information after stitching.

[0088] 6) Step 6: After completing the unified preprocessing and basic data augmentation, perform advanced data augmentation operations on the training set images to increase the robustness of the tongue localization model to image geometric deformation interference and acquisition noise.

[0089] The advanced data augmentation strategy includes two parts: geometric transformation and noise generation.

[0090] The geometric transformation employs various affine transformation operations such as rotation, translation, scaling, and shearing to increase the morphological diversity of the tongue image and improve the tongue localization model's ability to detect different tongue shapes. To uniformly describe these various transformations, an affine transformation matrix is ​​used for mathematical expression. Let (x, y) represent the two-dimensional coordinates of a pixel in the image, and (x', y') represent the two-dimensional coordinates of that pixel after the transformation. Then, the affine transformation relationship is expressed as:

[0091] (1)

[0092] In the formula, M is the affine transformation matrix, which is used to uniformly describe geometric transformations such as rotation, translation, scaling, and shearing;

[0093] For the rotational transformation R in the geometric transformation, the affine transformation matrix M is expressed as:

[0094] (2)

[0095] In the formula, θ represents the rotation angle (in radians, a positive value indicates counterclockwise rotation, and a negative value indicates clockwise rotation), and rx and ry represent the translation compensation amount; by adjusting the values ​​of θ, rx, and ry, rotation transformations at different angles and positions can be achieved.

[0096] For the translation transformation T in the geometric transformation, the affine transformation matrix M is expressed as:

[0097] (3)

[0098] In the formula, tx represents the horizontal translation amount, and ty represents the vertical translation amount; by randomly setting the values ​​of tx and ty, random translation enhancement in different directions and at different distances can be achieved.

[0099] For the scaling transformation S in the geometric transformation, the affine transformation matrix M is expressed as:

[0100] (4)

[0101] In the formula, sx and sy represent the scaling factors in the horizontal and vertical directions, respectively; by adjusting the values ​​of sx and sy, the image can be scaled proportionally or non-proportionally in the horizontal and vertical directions.

[0102] For the shearing transformation H in the geometric transformation, the affine transformation matrix M is expressed as:

[0103] (5)

[0104] In the formula, hx and hy represent the shearing coefficients in the horizontal and vertical directions; by adjusting the values ​​of hx and hy, random shearing and tilting of the image in the horizontal and vertical directions can be achieved.

[0105] In summary, if the image is subjected to a combination of four transformations—rotation, translation, scaling, and shearing—in sequence, then the combined transformation matrix M is represented as:

[0106] (6)

[0107] In the formula, the multiplication sign represents the progressive relationship in which each transformation is performed from right to left;

[0108] The noise generation in the advanced data augmentation is performed after the geometric transformation is completed; specifically, random Gaussian noise is added to the geometrically transformed image to simulate the image noise introduced by the sensor during the actual acquisition process, thereby improving the robustness of the tongue localization model to noise interference.

[0109] Let I(x,y) be the pixel value at position (x,y) in the image. Then the pixel value I'(x,y) after adding Gaussian noise is expressed as:

[0110] (7)

[0111] In the formula, The variance is Gaussian distributed random variable, has a value range of [0.01, 0.1]; to ensure that the pixel values after adding noise fall within the interval [0, 1], the pixel values are truncated:

[0112] (8)

[0113] The variance of the Gaussian noise adopts a dynamic calculation strategy to generate noise interferences of different intensities, and the calculation method is shown in the following formula:

[0114] (9)

[0115] The above formula means that the variance is randomly sampled according to a uniform distribution between the preset lower limit σ 2 min and upper limit σ 2 max , so that the noise intensity superimposed on each training image in each iteration is different. Preferably, σ 2 min =0.0001, σ 2 max =0.01;

[0116] Considering that frequent application of noise augmentation will cause problems such as reduced positioning accuracy and confused feature learning of the tongue positioning model, this embodiment adopts a random probability control strategy in each training iteration, and determines whether to perform noise augmentation on the current batch of images according to the preset enable probability p. The value range of p is [0, 1], preferably, p=0.5; at the beginning of each training iteration, a random number r with a value range of [0, 1] is generated. If r < p, this operation is skipped directly; otherwise, the noise augmentation operation is executed;

[0117] 7) Step 7: As shown Figure 2 , the original YOLOv5s network architecture sequentially includes an input end, a backbone network, a neck network and a detection head network along the data flow direction;

[0118] This embodiment is improved based on the original YOLOv5s network architecture, and a tongue positioning model is built, which is recorded as the Tongue_Yolo model; the Tongue_Yolo model sequentially includes an input end, a backbone network, a neck network and a detection head network along the data flow direction;

[0119] As shown Figure 3 , the backbone network of the Tongue_Yolo model is used to perform multi-level downsampling feature extraction on the standardized input data, generate multi-scale feature maps including at least a first-scale feature map and a second-scale feature map, and output the multi-scale feature maps to the neck network of the Tongue_Yolo model;

[0120] Specifically, the backbone network of the Tongue_Yolo model, along the downsampling direction, sequentially includes: a first Conv module, a second Conv module, a first C3 module, a third Conv module, a second C3 module, a fourth Conv module, a third C3 module, a fifth Conv module, a fourth C3 module, the coordinate attention module, and the pooling module; wherein, the Conv module is composed of convolutional layers, batch normalization layers, and SiLU activation functions in sequence; the C3 module is composed of three standard convolutional layers and several CSP bottleneck layers; the pooling module adopts the Fast Spatial Pyramid Pooling (SPPF) module;

[0121] The coordinate attention module is located between the fourth C3 module and the SPPF module. It is used to perform spatial position attention weighting on the feature map output by the fourth C3 module in both horizontal and vertical directions to enhance the feature salience of the tongue region and reduce the interference of the background region.

[0122] like Figure 4 As shown, the working process of the coordinate attention module is as follows:

[0123] The input feature map of the coordinate attention module is denoted as X, and Where C, H, and W represent the total number of channels, height dimension, and width dimension of the feature map, respectively; c, h, and w are defined as index variables in the corresponding dimensions, satisfying 1≤c≤C, 1≤h≤H, and 1≤w≤W; the feature response value of the feature map at channel index c and spatial coordinates (h,w) is denoted as x. c (h,w);

[0124] Global average pooling is performed on each pixel row of the feature map along the width direction to aggregate features in the width direction while preserving height dimension information. Specifically, for any channel index c and any height index h, the channel index c and the height index h are fixed, and the arithmetic mean of all feature response values ​​of the feature map in the c-th channel and h-th row along the width direction is calculated to obtain the horizontal pooling result z. c h (h), its calculation formula is:

[0125] (1)

[0126] All z corresponding to all values ​​of c and h c h The (h) values ​​are arranged according to their spatial location, forming the horizontal feature tensor Z. h ,Right now:

[0127] (2)

[0128] The horizontal feature tensor Z h The shape is C×H×1, where z c h (h) represents Z h The feature response value at the c-th channel with height index h;

[0129] Global average pooling is performed on each pixel column of the feature map along the height direction to aggregate features in the height direction while preserving width dimension information. Specifically, for any channel index c and any width index w, the channel index c and the width index w are fixed, and the arithmetic mean of all feature response values ​​of the feature map in the c-th channel and w-th column along the height direction is calculated to obtain the vertical pooling result z. c w (w), its calculation formula is:

[0130] (3)

[0131] All z corresponding to all values ​​of c and w c w The (w) values ​​are arranged according to their spatial location, forming the vertical feature tensor Z. v ,Right now:

[0132] (4)

[0133] The vertical direction feature tensor Z v The shape is C×1×W, where z c w (w) represents Z v The feature response value at the c-th channel with width index w;

[0134] The horizontal feature tensor Z h and the vertical direction feature tensor Z v The feature map is obtained by stitching along the spatial dimension; specifically, the Z-shaped feature map with shape C×H×1 is stitched together. h With Z of shape C×1×W v The feature map is obtained by concatenating along the spatial dimension to obtain a concatenated feature map with shape C×(H+W)×1; the concatenated feature map is then compressed using a 1×1 convolution kernel, reducing the number of channels from C to C / r, where r is the channel reduction ratio, preferably 16; a nonlinear mapping is then performed using a nonlinear activation function to generate an intermediate feature tensor F; the expression corresponding to the intermediate feature tensor F is:

[0135] (5)

[0136] In the formula, δ() is a nonlinear activation function; [,] represents the splicing operation along the spatial dimension; Conv 1×1 () indicates a convolutional layer transformation with a kernel size of 1×1;

[0137] The intermediate feature tensor F has a shape of C / r×(H+W)×1, which simultaneously encodes the spatial position information of the input feature map in the horizontal and vertical directions;

[0138] The intermediate feature tensor F is split into horizontal sub-tensors F along the spatial splicing direction. h and the vertical direction subtensor F v Specifically, the intermediate feature tensor F has a shape of C / r×(H+W)×1. The feature slices at the first H positions are merged along its height-width dimension and used as the horizontal sub-tensor F. h Its shape is C / r×H×1; the feature slices at the last W positions are used as the vertical direction sub-tensor F. v Its shape is C / r×1×W; the horizontal subtensor F h The vertical direction subtensor F encodes the row direction context information after nonlinear transformation. v The column direction context information after nonlinear transformation is encoded;

[0139] The horizontal direction subtensor F h The input is fed into the first 1×1 convolutional layer, restoring the number of channels from C / r back to C. Then, it is mapped using a Sigmoid activation function to generate the horizontal attention weight tensor g. h Its shape is C×H×1, and the transformation expression is:

[0140] (6)

[0141] In the formula, σ() represents the Sigmoid activation function; Conv (h) 1×1 () indicates a 1×1 convolutional layer acting on the horizontal subtensors;

[0142] The vertical direction sub-tensor F v The input is fed into the second 1×1 convolutional layer, restoring the number of channels from C / r to C. Then, it is mapped using a Sigmoid activation function to generate the vertical attention weight tensor g. v Its shape is C×1×W, and the transformation expression is:

[0143] (7)

[0144] In the formula, Conv (v) 1×1 () indicates a 1×1 convolutional layer that acts on the vertical subtensor;

[0145] The input feature map X is compared with the horizontal attention weight tensor g. h and the vertical attention weight tensor g v Element-wise multiplication yields the attention-weighted output feature map Y, whose expression is:

[0146] (8)

[0147] The shape of the output feature map Y is the same as that of the input feature map X, which is C×H×W;

[0148] The coordinate attention module retains the precise spatial position information of the input feature map in both the horizontal and vertical directions, enabling the Tongue_Yolo model to simultaneously focus on the coordinate position of the tongue target in both the row and column directions, thereby improving the localization accuracy of the tongue bounding box.

[0149] The SPPF module achieves a multi-scale receptive field equivalent to pooling kernels of different scales by using multiple small-sized pooling kernels in series, thereby increasing the receptive field and separating the most representative contextual features while reducing the computational load.

[0150] The neck network of the Tongue_Yolo model is set between its backbone network and the detection head network to fuse the multi-scale feature maps generated by the backbone network; the neck network adopts a bidirectional fusion structure that combines a feature pyramid structure and a path aggregation network.

[0151] like Figure 5 As shown, in the upsampling fusion path of the feature pyramid structure, the neck network uses bilinear interpolation to replace the nearest neighbor interpolation method of the original YOLOv5s network architecture for upsampling. A 3×3 convolutional layer is added after each upsampling operation to compensate for the loss of high-frequency details caused by bilinear interpolation smoothing. Specifically, the kernel size of the 3×3 convolutional layer is 3×3, the stride is set to 1, and the padding method is set to 'same' to ensure that the height and width of the feature map remain unchanged after upsampling. Simultaneously, the number of input channels and output channels of the 3×3 convolutional layer are set to the same value to ensure that the number of channels in the feature map remains unchanged, avoiding the impact of dimensionality changes on the splicing or fusion operations of subsequent path aggregation networks. The 3×3 convolutional layer adaptively calibrates the interpolated feature map, compensating for the deficiency of bilinear interpolation's fixed linear weights in failing to recover high-frequency edge information, thereby improving the localization fit of the tongue edge.

[0152] In the downsampling fusion path of the path aggregation network, the neck network uses a Conv module with a stride of 2 for downsampling to gradually fuse feature information at different scales; wherein, the Conv module has the same structure as the Conv module in the backbone network of the Tongue_Yolo model, both consisting of convolutional layers, batch normalization layers and SiLU activation functions in sequence;

[0153] The detection head network of the Tongue_Yolo model has the same structure as the detection head network of the original YOLOv5s network architecture, and is used to perform target detection on the multi-scale feature maps output by the neck network. Specifically, the detection head network performs convolution operation on the feature maps of each scale output by the neck network through convolutional layers with a kernel size of 1×1 and a stride of 1. The number of output channels of the 1×1 convolutional layer is set to (C+5)×A, where C is the number of target categories to be detected and A is the number of anchor boxes predicted for each grid cell on the feature map. Preferably, C=1 and A=3. Correspondingly, for each grid cell on the feature map at each scale, each of the A anchor boxes output by the model outputs a (C+5) dimensional vector. The vector consists of 4 bounding box coordinate parameters, 1 target confidence, and C category confidences arranged in order.

[0154] 8) Step 8: Construct the loss function of the Tongue_Yolo model; the total loss function of the Tongue_Yolo model consists of a weighted sum of three parts: bounding box regression loss, target confidence loss, and classification loss;

[0155] This embodiment replaces the bounding box regression loss and classification loss of the original YOLOv5s network architecture. The specific replacement measures are as follows:

[0156] The original YOLOv5s network architecture uses CIoU Loss as the bounding box regression loss function, which is defined as:

[0157] (1)

[0158] In the formula, IoU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box; b p b is the coordinate of the center point of the prediction box; t ρ(b) represents the coordinates of the center point of the true bounding box. p , b t ) represents the Euclidean distance between the center of the predicted bounding box and the center of the ground truth bounding box; c represents the diagonal length of the minimum bounding box; α represents the weighting coefficient; v represents the aspect ratio consistency measure.

[0159] However, the aspect ratio penalty term v in the CIoU Loss only reflects the difference in aspect ratio and is not directly related to the actual width and height values ​​of the predicted box and the ground truth box. This means that when the aspect ratios of the predicted box and the ground truth box are the same but the actual width and height are different, the value of v may be the same, making the direction of gradient optimization unclear and limiting the regression accuracy.

[0160] To address the shortcomings of the CIoU Loss, this embodiment replaces the CIoU Loss with the EIoU Loss as the bounding box regression loss for the Tongue_Yolo model. The EIoU Loss, based on the IoU loss and center point distance loss, introduces a side length difference penalty term, the expression of which is:

[0161] (2)

[0162] In the formula, This represents the squared difference in width between the predicted bounding box and the true bounding box; This represents the squared difference in height between the predicted bounding box and the true bounding box. and Let represent the width and height of the smallest bounding rectangle, respectively.

[0163] The and The expressions are as follows:

[0164] (3)

[0165] (4)

[0166] In the formula, Indicates the x-coordinate of the center point of the prediction box; Represents the x-coordinate of the center point of the true bounding box; Indicates the width of the prediction box; Indicates the actual frame width; This indicates the ordinate of the center point of the prediction box; This represents the y-coordinate of the center point of the true bounding box; Indicates the height of the prediction box; Indicates the actual height of the bounding box;

[0167] The original YOLOv5s network architecture uses binary cross-entropy (BCE) as the classification loss function, which is defined as:

[0168] (5)

[0169] In the formula, x represents the original logical value output by the classification branch in the detection head network for the current anchor box; σ() represents the Sigmoid activation function; y represents whether the anchor box contains the true label of the tongue target (y=1 represents a positive sample, y=0 represents a negative sample).

[0170] In the tongue detection task, the BCE loss is calculated only for positive sample anchor boxes. However, there is a significant class imbalance within the positive samples: the main body region of the tongue has a large number of samples and is easy to classify, while the edge region of the tongue has a small number of samples and is difficult to classify. Its loss contribution is masked by a large number of easily classified samples in the total gradient, resulting in insufficient representation ability of the original YOLOv5s network architecture for the edge region of the tongue.

[0171] Therefore, this embodiment uses Focal Loss to replace the BCE loss as the classification loss of the Tongue_Yolo model. Focal Loss introduces class weight coefficients and modulation factors on top of the BCE loss, dynamically reducing the loss contribution of easily classified samples and relatively increasing the loss weight of difficult-to-classify samples, thereby overcoming the technical defect of insufficient attention to the tongue edge region caused by sample imbalance in the BCE loss. The Focal Loss is expressed as:

[0172] (6)

[0173] In the formula, t is the category index; These are weighting coefficients used to balance the overall contribution ratio of positive and negative samples; This is a modulation factor used to adjust the weight decay rate of easily classified samples; It is the predicted probability of the Tongue_Yolo model for the true class;

[0174] For a binary classification problem, let y∈{0,1} represent the true label of the sample, p∈[0,1] represent the probability that the Tongue_Yolo model predicts the sample as a positive sample, and x be the original output value of the Tongue_Yolo model's detection head network predicting that the current anchor box belongs to the positive sample category. Then p is calculated using the Sigmoid function, and its expression is:

[0175] (7)

[0176] Furthermore, to unify the representation of positive and negative samples, the predicted probability of the Tongue_Yolo model for the true class is defined. for:

[0177] (8)

[0178] The The class weight coefficients are obtained by conditionally assigning values ​​to the base weights α based on the true class y of the sample. The expression is as follows:

[0179] (9)

[0180] In the formula, The basic weighting coefficient has a value range of (0,1).

[0181] Preferably, The value is 2.0. The value is 0.25;

[0182] The target confidence loss of the original YOLOv5s network architecture is the BCE loss by default; this embodiment also uses the BCE loss as the target confidence loss of the Tongue_Yolo model.

[0183] In this embodiment, the total loss function is composed of the weighted sum of the bounding box regression loss, the classification loss, and the target confidence loss, and its expression is:

[0184] (10)

[0185] In the formula, L total Let be the total loss function; λ1, λ2, and λ3 are the weight coefficients of the bounding box regression loss, classification loss, and target confidence loss, respectively.

[0186] In this embodiment, the weight coefficients of the total loss function are set in the same way as the original YOLOv5s network architecture, preferably λ1=0.05, λ2=0.5, and λ3=1.0;

[0187] 9) Step 9: After completing the construction of the Tongue_Yolo model, train the model. One iteration of the training process includes the following steps:

[0188] First, the tongue images in the training set are used as input data and fed into the backbone network of the model. The backbone network performs multi-level downsampling feature extraction on the tongue images to generate multi-scale feature maps. The coordinate attention module applies attention weights to the multi-scale feature maps in both the horizontal and vertical directions to enhance the feature saliency of the tongue region. The SPPF module aggregates the multi-scale receptive fields through multiple small-sized pooling kernels in series to increase the receptive field and separate the most representative contextual features.

[0189] Furthermore, the multi-scale feature maps output by the backbone network are input into the neck network of the model. The neck network uses a bidirectional fusion structure that combines a feature pyramid structure and a path aggregation network to fuse the multi-scale feature maps.

[0190] In the upsampling fusion path of the feature pyramid structure, the neck network uses bilinear interpolation for upsampling and adds a 3×3 convolutional layer after each upsampling operation to compensate for the detail information in the multi-scale feature map and reduce the loss of detail caused by bilinear interpolation smoothing.

[0191] In the downsampling fusion path of the path aggregation network, the neck network uses a Conv module with a stride of 2 for downsampling to gradually fuse feature information at different scales;

[0192] Furthermore, the fused multi-scale feature map output by the neck network is input into the detection head network of the model. The detection head network performs convolution operation on the feature map of each scale through a convolutional layer with a kernel size of 1×1 and a stride of 1, mapping the feature map of each scale into a prediction tensor, and outputting the bounding box position coordinates, target confidence and confidence of each category corresponding to each anchor box.

[0193] After obtaining the anchor box prediction results output by the detection head network, the total loss value of the model is calculated based on the difference between the prediction results and the corresponding ground truth labels in the training set; the total loss value is obtained by weighted summation of the bounding box regression loss, classification loss and target confidence loss.

[0194] After calculating the total loss value, the backpropagation algorithm is used to calculate the partial derivative of the total loss value with respect to the trainable parameters of each network layer in the reverse order of the data flow (i.e., starting from the detection head network of the model, passing through the neck network to the backbone network in sequence) to obtain the gradient value of each parameter.

[0195] Furthermore, a Stochastic Gradient Descent (SGD) optimizer is employed to update all trainable parameters of the model based on the calculated gradient values. The preset hyperparameter settings for the SGD optimizer are as follows: the initial learning rate is preferably set to 0.01, the momentum coefficient is preferably set to 0.937, and the weight decay coefficient is preferably set to 0.0005. The trainable parameters are iteratively updated by the SGD optimizer according to the preset learning rate, momentum, and weight decay coefficient to minimize the total loss value, thus completing this iteration.

[0196] Repeat the above steps until the preset number of training rounds is reached, preferably 300 rounds. After training, save the model weight with the highest mAP@0.5:0.95 value on the validation set as the final optimized Tongue_Yolo model. The mAP@0.5:0.95 value is calculated as follows: set the intersection-union ratio (IU) threshold from 0.50 to 0.95 with a step size of 0.05, resulting in 10 different IU thresholds. Calculate the mean average precision (mAP) at each threshold. Finally, take the arithmetic mean of the 10 mAP values ​​as the mAP@0.5:0.95 value.

[0197] 10) Step 10: Input the test set into the trained Tongue_Yolo model, and after forward propagation, obtain the prediction results of each test sample output by the model's detection head network (including the position coordinates of the tongue body bounding box and the corresponding class confidence); compare the prediction results with the true annotations of the test set, and calculate the average precision (mAP@0.5:0.95) of the model on the test set to evaluate the generalization performance of the model.

[0198] To verify the beneficial effects of the tongue localization method described in this invention, a comparative experiment was designed in this embodiment to compare the performance of the Tongue_Yolo model described in this embodiment with that of the Single Shot MultiBox Detector (SSD). The reason for choosing the SSD model as the comparison object is that the SSD model and the Tongue_Yolo model described in this embodiment both belong to the single-stage object detection network and both use multi-scale feature maps for object prediction, which makes them comparable and can objectively reflect the improvement effect of this embodiment compared with similar baseline models.

[0199] The comparative experiment was built on the Windows 10 operating system and the PyTorch deep learning software platform. The experiment used a self-built human tongue image dataset, which consists of three parts: a training set, a validation set, and a test set, totaling 5,000 tongue images. The tongue images were annotated and quality-confirmed by physicians in the field of traditional Chinese medicine tongue diagnosis. The annotation content includes the polygonal contour vertex sequence and category label of the tongue region. During training, the polygonal contour vertex sequence was converted into a rectangular bounding box. Specifically, the minimum bounding rectangle of the coordinates of each vertex of the polygon was taken as the true bounding box annotation of the sample and participated in the calculation of the bounding box regression loss.

[0200] The Tongue_Yolo model and the SSD model were trained separately, and both models used the same training strategy to ensure the fairness of the comparison: both used the SGD optimizer, with an initial learning rate of 0.01, a momentum coefficient of 0.937, a weight decay coefficient of 0.0005, a batch size of 4, and the input image size was uniformly scaled to 640×640 pixels. The number of training rounds was set to 300 rounds for both models.

[0201] After training, the two models were tested on the same test set, and a multi-index comprehensive evaluation system was used for performance comparison. The mean precision (mAP@0.5 and mAP@0.5:0.95) was used as the primary evaluation index to comprehensively measure the overall detection capability and localization accuracy of the two models in the tongue detection task. Precision, recall, and F1-score were introduced as auxiliary evaluation indexes to measure the classification accuracy and recall capability of the two models for positive samples. Furthermore, the average intersection-union ratio (IoU) was used as a localization accuracy index to evaluate the degree of overlap between the predicted bounding box and the ground truth bounding box in the tongue region.

[0202] Figure 6 This is a graph showing the loss versus evaluation metrics of the Tongue_Yolo model during training. From... Figure 6 As can be seen, in terms of the bounding box regression loss metric, the minimum loss on the validation set (val / box_loss) is less than the minimum loss on the training set (train / box_loss), indicating that the model has good generalization ability and does not exhibit significant overfitting. In terms of the target confidence loss metric, both the confidence loss on the training set (train / obj_loss) and the confidence loss on the validation set (val / obj_loss) remain at low levels, indicating that the model has high confidence in distinguishing the tongue region from the oral cavity background, and the confidence scores of the predicted boxes are accurate and reliable. In terms of mean accuracy, mAP@0.5 approaches 1.0, and mAP@0.5:0.95 approaches 0.9, indicating that the model not only performs well under a relaxed threshold (IoU=0.5), but also maintains high localization accuracy under strict thresholds (IoU=0.5, 0.55, …, 0.95). The predicted bounding boxes closely match the actual tongue in both position and aspect ratio. This is attributed to the model's neck network sampling method, which uses bilinear interpolation instead of nearest neighbor interpolation and adds a 3×3 convolutional layer after upsampling for detail compensation, effectively enhancing the expressive power of tongue edge and texture features. Under standard evaluation metrics, the precision and recall of the Tongue_Yolo model both converge to 1.0, indicating that the coordinate attention module introduced in the model enables the model to accurately focus on the tongue region, significantly reducing false negatives and false positives.

[0203] Figure 7 This is an example image showing the tongue detection performance of the Tongue_Yolo model on the validation set. Figure 7 As can be seen, the verification images contain tongue images of different sizes and shapes. The bounding box of each tongue image fits closely with the actual outline of the tongue and covers it completely without any offset, excessive size or small size. This shows that the coordinate attention module and the bilinear interpolation upsampling method used in the model can effectively avoid the jagged effect and suppress the response of non-tongue regions in the spatial dimension, so that the prediction box can accurately locate the tip, root and side edges of the tongue, showing good generalization ability and robustness.

[0204] Figure 8 The image shows a comparison of the tongue localization results between the Tongue_Yolo model and the SSD model, where (a) is the detection result of the Tongue_Yolo model and (b) is the detection result of the SSD model. Figure 8 It can be seen that the Tongue_Yolo model can accurately locate the tongue, achieving the extraction of the complete tongue in complex backgrounds. It exhibits lightweight design, enhanced feature extraction and fusion, and good robustness and generalization ability. Conversely, the SSD model's tongue localization result includes the entire oral cavity, failing to precisely isolate the tongue and resulting in significant localization loss. The main reason is that although the SSD model uses multi-scale feature maps, its feature pyramid structure is relatively simple, and the information fusion between different layers is insufficient. This makes it difficult for the model to simultaneously utilize deep semantics (determining whether it is a tongue) and shallow details (where the tongue edges are), leading to inaccurate tongue localization boundaries in complex oral cavity backgrounds.

[0205] Table 1 lists the comparison results between the Tongue_Yolo model and the SSD model on multiple evaluation metrics. As shown in Table 1, the Tongue_Yolo model outperforms the SSD model in all six evaluation metrics. Specifically, the Tongue_Yolo model achieved higher values ​​in both mAP@0.5 and mAP@0.5:0.95, improving upon the SSD model by 37.6 and 41.7 percentage points respectively. This indicates that the Tongue_Yolo model has higher detection accuracy and stronger localization ability in the tongue detection task, especially with a more significant performance advantage under high cross-validation threshold requirements. In terms of Precision, Recall, and F1-score, the Tongue_Yolo model improved upon the SSD model by 70.7, 21.1, and 81.9 percentage points respectively. This shows that the Tongue_Yolo model significantly outperforms the SSD model in classification accuracy, positive sample recall, and overall performance, effectively reducing the false negative rate while maintaining high detection accuracy. In terms of Average IoU, the Tongue_Yolo model improved upon the SSD model by 13.1 percentage points, indicating that the Tongue_Yolo model has a higher overlap between the predicted and ground truth bounding boxes and more accurate localization.

[0206] Table 1. Performance comparison results of Tongue_Yolo model and SSD model on tongue detection task

[0207] Model Name mAP@0.5 mAP@0.5:0.95 Precision Recall F1 score Average IoU Tongue_Yolo 0.982 0.809 0.93 1 0.964 0.893 SSD 0.606 0.392 0.223 0.789 0.145 0.762

[0208] This invention addresses the challenges of tongue detection tasks, such as variable tongue morphology, blurred edges, and imbalanced positive and negative samples. Based on the YOLOv5s network architecture, it proposes an improved tongue localization method. Specifically, this invention enhances the extraction capability of key tongue region features and suppresses background interference by introducing a coordinate attention mechanism into the backbone network. It also improves the jagged edges of the tongue and restores high-frequency details lost during smoothing by employing bilinear interpolation and adding a 3×3 convolutional layer in the upsampling path of the neck network for detail compensation. Furthermore, it effectively improves the localization accuracy of bounding boxes and alleviates the imbalanced positive and negative sample problem by replacing CIoU Loss with EIoU Loss for bounding box regression and replacing binary cross-entropy loss with Focal Loss for classification. Experimental results show that the Tongue_Yolo model described in this invention outperforms the SSD model on a self-built tongue image dataset, verifying the effectiveness of the above improvements.

[0209] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / height> < / width>

Claims

1. A tongue positioning method based on improved YOLOv5, characterized in that, include: Step S1: Obtain the patient's tongue image dataset, and perform polygon annotation of the tongue body region on each tongue image in the tongue image dataset to obtain the annotation information of the tongue body region; Based on the annotation information, extract the polygon vertex coordinate sequence of the tongue region in each tongue image, and generate a YOLO format label file for each tongue image based on the polygon vertex coordinate sequence. Step S2: Group the tongue image dataset by source based on the patient's medical record number. All tongue images corresponding to the same medical record number belong to the same source group. Using the source group as the smallest division unit, randomly allocate each source group to the training set, validation set, and test set according to a preset ratio. The source groups allocated to each subset constitute the training set, validation set, and test set, respectively. Step S3: Construct a standardized dataset storage directory; the storage directory includes a parallel image storage main directory and a label storage main directory, and each of the image storage main directory and the label storage main directory contains three subdirectories that correspond one-to-one with the training set, validation set and test set; Store the image files and their corresponding label files in the training set, validation set, and test set into their respective subdirectories; Step S4: Perform preprocessing and data augmentation on the sample images in the training set, and perform preprocessing but no data augmentation on the sample images in the validation set and test set to generate corresponding standardized input data respectively; Step S5: Construct a tongue localization model; the tongue localization model is an improvement based on the YOLOv5s model, including an input terminal, a backbone network, a neck network, and a detection head network connected in sequence; The backbone network includes a downsampling module, a feature extraction module, an attention module, and a pooling module arranged sequentially along the downsampling direction; the downsampling module is used to perform spatial resolution downsampling and channel expansion on the input feature map; the feature extraction module is used to perform layer-by-layer convolution operations on the downsampled feature map to extract the edge contour features of the tongue region; The attention module is used to perform spatial location attention weighting on the input feature map to enhance the feature activation value of the target tongue region and suppress the feature response of the background interference region; the pooling module is used to perform multi-scale spatial pooling on the attention-weighted feature map to increase the network receptive field and capture the contextual semantic information of the tongue at different scales. The neck network adopts a bidirectional fusion structure that combines a feature pyramid structure with a path aggregation network, which is used to perform feature fusion on the multi-scale feature maps generated by the backbone network. The detection head network is used to perform target detection on the multi-scale feature map fused by the neck network, and outputs the position coordinates of the tongue body bounding box and its corresponding category confidence. Step S6: Based on the preset total loss function, iteratively train the tongue localization model using the training set until the model converges; wherein, the total loss function includes bounding box regression loss, classification loss and confidence loss; Step S7: During the training process in step S6, after each preset round of iterative training, the validation set is input into the current model for forward propagation, the loss value on the validation set is calculated, and the training strategy is adjusted according to the trend of the loss value. The model weight with the smallest loss on the validation set is selected as the training result. Step S8: Input the test set into the trained tongue localization model for forward propagation, obtain the predicted tongue bounding box coordinates and confidence scores, and calculate the evaluation index based on the prediction results on the test set and the ground truth bounding boxes to verify the generalization performance of the tongue localization model.

2. The tongue localization method based on the improved YOLOv5s according to claim 1, characterized in that, The attention module is a coordinate attention module, which is used for: The input feature map is subjected to global average pooling in the horizontal direction and global average pooling in the vertical direction to generate a horizontal feature tensor and a vertical feature tensor. The horizontal and vertical feature tensors are concatenated along the spatial dimension and then subjected to channel compression and nonlinear activation through a 1×1 convolutional layer to generate an intermediate feature tensor. The intermediate feature tensor is split into horizontal and vertical sub-tensors along the spatial dimension, and then channel recovery and sigmoid activation are performed on them respectively through a 1×1 convolutional layer to generate horizontal attention weight tensors and vertical attention weight tensors. The input feature map is multiplied element-wise by the horizontal attention weight tensor and the vertical attention weight tensor to output the weighted feature map.

3. The tongue localization method based on the improved YOLOv5s according to claim 1, characterized in that, The total loss function is composed of the bounding box regression loss, classification loss, and confidence loss, each multiplied by its respective preset weight coefficient and then summed. The bounding box regression loss uses EIoU Loss, the classification loss uses Focal Loss, and the confidence loss uses binary cross-entropy loss.

4. The tongue localization method based on the improved YOLOv5s according to any one of claims 1 to 3, characterized in that, The preprocessing is a standardized process to ensure that all input images are in the same format, including resizing, pixel normalization, and label synchronization. The data augmentation is an operation applied only to the training set to increase the diversity of the samples, including geometric transformation operations and noise generation operations.

Images