Tongue picture image attitude correction and integrity detection method and system based on key point detection

Abstract

The invention relates to the technical field of image detection, and particularly discloses a tongue image attitude correction and integrity detection method and system based on key point detection, and the method comprises the basic steps: firstly obtaining and processing a tongue image containing various kinds of information, constructing a neural network model for evaluation training, carrying out the transfer learning through a pre-training model, and optimizing the speed; the posture, integrity and accuracy are judged according to the positions of the tongue tip and the tongue root and the anatomical relation, a user is reminded of adjustment for detection abnormity, and if multiple failures occur, suggestions are automatically adjusted, and shooting is assisted. In the aspect of data processing, polymorphic tongue images are collected, enhanced and normalized, and the generalization ability of the model is enhanced. The network architecture is fused with ResNet and Transform, and the key points are accurately positioned. Performance is improved, real-time performance is guaranteed, accurate judgment is carried out according to the tongue tip position relation through scientific posture and integrity detection logic, and image quality is guaranteed. Clear prompt and auxiliary adjustment can be provided, tongue picture image acquisition is more efficient and accurate, and development of an intelligent traditional Chinese medicine tongue diagnosis technology is powerfully promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a method and system for detecting posture correction and integrity of tongue images based on key point detection. Background Art

[0002] In the field of TCM diagnosis, tongue diagnosis is one of the four major diagnostic methods. By observing the color, shape and coating of the tongue, it provides a key basis for judging the patient's health status. With the vigorous development of smart devices and telemedicine technology, it has become a trend for patients to take pictures of their tongues themselves. However, this process currently faces many difficulties.

[0003] On the one hand, the shooting skills of different patients vary significantly, resulting in uneven quality of the collected tongue images. Some images have serious problems such as tilted, skewed or incomplete tongues. For example, the patient may not understand the shooting requirements well enough or is not proficient in operation, which may cause the head angle to deviate during shooting, causing the tongue to appear obviously tilted in the image, making it difficult to accurately analyze the tongue features; or the shooting range may be improperly controlled, resulting in the missing of the tongue tip, tongue root or tongue side, making the data incomplete during image analysis.

[0004] On the other hand, existing technologies have obvious shortcomings in solving these problems. Traditional methods mainly rely on simple image adjustment methods, such as rotation and cropping, but these methods often lack precision and are difficult to effectively correct tongue posture or ensure image integrity. Although relying on manual judgment by doctors has a certain degree of accuracy, it is extremely inefficient and difficult to achieve in remote diagnosis and treatment or self-service health monitoring scenarios due to the inability to obtain the judgment of professional doctors in real time.

[0005] In view of this, there is an urgent need for a technology that can automatically detect tongue posture and integrity to improve the quality of tongue image acquisition and ensure the accuracy and reliability of subsequent diagnostic analysis. Summary of the invention

[0006] The purpose of the present invention is to provide a tongue image posture correction and integrity detection method based on key point detection to solve the technical problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions: The tongue image posture correction and integrity detection method based on key point detection includes: Obtaining a tongue image, wherein the tongue image includes image information of different tongue postures, lighting conditions, and tongue shapes; Perform data enhancement and normalization processing on the collected image information; Build a neural network model that uses ResNet to extract features and combines it with the self-attention mechanism Transformer; Performing evaluation training on the neural network model; Based on the model trained by the evaluation, a pre-trained model trained on a large-scale human posture dataset is used for transfer learning, and the inference speed is optimized; The angle between the center line of the tongue and the vertical direction of the image is calculated based on the position of the tongue tip and tongue root. If it exceeds 10 degrees, the tongue posture is considered to be incorrect and the user is prompted to adjust it. The tongue tip, tongue root and left and right edge points are detected to determine whether the tongue in the image is complete. If the key point is not detected or is too close to the edge of the image, the user is prompted to retake the photo; The accuracy of the test is determined based on the anatomical relationship between the tongue tip, tongue root, and left and right edges. If it is unreasonable, it indicates that there is a test error; If the user fails to take pictures multiple times, the system will automatically adjust the shooting suggestions and display the real-time detection results of the tongue key points to assist the user in making adjustments.

[0008] Furthermore, the step of obtaining the tongue image includes: By collecting image information of different tongue postures, lighting conditions, and tongue shapes, a mixed data set containing complete tongues and tongues with missing tongue tips is formed; Accurately mark the tongue tip, tongue root, two points on each of the left and right tongue edges, and the midline key points in the image. If the tongue tip is missing, mark "tongue tip missing" and "imaginary tongue tip" and add a "missing" label; Set "visibility annotation" for invisible key points; Use LabelMe and VGG Image Annotator tools to save annotation data in JSON or XML format, making sure that the annotation files correspond to the image file names.

[0009] Furthermore, the step of performing data enhancement and normalization processing on the collected image information includes: Divide the dataset into 80% training set and 20% test set. K-fold cross validation is used, where K=5, to improve the generalization ability of the model.

[0010] Furthermore, the step of constructing a neural network model that extracts features using ResNet and combines the self-attention mechanism Transformer includes: To solve the problem of key point detection at the bottom boundary of an image, image padding, mirror filling, cropping, flipping, rotation, multi-scale training and loss function are used for weighted calculation. Calculate the error between the key point coordinates and the true annotation value based on the mean square error and heat map loss function; The input tongue image is subjected to the convolution layer to extract features, and the deconvolution layer or upsampling layer to generate a key point heat map or coordinate prediction value; The network weights are updated via back-propagation and the Adam or SGD optimizer, and the learning rate hyperparameters are adjusted.

[0011] Furthermore, the step of obtaining the tongue image further includes: Using a camera to capture tongue image video stream in real time in video streaming mode; Extract key frames at set time intervals T and convert them into image formats suitable for algorithm input; After resizing and normalizing the input image, it is input into the trained model for forward inference; If the model outputs the key point positions in the form of a heat map, the key point coordinates are determined based on the maximum value position in the heat map.

[0012] Furthermore, the step of performing transfer learning based on the model trained after evaluation using a pre-trained model trained on a large-scale human posture dataset and optimizing the inference speed includes: Based on key point detection accuracy, where key point detection accuracy, PCK, and model convergence are used to judge model performance; If the dataset is lower than the preset value, a pre-trained model trained on a large-scale human posture dataset is used for transfer learning; In actual deployment, model pruning and quantization techniques are used to reduce model complexity, and the inference speed is optimized through the TensorRT and ONNX frameworks.

[0013] The present invention also discloses a tongue image posture correction and integrity detection system based on key point detection, comprising: A first acquisition module acquires a tongue image, wherein the tongue image includes image information of different tongue postures, lighting conditions, and tongue shapes; A processing module, used for performing data enhancement and normalization processing on the collected image information; A building block for building a neural network model that extracts features using ResNet and combines it with the self-attention mechanism Transformer; An evaluation module, used for evaluating and training the neural network model; An optimization module, configured to perform transfer learning using a pre-trained model trained on a large-scale human posture dataset according to the evaluated trained model, and optimize the inference speed; The judgment module is used to calculate the angle between the center line of the tongue and the vertical direction of the image based on the position of the tongue tip and the tongue root. If it exceeds 10 degrees, the tongue posture is judged to be incorrect and the user is prompted to adjust it. The tongue tip, tongue root and left and right edge points are detected to determine whether the tongue in the image is complete. If the key point is not detected or is too close to the edge of the image, the user is prompted to retake the photo; The accuracy of the test is determined based on the anatomical relationship between the tongue tip, tongue root, and left and right edges. If it is unreasonable, it indicates that there is a test error; If the user fails to take pictures multiple times, the system will automatically adjust the shooting suggestions and display the real-time detection results of the tongue key points to assist the user in making adjustments.

[0014] Furthermore, the first acquisition module includes: A first mixing unit is used to form a mixed data set including a complete tongue and a tongue with a missing tongue tip by using the collected image information of different tongue postures, lighting conditions, and tongue shapes; The annotation unit is used to accurately annotate the tongue tip, tongue root, two points on each of the left and right tongue edges, and the midline key points in the image. When the tongue tip is missing, it is annotated as "tongue tip missing" and "imaginary tongue tip" and a "missing" label is added; A setting unit is used to set "visibility annotation" for invisible key points; The save unit is used to save the annotation data in JSON or XML format using the LabelMe and VGG Image Annotator tools, ensuring that the annotation files correspond to the image file names.

[0015] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0016] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0017] The beneficial effects of the present application are as follows: In data processing, the present invention collects polymorphic tongue images and enhances and normalizes them to enhance the generalization ability of the model. Its network architecture integrates ResNet and Transformer to accurately locate key points. Transfer learning and inference speed optimization not only improve performance but also ensure real-time performance. Through scientific posture and integrity detection logic, accurate judgment is made based on the position relationship of the tongue tip to ensure image quality. At the same time, it provides users with clear prompts and auxiliary adjustments, effectively improves the user's shooting experience, makes tongue image acquisition more efficient and accurate, and strongly promotes the development of smart Chinese medicine tongue diagnosis technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present application.

[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0021] Embodiment 1 like Figure 1 As shown, the present application provides a tongue image posture correction and integrity detection method based on key point detection, including: The tongue image posture correction and integrity detection method based on key point detection includes: S1. Obtaining a tongue image, wherein the tongue image includes image information of different tongue postures, lighting conditions, and tongue shapes; S2, performing data enhancement and normalization processing on the collected image information; S3. Build a neural network model that uses ResNet to extract features and combines it with the self-attention mechanism Transformer; S4, evaluating and training the neural network model; S5. Based on the model trained after the evaluation, a pre-trained model trained on a large-scale human posture dataset is used for transfer learning, and the inference speed is optimized; S6. Calculate the angle between the center line of the tongue and the vertical direction of the image based on the position of the tongue tip and tongue root. If it exceeds 10 degrees, the tongue posture is judged to be incorrect and the user is prompted to adjust it. Determine whether the tongue in the image is complete by detecting the tongue tip, tongue root and left and right edge points. S7, if the key point is not detected or is too close to the edge of the image, prompt the user to retake the photo; S8. Determine the accuracy of the test based on the anatomical relationship between the tongue tip, tongue root and left and right edges. If it is unreasonable, it indicates that there is a test error; S9. If the user fails to take photos multiple times, the system will automatically adjust the shooting suggestions and display the real-time detection results of the tongue key points to assist the user in making adjustments.

[0022] As described in the above steps S1-S9, in the data collection and labeling steps, the resolution of the collected tongue image must meet the requirement of being able to clearly distinguish the key structural points of the tongue, and during the labeling process, the operator must undergo professional training to ensure the accuracy and consistency of the labeling.

[0023] In terms of data processing, the present invention collects polymorphic tongue images and enhances and normalizes them to enhance the generalization ability of the model. Its network architecture integrates ResNet and Transformer to accurately locate key points. Transfer learning and inference speed optimization not only improve performance but also ensure real-time performance. Through scientific posture and integrity detection logic, accurate judgment is made based on the position relationship of the tongue tip to ensure image quality. At the same time, it provides users with clear prompts and auxiliary adjustments, effectively improves the user's shooting experience, makes tongue image acquisition more efficient and accurate, and strongly promotes the development of smart Chinese medicine tongue diagnosis technology.

[0024] By collecting tongue images containing multiple difference factors and performing data enhancement and normalization, the model can learn richer features and unified feature distribution, enhance the generalization ability of the model, reduce the risk of overfitting, and better cope with the complex and diverse tongue image situations in practical applications. The neural network model that combines ResNet with the self-attention mechanism Transformer makes full use of ResNet's feature extraction capabilities and Transformer's ability to enhance perception of specific areas, and can more accurately locate the key points of the tongue and improve the accuracy and reliability of detection.

[0025] By using pre-trained models on large-scale human posture datasets for transfer learning, the model's performance in tongue image posture correction and integrity detection tasks is improved while saving training time and resources. By optimizing the inference speed, real-time performance in practical applications is guaranteed, meeting the needs of intelligent tongue image collection equipment and telemedicine systems for rapid detection.

[0026] Accurate posture and integrity detection logic: The posture and integrity judgment standards based on the position relationship between the tongue tip, tongue root and edge points, as well as the consideration of the rationality of anatomical relationships, make the detection results more scientific and accurate, and can effectively identify situations where the tongue posture is incorrect, the image is incomplete and the detection is wrong, providing a high-quality image basis for subsequent diagnosis. When a problem is detected, it can provide clear prompt information to the user in a timely manner to guide the user to adjust the shooting; for users who have failed to shoot multiple times, it automatically adjusts the shooting suggestions and displays the real-time detection results of key points to assist in the adjustment, which greatly optimizes the user's shooting experience and improves the user's success rate in obtaining qualified tongue images.

[0027] Specifically, the step of obtaining the tongue image includes: By collecting image information of different tongue postures, lighting conditions, and tongue shapes, a mixed data set containing complete tongues and tongues with missing tongue tips is formed; Accurately mark the tongue tip, tongue root, two points on each of the left and right tongue edges, and the midline key points in the image. If the tongue tip is missing, mark "tongue tip missing" and "imaginary tongue tip" and add a "missing" label; Set "visibility annotation" for invisible key points; Use LabelMe and VGG Image Annotator tools to save annotation data in JSON or XML format, making sure that the annotation files correspond to the image file names.

[0028] Through the above steps, image information of different tongue postures, lighting conditions, and tongue shapes is collected to form a mixed data set, covering a variety of situations, making the data more representative and practical, and better training the model to adapt to various tongue image scenarios in practical applications. Accurately mark the key points of the tongue tip, tongue root, left and right tongue edges, and midline, special marking when the tongue tip is missing, and visibility marking for invisible key points, ensuring the integrity and accuracy of the marking, providing a reliable foundation for subsequent model training.

[0029] Specifically, the step of performing data enhancement and normalization processing on the collected image information includes: Divide the dataset into 80% training set and 20% test set. K-fold cross validation is used, where K=5, to improve the generalization ability of the model.

[0030] In the data processing and segmentation steps, the parameter settings of the data enhancement operation should be reasonably adjusted according to the actual characteristics and variation range of the tongue image, such as the rotation angle between -45 degrees and 45 degrees, and the scaling ratio between 0.8 and 1.2, to ensure that the enhanced data is both diverse and in line with actual application scenarios.

[0031] Specifically, the steps of constructing a neural network model that extracts features using ResNet and combines the self-attention mechanism Transformer include: To solve the problem of key point detection at the bottom boundary of an image, image padding, mirror filling, cropping, flipping, rotation, multi-scale training and loss function are used for weighted calculation. Calculate the error between the key point coordinates and the true annotation value based on the mean square error (MSE) and heatmap loss function; Assume that there are n key points, for each key point, its predicted coordinates are , the actual marked coordinates are , then the formula for the error of the true annotation value is: .

[0032] The input tongue image is subjected to the convolution layer to extract features, and the deconvolution layer or upsampling layer to generate a key point heat map or coordinate prediction value; The network weights are updated via back-propagation and the Adam or SGD optimizer, and the learning rate hyperparameters are adjusted.

[0033] Through this method, a variety of measures were taken to solve the common and difficult problem of key point detection at the bottom boundary of the image, including image padding, mirror filling, cropping and flipping, rotation, multi-scale training, and weighted calculation of loss function. This comprehensively and specifically improved the model's detection capability in the boundary area, ensured the integrity and accuracy of key point detection, and avoided detection errors caused by boundary problems.

[0034] The error between the key point coordinates and the true annotation value is calculated based on the mean square error (MSE) and heatmap loss functions. MSE can measure the difference in coordinate values, and Heatmap Loss can further optimize the positioning accuracy of the key points from the perspective of the heat map. The combination of the two makes the error calculation more comprehensive and accurate, which helps the model to learn and converge better and improve the accuracy of key point detection.

[0035] The above steps clearly explain the process of extracting features through the convolution layer after inputting the tongue image, and then generating the key point heat map or coordinate prediction value through the deconvolution layer or upsampling layer. This is in line with the professional process of deep learning model processing images, making the working principle and data processing process of the model clear at a glance, easy to understand and reproduce. Updating the network weights through backpropagation and Adam or SGD optimizer, and adjusting the hyperparameters of the learning rate are common and effective optimization methods in deep learning model training. They can continuously optimize the model performance, make it better adapt to the tongue image key point detection task, and ensure efficient training and good performance of the model.

[0036] Specifically, the step of obtaining the tongue image further includes: Using a camera to capture tongue image video stream in real time in video streaming mode; Extract key frames at set time intervals T and convert them into image formats suitable for algorithm input; After resizing and normalizing the input image, it is input into the trained model for forward inference; If the model outputs the key point positions in the form of a heat map, the key point coordinates are determined based on the maximum value position in the heat map.

[0037] If the model outputs the key point positions in the form of a heat map, the key point coordinates are determined based on the maximum value position in the heat map. This method, based on the characteristics of the heat map, can accurately locate the key points, fully utilize the information output by the model, improve the accuracy and reliability of key point detection, and provide an accurate data basis for tongue posture correction and integrity detection.

[0038] Specifically, the step of performing transfer learning based on the evaluated trained model using a pre-trained model trained on a large-scale human posture dataset and optimizing the inference speed includes: Based on key point detection accuracy, where key point detection accuracy, PCK, and model convergence are used to judge model performance; Assuming that the historical maximum value of the dataset is S, when the current dataset is less than 30% of S, the pre-trained model trained on a large-scale human posture dataset is used for transfer learning; In actual deployment, model pruning and quantization techniques are used to reduce model complexity, and the inference speed is optimized through the TensorRT and ONNX frameworks.

[0039] Among them, the key point detection accuracy is measured by calculating the average Euclidean distance between the predicted key points and the true key points, PCK is measured by calculating the percentage of key points correctly predicted within a given tolerance error range, and model convergence is measured by observing the change trend of the loss function during training.

[0040] The accuracy of the model in locating key points is measured by calculating the distance error between the predicted key points and the actual annotated key points. For example, for the tongue tip key point, the actual annotated position is (100, 150), and the model predicts the position to be (102, 153). The Euclidean distance between the two points is calculated. The smaller the average distance error of multiple key points, the higher the key point detection accuracy of the model.

[0041] Set a tolerance error range, such as 5 pixels. When the distance between the predicted key point and the real key point is within this range, the key point is considered to be detected correctly. Count the correct proportion of all key points. The higher the PCK value, the better the model performance. For example, in a set of test data, there are 100 key points, of which 80 key points have predicted positions within the tolerance error range, then the PCK value is 80%. Observe the trend of the loss function value of the model during training. If the loss function value gradually decreases and tends to stabilize with the increase in the number of training iterations, it means that the model is constantly learning and gradually converging to a better state. For example, at the beginning of training, the loss function value is 0.8, and after 1000 iterations, it decreases to 0.2 and remains basically unchanged, which indicates that the model has good convergence.

[0042] If the data set is lower than the preset value, the pre-trained model trained on a large-scale human posture data set is used for transfer learning. Directly training the model will lead to overfitting, that is, the model performs well on the training set, but has poor generalization ability on new data. The large-scale human posture data set contains rich human posture information, which has certain similarities and correlations with tongue key point detection. Through transfer learning, the general features and knowledge learned by the pre-trained model on large-scale data can be used to quickly adapt to the tongue key point detection task, reduce training time and data requirements, and improve the performance and generalization ability of the model.

[0043] Assume that there are only 1,000 tongue images for training the model, which is a relatively small amount of data for a deep learning model. However, there are 100,000 images in the human posture dataset, and the pre-trained model has learned the relevant knowledge of the characteristics and positional relationships of various parts of the human body on this large dataset. Migrate some of the network structures and parameters of the pre-trained model to the tongue key point detection model, and then fine-tune the training on 1,000 tongue images. For example, the first few layers of the pre-trained model mainly learn the underlying features of the image, such as edges and textures. These features are common in tongue images and human posture images. The parameters of these layers can be fixed, and only the subsequent network layers can be trained to adapt them to the task of tongue key point detection. In this way, even if the tongue image dataset is small, you can take advantage of the pre-trained model to quickly obtain a tongue key point detection model with good performance.

[0044] The present invention also discloses a tongue image posture correction and integrity detection system based on key point detection, comprising: A first acquisition module acquires a tongue image, wherein the tongue image includes image information of different tongue postures, lighting conditions, and tongue shapes; A processing module, used for performing data enhancement and normalization processing on the collected image information; A building block for building a neural network model that extracts features using ResNet and combines it with the self-attention mechanism Transformer; An evaluation module, used for evaluating and training the neural network model; An optimization module, configured to perform transfer learning using a pre-trained model trained on a large-scale human posture dataset according to the evaluated trained model, and optimize the inference speed; The judgment module is used to calculate the angle between the center line of the tongue and the vertical direction of the image based on the position of the tongue tip and the tongue root. If it exceeds 10 degrees, the tongue posture is judged to be incorrect and the user is prompted to adjust it. The tongue tip, tongue root and left and right edge points are detected to determine whether the tongue in the image is complete. If the key point is not detected or is too close to the edge of the image, the user is prompted to retake the photo; The accuracy of the test is determined based on the anatomical relationship between the tongue tip, tongue root, and left and right edges. If it is unreasonable, it indicates that there is a test error; If the user fails to take pictures multiple times, the system will automatically adjust the shooting suggestions and display the real-time detection results of the tongue key points to assist the user in making adjustments.

[0045] Furthermore, the first acquisition module includes: A first mixing unit is used to form a mixed data set including a complete tongue and a tongue with a missing tongue tip by using the collected image information of different tongue postures, lighting conditions, and tongue shapes; The annotation unit is used to accurately annotate the tongue tip, tongue root, two points on each of the left and right tongue edges, and the midline key points in the image. When the tongue tip is missing, it is annotated as "tongue tip missing" and "imaginary tongue tip" and a "missing" label is added; A setting unit is used to set "visibility annotation" for invisible key points; The save unit is used to save the annotation data in JSON or XML format using the LabelMe and VGG Image Annotator tools, ensuring that the annotation files correspond to the image file names.

[0046] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0047] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0048] Embodiment 2 The difference between this embodiment and the first embodiment is that a 5-megapixel camera is used to capture the tongue video stream in real time in the video stream mode. The high-pixel camera can provide clear and detailed tongue images. The video stream mode can capture the dynamic changes of the tongue, which helps to fully observe the characteristics of the tongue in different states. For example, during the patient's speaking and swallowing, the shape and color of the tongue will change slightly. The video stream can record this information and provide more comprehensive data for subsequent analysis.

[0049] For example, when a doctor needs to observe the color changes of a patient's tongue in its natural state, the video stream can capture the subtle differences in color of the tongue from protruding to retracting, and discover temporary changes in tongue color caused by certain diseases, while static images would miss this information.

[0050] Extract key frames at the set time interval T and convert them into an image format suitable for algorithm input. The time interval T is set in different ranges according to different observation requirements: when it is necessary to observe rapid dynamic changes of the tongue (such as speaking, rapid swallowing), the time interval T ranges from 0.5 seconds to 1 second; when observing relatively slow state changes of the tongue (such as natural extension, gradual color change), the time interval T ranges from 1 second to 5 seconds. By setting the time interval to extract key frames, the amount of data can be reduced, the processing efficiency can be improved, and representative tongue images can be obtained. Converting video frames into a format suitable for algorithm input ensures data consistency and compatibility, making subsequent model processing smoother. For example, if the time interval T is set to 1 second, when the patient sticks out his tongue for 5 seconds, 5 key frames will be extracted. In this way, without losing important information, the original large amount of video data is streamlined into a few key images, which is convenient for subsequent analysis and processing. For a tongue image analysis software, it can select within the corresponding time interval T range according to the actual tongue state to be observed to perform key point detection and analysis, avoiding too many images to increase the computational burden.

[0051] After resizing and normalizing the input image, it is input into the trained model for forward reasoning. The resizing makes the image meet the model input requirements to avoid model processing difficulties or inaccurate results due to image size differences. Normalization can eliminate data differences between different images, allowing the model to better learn and process data, and improve the accuracy and stability of the model. If the model requires the input image size to be 224x224 pixels, and the original images collected have different sizes, all images can be unified to 224x224 pixels through resizing. Normalization, for example, normalizes the pixel value range of the image from 0-255 to 0-1, so that the model will not be affected by differences in pixel values ​​when processing different images. For example, a brighter tongue image and a darker tongue image, after normalization, their data distribution is more consistent, and the model can detect and analyze key points more accurately.

[0052] If the model outputs the key point position in the form of a heat map, the key point coordinates are determined based on the maximum value position in the heat map. The heat map is an effective way to represent the key point position. The key point coordinates are determined by finding the maximum value position in the heat map, which has high accuracy and reliability. The heat map can reflect the model's predicted probability distribution of the key point position, and the maximum value position usually corresponds to the most likely key point position. Assume that in the heat map output by the model, the heat map area at the tip of the tongue position shows an obvious peak, and the location of this peak is the key point coordinate of the tip of the tongue predicted by the model. For example, in a tongue image heat map, the pixel value of the tip of the tongue area reaches the highest at a certain point, such as the pixel value at the coordinate (100, 150) is 255, and the surrounding pixel values ​​gradually decrease, then the key point coordinates of the tip of the tongue can be determined to be (100,150). This method can accurately locate the key points of the tip of the tongue, the root of the tongue, and the left and right edges of the tongue, providing an accurate data basis for posture correction and integrity detection of the tongue image. For example, by accurately detecting the position of the tongue tip and tongue root, the angle between the center line of the tongue and the vertical direction of the image can be calculated to determine whether the tongue posture is correct; by detecting the left and right tongue edge points, it can be determined whether the tongue is complete in the image or whether there is any part missing.

[0053] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0054] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0055] The above description is only a preferred embodiment of the present invention, and does not limit the scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the protection scope of the present application.

Claims

1. A method for detecting tongue image posture correction and integrity based on key point detection, characterized in that: The following steps are involved: Obtaining a tongue image, wherein the tongue image includes image information of different tongue postures, lighting conditions, and tongue shapes; Perform data enhancement and normalization processing on the collected image information; Build a neural network model that uses ResNet to extract features and combines it with the self-attention mechanism Transformer; Performing evaluation training on the neural network model; Based on the model trained by the evaluation, a pre-trained model trained on a large-scale human posture dataset is used for transfer learning, and the inference speed is optimized; The angle between the center line of the tongue and the vertical direction of the image is calculated based on the position of the tongue tip and tongue root. If it exceeds 10 degrees, the tongue posture is considered to be incorrect and the user is prompted to adjust it. The tongue tip, tongue root and left and right edge points are detected to determine whether the tongue in the image is complete. If the key point is not detected or is too close to the edge of the image, the user is prompted to retake the photo; The accuracy of the test is determined based on the anatomical relationship between the tongue tip, tongue root, and left and right edges. If it is unreasonable, it indicates that there is a test error; If the user fails to take pictures multiple times, the system will automatically adjust the shooting suggestions and display the real-time detection results of the tongue key points to assist the user in making adjustments.

2. The method for detecting tongue image posture correction and integrity based on key point detection according to claim 1, characterized in that: The step of obtaining the tongue image comprises: By collecting image information of different tongue postures, lighting conditions, and tongue shapes, a mixed data set containing complete tongues and tongues with missing tongue tips is formed; Accurately mark the tongue tip, tongue root, two points on each of the left and right tongue edges, and the midline key points in the image. If the tongue tip is missing, mark "tongue tip missing" and "imaginary tongue tip" and add a "missing" label; Set "visibility annotation" for invisible key points; Use LabelMe and VGG Image Annotator tools to save annotation data in JSON or XML format, making sure that the annotation files correspond to the image file names.

3. The method for detecting tongue image posture correction and integrity based on key point detection according to claim 1, characterized in that: The step of performing data enhancement and normalization processing on the collected image information includes: Divide the dataset into 80% training set and 20% test set. K-fold cross validation is used, where K=5, to improve the generalization ability of the model.

4. The method for detecting tongue image posture correction and integrity based on key point detection according to claim 1, characterized in that: The steps of constructing a neural network model that extracts features using ResNet and combines the self-attention mechanism Transformer include: To solve the problem of key point detection at the bottom boundary of an image, image padding, mirror filling, cropping, flipping, rotation, multi-scale training and loss function are used for weighted calculation. Calculate the error between the key point coordinates and the true annotation value based on the mean square error and heat map loss function; The input tongue image is subjected to the convolution layer to extract features, and the deconvolution layer or upsampling layer to generate a key point heat map or coordinate prediction value; The network weights are updated via back-propagation and the Adam or SGD optimizer, and the learning rate hyperparameters are adjusted.

5. The method for detecting tongue image posture correction and integrity based on key point detection according to claim 1, characterized in that: The step of obtaining the tongue image also includes: Using a camera to capture tongue image video stream in real time in video streaming mode; Extract key frames at set time intervals T and convert them into image formats suitable for algorithm input; After resizing and normalizing the input image, it is input into the trained model for forward inference; If the model outputs the key point positions in the form of a heat map, the key point coordinates are determined based on the maximum value position in the heat map.

6. The method for detecting tongue image posture correction and integrity based on key point detection according to claim 1, characterized in that: The step of using a pre-trained model trained on a large-scale human posture dataset to perform transfer learning based on the model trained after the evaluation, and optimizing the inference speed, comprises: Based on key point detection accuracy, where key point detection accuracy, PCK, and model convergence are used to judge model performance; If the dataset is lower than the preset value, a pre-trained model trained on a large-scale human posture dataset is used for transfer learning; In actual deployment, model pruning and quantization techniques are used to reduce model complexity, and the inference speed is optimized through the TensorRT and ONNX frameworks.

7. A tongue image posture correction and integrity detection system based on key point detection, characterized in that: include: A first acquisition module acquires a tongue image, wherein the tongue image includes image information of different tongue postures, lighting conditions, and tongue shapes; A processing module, used for performing data enhancement and normalization processing on the collected image information; A building block for building a neural network model that extracts features using ResNet and combines it with the self-attention mechanism Transformer; An evaluation module, used for evaluating and training the neural network model; An optimization module, configured to perform transfer learning using a pre-trained model trained on a large-scale human posture dataset according to the evaluated trained model, and optimize the inference speed; The judgment module is used to calculate the angle between the center line of the tongue and the vertical direction of the image based on the position of the tongue tip and the tongue root. If it exceeds 10 degrees, the tongue posture is judged to be incorrect and the user is prompted to adjust it. The tongue tip, tongue root and left and right edge points are detected to determine whether the tongue in the image is complete; If the key point is not detected or is too close to the edge of the image, the user is prompted to retake the photo; The accuracy of the test is determined based on the anatomical relationship between the tongue tip, tongue root, and left and right edges. If it is unreasonable, it indicates that there is a test error; If the user fails to take pictures multiple times, the system will automatically adjust the shooting suggestions and display the real-time detection results of the tongue key points to assist the user in making adjustments.

8. The tongue image posture correction and integrity detection system based on key point detection according to claim 7 is characterized in that: The first acquisition module includes: A first mixing unit is used to form a mixed data set including a complete tongue and a tongue with a missing tongue tip by using the collected image information of different tongue postures, lighting conditions, and tongue shapes; The annotation unit is used to accurately annotate the tongue tip, tongue root, two points on each of the left and right tongue edges, and the midline key points in the image. When the tongue tip is missing, it is annotated with "tongue tip missing" and "imaginary tongue tip" and an additional "missing" label; Setting unit, used to set "visibility annotation" for invisible key points; The save unit is used to save the annotation data in JSON or XML format using the LabelMe and VGG Image Annotator tools, ensuring that the annotation files correspond to the image file names.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images