Tongue picture image prick identification method and system, terminal and medium
By constructing a prick recognition network based on object detection and single-point prick clustering segmentation, the improved YoloV5 network and SIoU loss function are used, combined with color clustering and semantic segmentation models, the poor generalization ability in prick recognition of tongue image prick recognition is solved. and the problem of low accuracy achieves high accuracy and fine-grained information acquisition, supporting the application of smart medical care and digitalization of traditional Chinese medicine.
Patent Information
- Application Number
- CN202310925236.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-25
- Publication Date
- 2024-11-01
AI Technical Summary
The prior art has problems such as poor generalization ability, low accuracy and insufficient access to fine-grained information in the recognition of tongue image pricks, making it difficult to accurately distinguish between pricks and petechiae.
A prick recognition network based on object detection and single-prick clustering segmentation is constructed, and the target detection is used using the improved YoloV5 network and SIoU loss function, and a high-precision prick recognition result is obtained by combining color clustering and semantic segmentation models.
It achieves high accuracy and fine-grained information acquisition for thorn recognition, strengthens generalization ability, and can assist in the output of quantitative numerical and severity analysis, which has research significance in smart medical care and Chinese medicine digitization.
Smart Images

Figure CN118887697A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of tongue image processing, and in particular to a tongue image prick recognition method, system, terminal and medium. Background Art
[0002] Tongue diagnosis is an important part of TCM diagnosis. TCM practitioners use their personal knowledge and experience to assist in the diagnosis of diseases by observing the patient's tongue. However, each TCM practitioner has different standards, which lack quantification and objectivity. In recent years, with the continuous advancement of computer technology, especially in image recognition, computer recognition of tongue images has become the main means of quantifying tongue images.
[0003] Points and pricks are an important feature of tongue appearance, which can be divided into points and thorns. Points refer to red, white or black star-shaped dots protruding from the tongue surface; thorns refer to red or yellow-black points on the tongue papilla that are like thorns and are difficult to touch. It indicates extreme heat in the internal organs or excessive heat in the blood. The number of points and pricks can reflect the nature and degree of the disease, especially in the judgment of febrile diseases.
[0004] Modern research generally believes that the appearance of tongue prick is related to the congestion and expansion of the tongue papillae. Some researchers have used a microscope to prick the tip of the tongue and found the phenomenon of tongue papilla expansion. In addition, current studies have found that the appearance of tongue prick is related to certain diseases. For example, Wu Xiqing et al. compared the prick area of patients with chronic hepatitis B before and after treatment with fasting blood sugar, white blood cell count, and platelet count, and found that there is a correlation. The prick area may become an evaluation index for the efficacy of chronic hepatitis B; Hu Guangqin et al. studied insomnia patients and believed that blood yield was negatively correlated with pricked tongue; Chen Jialiang found that patients with non-alcoholic fatty liver disease had a higher probability of pricked tongue than normal people; in addition, Zhang Ping found that the degree of pricked tongue was positively correlated with the severity of clinical symptoms of chronic gastritis spleen and stomach damp-heat syndrome. It can be seen that the judgment of pricked tongue has a certain auxiliary significance in the diagnosis and treatment of diseases. Different fine-grained characteristics such as the distribution area, density, overall number and monomer size of pricks are of great value in the diagnosis of diseases.
[0005] There are many methods for identifying acupuncture points in TCM tongue images. Xu Jiatuo first proposed computer identification of acupuncture points. He established a feature vector subspace to complete the definition of features, and then matched the tongue image with feature vectors to identify acupuncture points, with an accuracy of 77.10%. The low accuracy of this experiment is that acupuncture points and ecchymosis points are similar in two-dimensional images and difficult to distinguish. This method needs to be further improved. Wang Xuemin et al. used green light sources to enhance the contrast of acupuncture points, and used the Log (Laplacian of Gaussian) edge detection method to remove noise to identify acupuncture points and count the number of acupuncture points. Wang Sheng et al. identified the spots on the tongue surface through a spot detection algorithm, clustered the spots using K-means clustering, and identified acupuncture points and ecchymosis points based on color differences. However, the above methods have problems such as poor generalization ability, low accuracy, and insufficient acquisition of fine-grained information. Summary of the invention
[0006] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a tongue image prick recognition method, system, terminal and medium for solving the above-mentioned prior art problems.
[0007] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for prick recognition of tongue images, the method comprising: constructing a prick recognition network based on target detection and single-prick clustering segmentation; based on the constructed prick recognition network, performing target detection and single-prick clustering segmentation on the tongue image to be identified to obtain corresponding prick recognition results; wherein the prick recognition network comprises: a cascaded prick target detection model for target detection and a monomer prick segmentation model for single-prick clustering segmentation.
[0008] In one embodiment of the present invention, the construction of a prick recognition network based on target detection and clustering segmentation includes: obtaining multiple target detection tongue image samples obtained by performing prick target labeling based on multiple collected tongue image images; training a target detection network based on each target detection tongue image sample to obtain the prick target detection model; pre-labeling each single prick image in each prick target detection result output by the prick target detection model based on color clustering to obtain a corresponding tongue image monomer segmentation image sample; training a semantic segmentation model based on each tongue image monomer segmentation image sample to obtain the monomer prick segmentation model.
[0009] In one embodiment of the present invention, the target detection network is obtained by improving the benchmark YoloV5 network; wherein the benchmark YoloV5 network includes: an input network, a feature extraction network, a feature fusion network, and a model detection head network including multiple target detection layers.
[0010] In one embodiment of the present invention, the target detection network is obtained by adding a target detection layer for shallow feature target detection to the model detection head network on the benchmark YoloV5 network.
[0011] In one embodiment of the present invention, the target detection network is trained using a SIoU loss function to obtain the prick target detection model; wherein the SIoU loss function includes: angle loss, distance loss, shape loss, and IoU loss.
[0012] In one embodiment of the present invention, the method of pre-labeling each single-point thorn image based on color clustering includes: converting each single-point thorn image from RGB space to LAB space to obtain a corresponding single-point thorn color conversion image; extracting a* channel data from each single-point thorn color conversion image, and performing histogram equalization and gamma correction processing to obtain a corresponding single-point thorn enhanced image; performing K-means clustering on each single-point thorn enhanced image to obtain a single-point thorn cluster image; performing mask processing on each single-point thorn cluster image, and then performing bilateral filtering and corrosion operations to obtain a corresponding monomer thorn segmentation image, so as to obtain a corresponding tongue image monomer segmentation image sample.
[0013] In one embodiment of the present invention, the semantic segmentation model adopts the SeaFormer network.
[0014] In one embodiment of the present invention, the method further comprises: testing and verifying the constructed acupuncture point recognition network based on a plurality of tongue image images annotated with acupuncture points.
[0015] In one embodiment of the present invention, the target detection and cluster segmentation of the tongue image to be identified to obtain the corresponding prick recognition result includes: inputting the tongue image to be identified into the prick target detection model for target detection to obtain the corresponding tongue target detection result; inputting each single prick image in the tongue target detection result into the monomer prick segmentation model for single prick cluster segmentation to obtain the corresponding single prick cluster segmentation result.
[0016] To achieve the above-mentioned purpose and other related purposes, the present invention provides a tongue image prick recognition system, the system comprising: a prick recognition network construction module, used to construct a prick recognition network based on target detection and single prick clustering segmentation; a prick recognition module, connected to the prick recognition network construction module, used to perform target detection and single prick clustering segmentation on the tongue image to be identified based on the constructed prick recognition network to obtain corresponding prick recognition results; wherein the prick recognition network comprises: a cascaded prick target detection model for target detection and a monomer prick segmentation model for single prick clustering segmentation.
[0017] To achieve the above-mentioned purpose and other related purposes, the present invention provides a tongue image prick recognition terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store computer programs; the one or more processors are connected to the memories and are used to run the computer program to execute the tongue image prick recognition method.
[0018] To achieve the above-mentioned purpose and other related purposes, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program executes the tongue image prick recognition method when executed by one or more processors.
[0019] As described above, the present invention is a method, system, terminal and medium for prick recognition of tongue images, which has the following beneficial effects: the present invention constructs a prick recognition network based on target detection and single prick clustering segmentation, and performs target detection and single prick clustering segmentation on the tongue image to be identified based on the constructed prick recognition network to obtain the corresponding prick recognition result. The prick recognition network constructed by the present invention has strong generalization ability, can obtain sufficient fine-grained information and high prick recognition accuracy, can assist in outputting prick quantization values and intuitively analyzing the severity of pricks, and has research significance for realizing smart medical care and digitization of traditional Chinese medicine. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 Shown is a schematic flow chart of a tongue image prick recognition method in one embodiment of the present invention.
[0021] Figure 2 It is a schematic diagram of the flow of a method for constructing a prick recognition network in one embodiment of the present invention.
[0022] Figure 3 Shown is a schematic diagram of the structure of a benchmark YoloV5 network in one embodiment of the present invention.
[0023] Figure 4 Shown is a schematic diagram of the structure of an improved benchmark YoloV5 network in one embodiment of the present invention.
[0024] Figure 5 Shown is a schematic diagram of an angle loss calculation method according to an embodiment of the present invention.
[0025] Figure 6 Shown is a schematic diagram of a distance loss calculation method in an embodiment of the present invention.
[0026] Figure 7 Shown is a schematic diagram of a single-point spike clustering image in an embodiment of the present invention.
[0027] Figure 8 Shown is a schematic diagram of a single point segmentation image in an embodiment of the present invention.
[0028] Fig. 9 It is a schematic diagram of the process of constructing a thorn recognition algorithm based on target detection and clustering segmentation in one embodiment of the present invention.
[0029] Fig.10 Shown is a schematic diagram of the structure of a tongue image prick recognition system in one embodiment of the present invention.
[0030] Fig.11 Shown is a schematic diagram of the structure of a tongue image prick recognition terminal in one embodiment of the present invention. DETAILED DESCRIPTION
[0031] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0032] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present invention. It should be understood that other embodiments may also be used, and that mechanical composition, structural, electrical and operational changes may be made without departing from the spirit and scope of the present invention. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is limited only by the claims of the published patents. The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.
[0033] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a part is said to "include" a certain constituent element, unless otherwise stated, it does not exclude other constituent elements, but means that other constituent elements may be included.
[0034] The terms first, second and third mentioned herein are used to describe various parts, components, regions, layers and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer or segment from other parts, components, regions, layers or segments. Therefore, the first part, component, region, layer or segment described below may refer to the second part, component, region, layer or segment within the scope of the present invention.
[0035] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.
[0036] The present invention provides a method for spot prick recognition of tongue images, by constructing a spot prick recognition network based on target detection and single spot prick clustering segmentation, and performing target detection and single spot prick clustering segmentation on the tongue image to be identified based on the constructed spot prick recognition network to obtain the corresponding spot prick recognition result. The spot prick recognition network constructed by the present invention has strong generalization ability, can obtain sufficient fine-grained information and has high spot prick recognition accuracy, can assist in outputting spot prick quantization values and intuitively analyzing the severity of the prick prick, and has research significance for realizing smart medical care and digitization of traditional Chinese medicine.
[0037] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.
[0038] like Figure 1 A schematic diagram showing a flow chart of a tongue image prick recognition method according to an embodiment of the present invention.
[0039] The method comprises:
[0040] Step S1: Construct a prick recognition network based on target detection and single prick clustering and segmentation.
[0041] In detail, the prick recognition network includes: a cascaded prick target detection model and a single prick segmentation model. The prick target detection model is used to perform target detection to obtain the corresponding tongue image target detection result; the single prick segmentation model is used to perform single prick cluster segmentation on the tongue image target detection result to obtain the corresponding single prick cluster segmentation result, that is, to obtain the prick recognition result.
[0042] In one embodiment, if Figure 2 As shown, step S1 includes:
[0043] Step 1: Acquire multiple target detection tongue image samples obtained by performing point target annotation based on multiple collected tongue image images;
[0044] Specifically, multiple tongue images were collected by the tongue imager, and then the collected image data were screened. The screening criteria included inclusion criteria and exclusion criteria. The inclusion criteria were: 1) normal image brightness and clarity; 2) normal tongue posture; 3) natural tongue surface without human and pathological changes; 4) unprocessed images without human changes. The exclusion criteria were: 1) too bright, too dark, or blurred image brightness; 2) too short tongue extension, excessive force, curled tongue, or tongue extension to one side; 3) tongue surface abnormalities, such as tongue nails, tongue lines, bifurcated tongue, tongue surface sores, tongue bleeding, tongue surface staining, food residue or water marks on the tongue surface; 4) images were artificially edited, modified, photoshopped, beautified, stretched, or watermarked.
[0045] The screened multiple tongue images are annotated. Preferably, three experienced Chinese medicine practitioners use labelme software to annotate the tongue images with annotation boxes according to the internal tongue annotation specifications to obtain multiple target detection tongue image samples.
[0046] In a preferred embodiment, the annotated tongue images can be divided into training set: validation set: test set ratio of 6:2:2, wherein the training set and validation set are used to complete network model training and optimize model parameters, and the test set is used to analyze model performance.
[0047] Step 2: Train the target detection network based on each target detection tongue image sample to obtain the prick target detection model; specifically, input each target detection tongue image sample into the target detection network, combine the prick target detection results of each target detection tongue image sample output by the target detection network and the real annotation results to obtain the prick target detection model by using the corresponding loss function for training, and obtain the prick target detection results output by the prick target detection model.
[0048] Step 3: Pre-labeling each single prick image in the prick target detection results corresponding to each target detection tongue image sample output by the prick target detection model based on color clustering to obtain a corresponding tongue image monomer segmentation image sample;
[0049] Step 4: Train a semantic segmentation model based on each tongue image monomer segmentation image sample to obtain the monomer point segmentation model.
[0050] In one embodiment, the target detection network is obtained by improving the benchmark YoloV5 network.
[0051] Among them, the YoloV5 network structure has outstanding performance in actual landing projects due to its excellent detection performance. The schematic diagram of the benchmark YoloV5 network structure is shown in the figure. Figure 3 As shown in the figure, the baseline YoloV5 network includes: input network, feature extraction network (Backbone), feature fusion network (Neck), and model detection head network (Prediction) containing multiple target detection layers. The baseline YoloV5 network extracts features and fuses multiple features of the neck structure composed of FPN and PANet, and finally obtains the target detection frame coordinates and categories through the model detection head network to achieve specific target detection.
[0052] In one embodiment, compared with the entire tongue, the prick point is a small target. The detection of small targets has always been a difficult problem in target detection. The benchmark YoloV5 network performs poorly in the detection of small targets. The reason is that the model lacks the use of rich detail information in the shallow network. Therefore, we introduce a target detection layer specifically for small targets on the basis of the original one, and improve the benchmark YoloV5 network. The improved network is as follows: Figure 4 As shown. The target detection layer, before being inserted into the other three detection layers, includes: an upsampling module, a fusion module, and a convolution module connected to the feature fusion network; the feature map sizes output by the original three target detection layers are 80*80*3×(n+80), 40*40*3×(n+80), and 20*20*3×(n+80), respectively, where n is the number of categories required for target detection, and the feature map size output by the inserted target detection layer for shallow feature target detection is 160*160*3×(n+80).
[0053] The improved YoloV5 network greatly enhances the model's perception of high-resolution features, making it more sensitive to subtle information such as dots and thorns, thereby improving detection capabilities. At the same time, because it makes full use of the detail clues provided by the shallow network, the small target detection branch can give more accurate positioning results and improve the accuracy of dot selection.
[0054] In one embodiment, in the baseline YoloV5 model, GIoU is used as the loss of the regression frame, and the calculation formula of GIoU is shown in formula (1):
[0055]
[0056] Among them, A and B are two boxes used to calculate the loss (the model prediction box and the real annotation box), C is the minimum circumscribed rectangle of the two boxes AB, and IoU is the intersection-over-union ratio of the two boxes AB. From the calculation method of GIoU, it can be seen that when the prediction box and the target box are in an inclusion relationship, the relative positions of the two cannot be distinguished, which leads to divergence during regression calculation.
[0057] The root cause of the above problem is that the direction between the real box and the predicted box is not taken into account, which makes it difficult to converge. To solve this problem, this solution introduces SIoU loss to replace the initial GIoU loss to train the object detection network to obtain the prick object detection model. The SIoU loss function consists of four parts: angle loss (Angle cost), distance loss (Distancecost), shape loss (Shape cost) and IoU loss (IoU cost).
[0058] Among them, the calculation method of SIoU loss function includes:
[0059] 1. Calculate the angle cost, such as Figure 5 As shown, the calculation formula is shown in formula (2):
[0060]
[0061] in,
[0062]
[0063] 2. Calculate the distance cost, such as Figure 6 As shown, the calculation formula is shown in formula (5):
[0064] Δ=∑ t=x,y (1-e -γρt ); (5)
[0065] in,
[0066]
[0067] 3. Calculate the shape cost. The calculation formula is shown in formula (7):
[0068]
[0069] in,
[0070]
[0071] 4. Calculate the IoU loss (IoU cost). The calculation formula is shown in formula (9):
[0072]
[0073] In summary, the final loss function is shown in formula (10):
[0074]
[0075] The SIoU loss function introduces an angle penalty term to make the output change smoother and more continuous, unlike the traditional IoU with large jitter. This provides a more stable box selection result for the detection task. At the same time, considering the shape of the target box, it encourages the selection of a prediction box that is closer to a regular rectangle, so as to generate a box selection result that is more in line with human visual habits. In addition, the introduced angle penalty term can better distinguish dense small targets, and it is not easy to select multiple small targets as one box, which enhances the sensitivity of the box selection result to small targets, thereby improving the recognition accuracy of the model for small targets.
[0076] In one embodiment, due to the limitations of target detection, in order to obtain more fine-grained information, we need to obtain the shape and size information of a single point. On the basis of this detection result, if we want to obtain a high-precision point segmentation result, the segmentation method based on deep learning can certainly achieve good results, but it requires a large number of mask annotations, which increases the annotation cost. To improve this problem, this paper adopts a segmentation pre-annotation method based on color clustering. The specific operation is as follows:
[0077] Convert each single-point spike image from RGB space to LAB space to obtain the corresponding single-point spike color conversion image;
[0078] Extract a* channel data from each single-point thorn color conversion image, and perform histogram equalization and gamma correction processing to obtain the corresponding single-point thorn enhanced image;
[0079] Perform K-means clustering on each single-point spike-enhanced image to obtain a single-point spike clustered image, such as Figure 7 shown.
[0080] Mask processing is performed on each single point clustering image, and then bilateral filtering and corrosion operations are performed to obtain the corresponding single point segmentation image. Figure 8 As shown, the corresponding tongue image monomer segmentation image sample is obtained by combining with the corresponding single point prick image.
[0081] In one embodiment, the semantic segmentation model adopts the SeaFormer network, which combines the respective advantages of the CNN structure and the Transformer structure. It can not only quickly generate a local feature pyramid in high-resolution image processing, but also obtain rich semantic information and a huge receptive field. At the same time, the design adds an axis compression to enhance attention of the position information. On the one hand, the Q, K, and V features are axially compressed and then the attention is enhanced. At the same time, the Q, K, and V features are used to enhance the local information using a convolutional network. Finally, the two are fused to output enhanced features.
[0082] In one embodiment, the method further comprises: testing and verifying the constructed prick recognition network based on a plurality of tongue images annotated with prick targets. Preferably, the plurality of tongue images annotated with prick targets are divided into a training set: a validation set: a test set ratio of 6:2:2, wherein the validation set is used to optimize the model parameters, and the test set is used to analyze the performance of the model.
[0083] Step S2: Based on the constructed prick point recognition network, target detection and single prick point clustering segmentation are performed on the tongue image to be recognized to obtain the corresponding prick point recognition result.
[0084] In one embodiment, step S2:
[0085] Inputting the tongue image to be identified into the prick target detection model for target detection to obtain the corresponding tongue target detection result;
[0086] Each single-point prick image in the tongue target detection result is input into the monomer prick segmentation model to perform single-point prick cluster segmentation to obtain corresponding single-point prick cluster segmentation results.
[0087] In order to better illustrate the above-mentioned tongue image prick recognition method, the present invention provides the following specific embodiments.
[0088] Embodiment 1: A method for spot recognition based on target detection and clustering segmentation.
[0089] The method comprises:
[0090] 1. Data collection;
[0091] The tongue images of patients collected clinically by the company's self-developed closed standard light source tongue imager were screened according to the image requirements of the National Health Tongue Database Basic Database in order to improve the research data quality. Three experienced Chinese medicine practitioners used labelme software to annotate different forms of points on the tongue images, including red, white or black star points protruding from the tongue surface, according to the internal tongue image annotation specifications, to obtain a data set.
[0092] 2. Division of data set;
[0093] The data set is divided into a training set: validation set: test set ratio of 6: 2: 2. The training set and validation set are used to complete network model learning and optimize model parameters, and the test set is used to analyze the performance of the model.
[0094] 3. Construct a prick recognition algorithm based on target detection and clustering segmentation; Fig. 9 A flowchart for constructing a prick recognition algorithm based on target detection and clustering segmentation.
[0095] 3.1 Build target detection model;
[0096] The target detection network improved by the benchmark YoloV5 model is trained with the training set to obtain the target detection model. It is mainly composed of the feature extraction network (Backbone), the feature fusion network (Neck) of the model neck, and the model detection head (Prediction). Through feature extraction and multi-feature fusion of the neck structure composed of FPN and PANet, the target detection frame coordinates and categories are finally obtained through the detection head to achieve specific target detection.
[0097] Among them, SIoU loss is introduced to replace the original GIoU loss. SIoU loss consists of four parts: angle loss (Angle cost), distance loss (Distance cost), shape loss (Shape cost) and IoU loss (IoU cost).
[0098] The model detection head (Prediction) introduces a detection branch specifically for small targets on the original basis, which greatly enhances the model's perception of high-resolution features and makes the perception of fine information such as points and thorns more acute, thereby improving the detection capability. At the same time, because it makes full use of the detail clues provided by the shallow network, the small target detection branch can give more accurate positioning results and improve the accuracy of point selection.
[0099] We used 575 high-quality acupuncture points carefully annotated by professional TCM researchers as test data, and compared the detection scheme in this paper with the current mainstream detectors. Through comparative experiments, we obtained the comparison results shown in Table 1. According to the results in Table 1, our method achieved excellent results on the big data dataset, taking into account the precision rate while ensuring the recall rate.
[0100] Table 1 Comparison results of detection methods
[0101]
[0102] 3.2 Perform color clustering-based pre-labeling on the prick target detection results.
[0103] The specific operations are as follows:
[0104] (1) Convert a single prick image from RGB space to LAB space.
[0105] (2) Take the a* channel from the converted image for histogram equalization and gamma correction.
[0106] (3) Perform K-means clustering on the enhanced images.
[0107] (4) After masking the clustering results, bilateral filtering and corrosion operations are performed.
[0108] The obtained results can be used as qualified segmentation labels after simple manual fine-tuning.
[0109] 3.3. Build a semantic segmentation model;
[0110] The embodiment uses SeaFormer as the segmentation model for a single point, which combines the respective advantages of the CNN structure and the Transformer structure. It can not only quickly generate a local feature pyramid in high-resolution image processing, but also obtain rich semantic information and a huge receptive field. At the same time, the design adds axial compression to enhance attention of position information. On the one hand, the Q, K, and V features are axially compressed and then the attention is enhanced. At the same time, the Q, K, and V features are used to enhance local information using a convolutional network. Finally, the two are fused to output enhanced features.
[0111] 4. Perform tongue image prick recognition based on target detection model and semantic segmentation model.
[0112] Similar to the principle of the above embodiment, the present invention provides a tongue image prick recognition system.
[0113] The following provides specific embodiments in conjunction with the accompanying drawings:
[0114] like Fig.10 A schematic diagram showing the structure of a tongue image prick recognition system in an embodiment of the present invention.
[0115] The system comprises:
[0116] A prick recognition network construction module 21 is used to construct a prick recognition network based on target detection and single prick clustering and segmentation;
[0117] The prick recognition module 22 is connected to the prick recognition network construction module 21 and is used to perform target detection and single prick clustering segmentation on the tongue image to be recognized based on the constructed prick recognition network to obtain corresponding prick recognition results;
[0118] The prick recognition network includes: a cascaded prick target detection model for target detection and a monomer prick segmentation model for single prick cluster segmentation.
[0119] It should be understood that Figure 2 The division of each module in the system embodiment is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And these modules can be all implemented in the form of software calling through processing elements; or all implemented in the form of hardware; or some modules can be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware;
[0120] For example, each module may be one or more integrated circuits configured to implement the above method, such as: one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0121] Since the implementation principle of the tongue image prick recognition system has been described in the above-mentioned embodiment, it will not be repeated here.
[0122] In one embodiment, the construction of a prick recognition network based on target detection and clustering segmentation includes: obtaining multiple target detection tongue image samples obtained by performing prick target annotation based on multiple collected tongue image images; training a target detection network based on each target detection tongue image sample to obtain the prick target detection model; pre-annotating each single prick image in each prick target detection result output by the prick target detection model based on color clustering to obtain a corresponding tongue image monomer segmentation image sample; training a semantic segmentation model based on each tongue image monomer segmentation image sample to obtain the monomer prick segmentation model.
[0123] In one embodiment, the target detection network is obtained by improving the benchmark YoloV5 network; wherein the benchmark YoloV5 network includes: an input network, a feature extraction network, a feature fusion network, and a model detection head network including multiple target detection layers.
[0124] In one embodiment, the target detection network is obtained by adding a target detection layer for shallow feature target detection to the model detection head network on the baseline YoloV5 network.
[0125] In one embodiment, the target detection network is trained using a SIoU loss function to obtain the prick target detection model; wherein the SIoU loss function includes: angle loss, distance loss, shape loss, and IoU loss.
[0126] In one embodiment, the method of pre-labeling each single-point prick image based on color clustering includes: converting each single-point prick image from RGB space to LAB space to obtain a corresponding single-point prick color conversion image; extracting a* channel data from each single-point prick color conversion image, and performing histogram equalization and gamma correction processing to obtain a corresponding single-point prick enhanced image; performing K-means clustering on each single-point prick enhanced image to obtain a single-point prick cluster image; performing mask processing on each single-point prick cluster image, and then performing bilateral filtering and corrosion operations to obtain a corresponding monomer prick segmentation image, so as to obtain a corresponding tongue image monomer segmentation image sample.
[0127] In one embodiment, the semantic segmentation model adopts the SeaFormer network.
[0128] In one embodiment, the method further includes: testing and verifying the constructed acupuncture point recognition network based on a plurality of tongue image images annotated with acupuncture points.
[0129] In one embodiment, the target detection and cluster segmentation of the tongue image to be identified to obtain the corresponding prick recognition result includes: inputting the tongue image to be identified into the prick target detection model for target detection to obtain the corresponding tongue target detection result; inputting each single prick image in the tongue target detection result into the monomer prick segmentation model for single prick cluster segmentation to obtain the corresponding single prick cluster segmentation result.
[0130] like Fig.11 A schematic diagram showing the structure of the tongue image prick recognition terminal 30 in an embodiment of the present invention is shown.
[0131] The tongue image prick recognition terminal 30 includes: a memory 31 and a processor 32. The memory 31 is used to store computer programs; the processor 32 runs the computer programs to achieve the following Figure 1The tongue image prick recognition method.
[0132] Optionally, the number of the memory 31 can be one or more, the number of the processor 32 can be one or more, and Fig.11 Take one as an example.
[0133] Optionally, the processor 32 in the tongue image prick recognition terminal 30 will perform the following steps: Figure 1 In the above steps, one or more instructions corresponding to the process of the application are loaded into the memory 31, and the processor 32 runs the application stored in the first memory 31, thereby achieving the following Figure 1 Various functions in the tongue image prick recognition method.
[0134] Optionally, the memory 31 may include but is not limited to high-speed random access memory and non-volatile memory. For example, one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices; the processor 32 may include but is not limited to a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0135] Optionally, the processor 32 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0136] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed as follows: Figure 1The method for identifying tongue images by prick points is shown. The computer-readable storage medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (Compact Disc Read Only Memory), a magneto-optical disk, a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic card or an optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product that is not connected to a computer device, or may be a component that is connected to a computer device for use.
[0137] In summary, the method, system, terminal and medium for tongue image prick recognition of the present invention construct a prick recognition network based on target detection and single prick clustering segmentation, and perform target detection and single prick clustering segmentation on the tongue image to be identified based on the constructed prick recognition network to obtain the corresponding prick recognition result. The prick recognition network constructed by the present invention has a strong generalization ability, can obtain sufficient fine-grained information and has high accuracy in prick recognition, can assist in outputting prick quantization values and intuitively analyzing the severity of pricks, and has research significance for realizing smart medical care and digitization of traditional Chinese medicine. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.
[0138] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A tongue image prick recognition method, characterized in that: The method comprises: Construct a prick recognition network based on target detection and single prick clustering segmentation; Based on the constructed prick recognition network, target detection and single prick clustering segmentation are performed on the tongue image to be recognized to obtain the corresponding prick recognition results; The prick recognition network includes: a cascaded prick target detection model for target detection and a monomer prick segmentation model for single prick cluster segmentation.
2. The tongue image prick recognition method according to claim 1, characterized in that: The construction of the prick recognition network based on target detection and clustering segmentation includes: Acquire multiple target detection tongue image samples obtained by performing point-puncture target labeling based on the collected multiple tongue image images; Training a target detection network based on each target detection tongue image sample to obtain the prick target detection model; Pre-labeling each single prick image in each prick target detection result output by the prick target detection model based on color clustering to obtain a corresponding tongue image monomer segmentation image sample; The semantic segmentation model is trained based on each tongue image monomer segmentation image sample to obtain the monomer point segmentation model.
3. The tongue image prick recognition method according to claim 2, characterized in that: The target detection network is obtained by improving the benchmark YoloV5 network; wherein the benchmark YoloV5 network includes: an input network, a feature extraction network, a feature fusion network and a model detection head network including multiple target detection layers.
4. The tongue image prick recognition method according to claim 3, characterized in that: The target detection network is obtained by adding a target detection layer for shallow feature target detection to the model detection head network on the baseline YoloV5 network.
5. The tongue image point prick recognition method according to claim 3 or 4, characterized in that: The target detection network is trained using the SIoU loss function to obtain the prick target detection model; wherein the SIoU loss function includes: angle loss, distance loss, shape loss and IoU loss.
6. The tongue image prick recognition method according to claim 2, characterized in that: The method of pre-labeling each single-point spike image based on color clustering includes: Convert each single-point spike image from RGB space to LAB space to obtain the corresponding single-point spike color conversion image; Extract a* channel data from each single-point thorn color conversion image, and perform histogram equalization and gamma correction processing to obtain the corresponding single-point thorn enhanced image; Perform K-means clustering on each single-point spike-enhanced image to obtain a single-point spike clustered image; Mask processing is performed on each single-point prick cluster image, and then bilateral filtering and corrosion operations are performed to obtain the corresponding monomer prick segmentation image, so as to obtain the corresponding tongue image monomer segmentation image sample.
7. The tongue image prick recognition method according to claim 2, characterized in that: The semantic segmentation model adopts the SeaFormer network.
8. The tongue image prick recognition method according to claim 2, characterized in that: The method further includes: testing and verifying the constructed acupuncture point recognition network based on a plurality of tongue image images annotated with acupuncture points.
9. The tongue image prick recognition method according to claim 1, characterized in that: The target detection and cluster segmentation of the tongue image to be identified to obtain the corresponding prick recognition result includes: Inputting the tongue image to be identified into the prick target detection model for target detection to obtain the corresponding tongue target detection result; Each single-point prick image in the tongue target detection result is respectively input into the monomer prick segmentation model for single-point prick cluster segmentation to obtain the corresponding single-point prick cluster segmentation result.
10. A tongue image prick recognition system, characterized in that: The system comprises: The prick recognition network construction module is used to build a prick recognition network based on target detection and single prick clustering segmentation; A prick recognition module, connected to the prick recognition network construction module, is used to perform target detection and single prick clustering segmentation on the tongue image to be recognized based on the constructed prick recognition network to obtain corresponding prick recognition results; The prick recognition network includes: a cascaded prick target detection model for target detection and a monomer prick segmentation model for single prick cluster segmentation.
11. A tongue image prick recognition terminal, characterized in that: include: one or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors, connected to the memory, are configured to run the computer program to perform the method according to claims 1 to 9.
12. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 9 is performed.
