Intelligent tongue picture processing method and system based on mobile phone photographing

Abstract

The invention provides an intelligent tongue picture processing method and system based on mobile phone photographing. The method comprises the following steps: acquiring an original image containing a tongue picture, which is photographed and acquired by using mobile phone equipment under the conditions that a special colorimetric card is placed at a specific position of a face and a subject stretches a tongue; the original image is preprocessed; cutting the preprocessed image by using a cutting model pre-trained based on deep learning to obtain a corresponding tongue image; and inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning to obtain an attribute tag as output. According to the invention, the color difference between different ambient light and shooting equipment can be overcome in the conventional mobile phone equipment and shooting environment, a standard and objective tongue picture intelligent processing method based on mobile phone shooting is provided, and objective and quantitative acquisition of tongue picture characteristics can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning image processing, and in particular to a tongue image intelligent processing method and system based on mobile phone photography. Background Art

[0002] Tongue is one of the important observation objects in traditional Chinese medicine and is widely used to assess the health status of individuals. By observing the shape and color of the tongue, doctors can understand the functions of internal organs and overall health status. However, traditional tongue examination often relies on the doctor's experience and subjective judgment, and the diagnosis results are easily affected by the doctor's knowledge reserves and the ambient lighting conditions. This subjectivity not only affects the consistency of clinical diagnosis, but also adds difficulties to the subsequent efficacy evaluation.

[0003] With the rapid development of computer image processing technology, more and more research is devoted to standardizing and objectifying tongue images. For example, tongue images are collected using photographic equipment, and color information of the tongue is obtained based on RGB color space, HSV color model, etc., and accurate numerical analysis of the color information is performed. At the same time, intelligent algorithms such as deep learning can be combined to extract tongue features such as tooth marks, cracks, and punctures.

[0004] However, the above methods usually involve complex color mapping and nonlinear relationships when used for color correction. Unsupervised learning cannot effectively capture these complex relationships, which leads to unsatisfactory correction results. For example, the color of the tongue image will have color differences due to different lighting and shooting equipment, making it difficult to obtain the objective value of the real color.

[0005] In addition, in the existing methods of extracting tongue features by combining intelligent algorithms such as deep learning, the length and size of tongue features such as tooth marks, cracks and punctures cannot be quantitatively measured due to the lack of reference objects.

[0006] Therefore, how to overcome the color difference of different ambient lights and shooting equipment in conventional mobile phone devices and photo-taking environments, and provide a method for judging tongue images based on tongue images that maintains standardization and objectivity, is a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a method and system for intelligently processing tongue images based on mobile phone photography, so as to eliminate or improve one or more defects existing in the prior art.

[0008] One aspect of the present invention provides a method for intelligently processing tongue images based on mobile phone photography, the method comprising the following steps: obtaining an original image containing a tongue image captured by taking a photo with a mobile phone device under the condition that a special colorimetric card is placed at a specific position on the face and the subject sticks out the tongue; wherein the special colorimetric card comprises identification codes at four corner points and a plurality of color blocks of preset specifications; preprocessing the original image; wherein the preprocessing step at least comprises: determining the position of the special colorimetric card based on the identification codes contained in the special colorimetric card, performing deformation correction on the original image based on the square area surrounded by the identification codes at the four corner points, and using the plurality of color blocks of preset specifications to correct the deformation of the original image. The color block performs color correction on the deformation-corrected image; a cutting model pre-trained based on deep learning is used to cut the corresponding tongue image from the pre-processed image; wherein the cutting model is trained with a pre-processed image of a preset scale and a tongue contour image with manually drawn contours as a training set; the cut tongue image is input into a classification model pre-trained based on deep learning to obtain an attribute label as an output; wherein the classification model is trained with a tongue image of a preset scale and manually annotated attribute labels as a training set, and the types of attribute labels include one or more of tongue width, tongue length, tongue area, tongue color, tongue coating color and constitution type.

[0009] In some embodiments of the present invention, the step of preprocessing the original image also includes: after determining the position of the special colorimetric card based on the identification code contained in the special colorimetric card, performing color card data identification on the special colorimetric card, judging the light environment for capturing the original image according to preset standards, and if it does not meet the standards, providing feedback to adjust the light environment and recapture the original image; wherein the dimensions of the preset standards include multiple ones of uniformity, brightness, illuminance and light distribution.

[0010] In some embodiments of the present invention, the method further includes: after obtaining a tongue image by cutting the tongue image from the preprocessed image using the cutting model, determining whether the tongue image exceeds the recognition area; if the tongue image exceeds the recognition area, providing feedback to adjust the position and recapture the original image.

[0011] In some embodiments of the present invention, the type of attribute label also includes tongue image; the step of inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning to obtain the attribute label as output includes: inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning, the classification model extracting the features contained in the tongue image, and calculating the corresponding attribute label as output.

[0012] In some embodiments of the present invention, the overall structure of the special colorimetric card is an "concave"-shaped structure, and the multiple color blocks of preset specifications are arranged on the side arms. When taking pictures to collect the original image, the subject sticks out his tongue so that the tongue is located between the two arms without obstruction; the step of performing deformation correction on the original image based on the square area surrounded by the identification codes located at the four corner points includes: performing deformation correction on the original image using an affine transformation method to find the position and data of each color block on the special colorimetric card; the step of performing color correction on the image after deformation correction using the multiple color blocks of preset specifications includes: performing color correction on the image after deformation correction using the multiple color blocks of preset specifications using a 3*3 matrix correction and a Gamma correction method; the step of preprocessing the original image also includes: removing noise, enhancing contrast, and adjusting white balance.

[0013] In some embodiments of the present invention, the method also includes a step of pre-training a cutting model based on deep learning, including: obtaining an untrained image cutting model, and a training set consisting of pre-processed images of a preset scale and tongue contour images with manually drawn contours; training the image cutting model using the training set based on deep learning model training technology to obtain a trained cutting model for segmenting the tongue image.

[0014] In some embodiments of the present invention, the method also includes a step of pre-training a classification model based on deep learning, including: obtaining an untrained classification model and a training set consisting of tongue images of a preset size and manually annotated attribute labels; training the classification model using the training set based on deep learning model training technology to obtain a trained classification model for judging attribute label information based on tongue images.

[0015] Corresponding to the above method, the present invention also provides an intelligent tongue image processing system based on mobile phone photography, including a processor, a memory and a computer program / instructions stored in the memory, the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described in any one of the above embodiments.

[0016] Corresponding to the above method, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0017] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0018] The method and system for intelligent processing of tongue images based on mobile phone photography proposed in the present invention can use the tongue images collected by mobile phone photography to perform intelligent processing on the tongue images, including preprocessing steps such as deformation correction and color correction, so as to realize the standardization of the collected images, and then use the pre-trained cutting model to cut out the tongue images from the collected images, and use the pre-trained classification model to analyze the tongue images to obtain attribute labels. Furthermore, when the attribute label is the tongue image result of TCM visual diagnosis, intelligent visual diagnosis based on mobile phone photography can be realized. In summary, the present invention proposes a standardized, objective, and easy-to-operate intelligent tool system that combines traditional TCM tongue image diagnosis with modern image processing and recognition technology, which can realize the objective reading of tongue image characteristics. This method and system not only lays a solid foundation for the objective diagnosis of tongue images, but also provides a new idea for the early identification of diseases, the evaluation of treatment effects, and the innovation of health management models, which is helpful to promote the objectification process of TCM.

[0019] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0020] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0022] Figure 1 This is a flow chart of a method for intelligently processing tongue images based on mobile phone photography in one embodiment of the present invention.

[0023] Figure 2 The figure is a schematic diagram of the structure of a handheld special colorimetric card in one embodiment of the present invention.

[0024] Figure 3 This is a flowchart of the intelligent processing of tongue images based on mobile phone photography in another embodiment of the present invention.

[0025] Figure 4 This is a photo taken with a mobile phone holding a special color card in one embodiment of the present invention.

[0026] Figure 5 This is a tongue image region diagram cut out after affine transformation of the recognition color card in one embodiment of the present invention.

[0027] Figure 6 FIG. 4 is an image after color correction in one embodiment of the present invention.

[0028] Figure 7 This is a diagram of the tongue position after image cutting in one embodiment of the present invention.

[0029] Figure 8 This is a tongue image after image cutting in one embodiment of the present invention.

[0030] Fig. 9A This is a cutting database of the tongue image in one embodiment of the present invention - a preprocessed image.

[0031] Fig. 9B This is a tongue image cutting database in one embodiment of the present invention - a tongue image position map after image cutting.

[0032] Fig. 9C This is a tongue image cutting database in one embodiment of the present invention - a tongue image after image cutting. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0034] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0035] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0036] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0037] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0038] In order to overcome the problems existing in the prior art, the present invention proposes an intelligent visual diagnosis method and system based on mobile phone photography. The method is based on an independently designed special colorimetric card that can be used for color correction and quantitative measurement of tongue characteristics, and is used for traditional Chinese medicine visual diagnosis and / or body surface visual diagnosis.

[0039] Figure 1 This is a flow chart of a tongue image intelligent processing method based on mobile phone photography in one embodiment of the present invention, the method comprises the following steps:

[0040] Step S110: obtaining an original image containing the tongue image by taking a photo with a mobile phone device under the condition that a special color comparison card is placed at a specific position on the face and the subject sticks out the tongue; wherein the special color comparison card includes identification codes at four corner points and multiple color blocks of preset specifications.

[0041] Step S120: preprocessing the original image; wherein the preprocessing step at least includes: determining the position of the special colorimetric card based on the identification code contained in the special colorimetric card, performing deformation correction on the original image based on the square area surrounded by the identification codes located at the four corner points, and performing color correction on the deformation-corrected image using the multiple color blocks of preset specifications.

[0042] In some embodiments of the present invention, since the specifications of the color block are known preset values, such as 5mm*5mm, the color block can be used as a reference to quantitatively measure the length and size of tongue features (such as tooth marks, cracks, and punctures, etc.) based on the known color block specifications.

[0043] Step S130: using a cutting model pre-trained based on deep learning to cut a corresponding tongue image from the pre-processed image; wherein the cutting model is trained using a pre-processed image of a preset scale and a tongue contour image with manually drawn contours as a training set.

[0044] Step S140: Input the tongue image obtained by cutting into a classification model pre-trained based on deep learning to obtain an attribute label as an output; wherein the classification model is trained with a tongue image of a preset scale and manually annotated attribute labels as a training set, and the types of attribute labels include one or more of tongue width, tongue length, tongue area, tongue color, tongue coating color and physical constitution type.

[0045] The method for intelligent processing of tongue images based on mobile phone photography proposed in the present invention can utilize the tongue images collected by mobile phone photography to perform intelligent processing on the tongue images, including preprocessing steps such as deformation correction and color correction, so as to realize the standardization of the collected images, and then use the pre-trained cutting model to cut out the tongue images from the collected images, and use the pre-trained classification model to analyze the tongue images to obtain attribute labels. In summary, the present invention proposes a standardized, objective, and easy-to-operate intelligent tool system that combines traditional Chinese medicine tongue image diagnosis with modern image processing and recognition technology, which can realize the objective reading of tongue image characteristics. This method and system not only lay a solid foundation for the objective diagnosis of tongue images, but also provide a new idea for the early identification of diseases, the evaluation of treatment effects, and the innovation of health management models, which is helpful to promote the objectification process of Chinese medicine.

[0046] In some embodiments of the present invention, step S120 also includes: after determining the position of the special colorimetric card based on the identification code contained in the special colorimetric card, performing color card data identification on the special colorimetric card, judging the light environment for capturing the original image according to preset standards, and if it does not meet the standards, providing feedback to adjust the light environment and recapture the original image; wherein the dimensions of the preset standards include uniformity, brightness, illuminance, and multiple types of light distribution.

[0047] By adopting the embodiment of the invention, a basic evaluation can be made from the perspective of the light environment of the mobile phone camera. If the image quality of the mobile phone camera is obviously poor, the image is screened out and the subject is notified to re-capture the image with the mobile phone, thereby ensuring the credibility of the final intelligent visual diagnosis result.

[0048] In some embodiments of the present invention, after obtaining a tongue image by cutting the preprocessed image using the cutting model, the method further includes: determining whether the tongue image exceeds the recognition area, and if the tongue image exceeds the recognition area, providing feedback to adjust the position and re-capture the original image. In the specific implementation process, this step can be used to determine whether the collected tongue image is blocked.

[0049] By adopting this embodiment of the invention, it can be ensured that the tongue inspection result is obtained based on a complete tongue image.

[0050] In a specific embodiment of the present invention, the type of attribute tag further includes tongue image.

[0051] Accordingly, the step of inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning to obtain an attribute label as an output includes: inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning, the classification model extracting the features contained in the tongue image, and calculating the corresponding attribute label as an output.

[0052] By adopting the embodiment of the invention, a classification model can be used to diagnose tongue images, thereby realizing intelligent visual diagnosis based on images collected by mobile phone photography.

[0053] In some embodiments of the present invention, the overall structure of the special color comparison card is a "concave"-shaped structure, and the plurality of color blocks of preset specifications are arranged on the side arms, and when taking photos to collect the original image, the subject extends the tongue so that the tongue is located between the two arms without obstruction. For example, Figure 2 FIG. 2 is a schematic diagram of a handheld colorimetric card according to an embodiment of the present invention, wherein the identification codes 210 are located at four corners, namely Figure 2 A, B, C, D, and color blocks 220 are distributed on both sides of the color chart.

[0054] Accordingly, in a specific implementation process, the step of performing deformation correction on the original image based on the square area surrounded by the identification codes at the four corner points includes: performing deformation correction on the original image by using an affine transformation method to find the position and data of each color block on the special colorimetric card.

[0055] Accordingly, in the specific implementation process, the step of performing color correction on the image after deformation correction using the color blocks of the preset specifications includes: adopting 3*3 matrix correction and Gamma correction method, and performing color correction on the image after deformation correction using the color blocks of the preset specifications.

[0056] In some other embodiments of the present invention, the step of preprocessing the original image further includes: removing noise, enhancing contrast and adjusting white balance. The present invention is not limited to this. The above embodiments are preferred embodiments and can also be replaced by: (1) Image preprocessing: including steps such as removing noise, enhancing contrast, and white balance to improve the recognizability of tongue images. Edge detection: using methods such as Canny edge detection and Sobel operator to extract the outline of the tongue. Feature extraction: extracting key features of the tongue image, such as tongue quality, tongue coating, color, etc. (2) Machine learning: ① Support vector machine (SVM): can be used to classify different states of tongue images and learn the relationship between features and health status through training data sets. Decision tree and random forest: classify tongue images by building a tree structure, which is suitable for processing complex features. ② KNN (K-nearest neighbor): classification is performed by measuring the distance between samples, which is suitable for judging the similarity of tongue image features. (3) Deep learning: ① Convolutional neural network (CNN): automatically learns features from the original image through a deep network, which is suitable for the classification and recognition of tongue images; ② Generative adversarial network (GAN): can be used to generate tongue images and enhance and transform tongue image features.

[0057] By adopting the above-mentioned embodiments of the invention, the accuracy, consistency and standardization of the images used for tongue contour recognition can be guaranteed to the greatest extent, thereby ensuring the standardization and consistency of the intelligent visual diagnosis results.

[0058] In some embodiments of the present invention, the method also includes a step of pre-training a cutting model based on deep learning, including: (1) obtaining an untrained image cutting model and a training set consisting of pre-processed images of a preset size and tongue contour images with manually drawn contours; (2) training the image cutting model using the training set based on deep learning model training technology to obtain a trained cutting model for segmenting the tongue image.

[0059] By adopting the embodiment of the invention, a cutting model for cutting tongue contour diagram can be trained, thereby making preliminary preparations for subsequent intelligent visual diagnosis.

[0060] In some embodiments of the present invention, the method also includes a step of pre-training a classification model based on deep learning, including: (1) obtaining an untrained classification model and a training set consisting of tongue images of a preset size and manually annotated attribute labels; (2) training the classification model using the training set based on deep learning model training technology to obtain a trained classification model for judging attribute label information based on tongue images.

[0061] By adopting this embodiment of the invention, a classification model for the inspection results of intelligent inspection to obtain the attribute labels of the subjects (including tongue image information) can be trained, thereby realizing the inspection function of the present invention.

[0062] Figure 3 FIG. 1 is a flowchart of the intelligent processing of tongue images based on mobile phone photography in another embodiment of the present invention. Figure 3 As shown, in the specific implementation process, the method includes the following steps:

[0063] (1) The subject holds a special color card and places it at a designated position on the face. The subject sticks out his tongue and uses a mobile phone to take a photo to obtain the original image. Then the QR code of the four corner points of the color card in the image is identified, and the position of the color card is determined by color card position recognition. The color card and hue are corrected to the correct position through the affine transformation algorithm. Then, based on the relative position of each color on the color card and the four corner points, the position and data of each color on the color card are found and identified.

[0064] (2) Compare the standard data of the color card with the actual shooting data, use 3*3 matrix correction and Gamma correction algorithm to correct the output image, and use the real data of the color card to correct the color of the color card shooting data based on the color correction algorithm. The principle formula of image correction is as follows:

[0065]

[0066] Among them, R, G, and B in the above formula are the color data after matrix correction. ij is the element of the correction matrix A. b R , b G , b B is a constant term. i , G i , B i is the color data of each pixel in the input image. o , G o , B o is the color data of each pixel in the output image. r , g g , g b is the gamma correction factor.

[0067] The RGB value of each color block of the input image color card is used as R i , G i , B i , color card standard value as R o , G o , B o , use the least squares method to perform iterative optimization and calculate a ij , b R , b G , b B , g r , g g , g b And other parameters, and then apply the above formula to each pixel of the entire image to obtain the corrected image.

[0068] (3) Accumulate data. Cut the tongue image and annotate the attribute information (tongue color, tongue coating color, physical constitution type, etc.) of the color-corrected image obtained by the above steps (1) and (2) by manually drawing the outline. After data accumulation, an image database for deep model training and the corresponding tongue image cutting diagram and attribute annotation information database are obtained. The image database is used to train the tongue image cutting model, and the attribute annotation information database is used to train the classification model for attribute label classification.

[0069] (4) Based on the database established in (3) above, a tongue image cutting model and an attribute label classification model are established and trained respectively through a deep learning algorithm, wherein the tongue image cutting model is used for tongue image contour recognition and cutting, and the attribute label classification model is trained using manually annotated attribute labels and tongue images as training sets.

[0070] (5) In practical application, the image acquisition method is the same as (1), and the color correction method is the same as (2). Then, the color-corrected image is cut through the tongue image model to obtain the tongue image contour (in addition, it can include the step of super-boundary judgment, that is, if the tongue image is partly at the boundary of the recognition area, the user is prompted to adjust the position and take a new photo). According to the tongue image contour, the image information outside the tongue image is removed, and then the tongue color, tongue coating color, and physical type of the tongue image are obtained through the classification model. Among them, the above attribute information is only an example, and the attribute label can be selectively annotated in the attribute annotation information database according to the actual situation.

[0071] In another specific implementation process of the present invention, the specific implementation process is described as follows in conjunction with the actual drawings:

[0072] (1) Hold the special color card, place it on the designated position of the face, stick out the tongue, and take a photo with a mobile phone. The designated position of the face is to put the upper part of the color card on the face of the person, and the color card body does not cover the tongue image. Figure 4 This is a photo taken with a mobile phone holding a special color card in one embodiment of the present invention.

[0073] (2) Then identify the QR codes of the four corner points of the color card in the image, determine the position of the color card, and use the affine transformation algorithm to perform deformation correction on the square area surrounded by the four corner points of the color card. Then, based on the relative positions of the colors on the color card and the four corner points, find and identify the positions and data of each color on the color card. Figure 5 This is a tongue image region diagram cut out after affine transformation of the identification color card in one embodiment of the present invention. In the figure, the green area is the location of the identified tongue image.

[0074] (3) Compare the standard data of the color card with the actual shooting data, and use the 3*3 matrix correction and gamma correction algorithm to correct the output image. For example, Figure 6 is an image after color correction in one embodiment of the present invention. Figure 4 and Figure 6 By comparing the colors, you can see the difference before and after color correction.

[0075] (4) Manually cut the tongue image and annotate the attribute information (tongue color, coating color, constitution type, etc.) to obtain an image database and a corresponding tongue image cutting diagram and attribute annotation information database. Figure 7 This is a tongue image position diagram after image cutting in one embodiment of the present invention. Figure 8 This is a tongue image after image cutting in one embodiment of the present invention. In the process of obtaining the training set, it is necessary to manually draw the contour of the tongue image for cutting. The goal of training the cutting model with the training set is to enable the cutting model to recognize the contour boundary of the tongue image from the image, and to use deep learning ideas for training, thereby realizing automatic cutting based on the trained cutting model.

[0076] (5) The color-corrected image is trained through deep learning for image segmentation and segmentation model training to obtain a tongue image segmentation model. This segmentation model is then used to segment the image database to obtain a pure tongue image database. Combined with the tongue image attribute annotation, the tongue image attributes are classified and trained to obtain the classification models for each attribute of the tongue image.

[0077] (6) In actual application, the image acquisition method is the same as (1), and the color correction method is the same as (2). Then, the color-corrected image is cut through the tongue image model to obtain the tongue image contour (if part of the tongue image is at the boundary of the recognition area, the user is prompted to adjust the position and take a new photo). According to the tongue image contour, the image information outside the tongue image is removed, and then the tongue color, tongue coating color, physical constitution type and other attribute information of the tongue image are obtained through the classification model.

[0078] In an embodiment of the present invention, Fig. 9A The cutting database of the tongue image in one embodiment of the present invention - the preprocessed image, Fig. 9B The tongue image cutting database in one embodiment of the present invention - the tongue image position map after image cutting, Fig. 9C This is a tongue image cutting database in one embodiment of the present invention - a tongue image after image cutting. Figures 9A-9C This is an example of an image database used to train a tongue image cutting model. The selection angles of the attribute data of the tongue image include but are not limited to: (1) age, divided into tender tongue and old tongue; (2) tongue size, divided into fat tongue, thin tongue, normal tongue, etc.; (3) pricked tongue, divided into red dot, no pricked tongue, etc.; (4) tooth-marked tongue, non-tooth-marked tongue, etc.; (5) cracked tongue, non-cracked tongue, etc.; (6) whether there are bruises, petechiae, ecchymosis, etc.; (7) whether there are sores, etc.

[0079] The method and system for intelligent processing of tongue images based on mobile phone photography proposed in the present invention can use the tongue images collected by mobile phone photography to perform intelligent processing on the tongue images, including preprocessing steps such as deformation correction and color correction, so as to realize the standardization of the collected images, and then use the pre-trained cutting model to cut out the tongue images from the collected images, and use the pre-trained classification model to analyze the tongue images to obtain attribute labels. Furthermore, when the attribute label is the tongue image result of TCM visual diagnosis, intelligent visual diagnosis based on mobile phone photography can be realized. In summary, the present invention proposes a standardized, objective, and easy-to-operate intelligent tool system that combines traditional TCM tongue image diagnosis with modern image processing and recognition technology, which can realize the objective reading of tongue image characteristics. This method and system not only lays a solid foundation for the objective diagnosis of tongue images, but also provides a new idea for the early identification of diseases, the evaluation of treatment effects, and the innovation of health management models, which is helpful to promote the objectification process of TCM.

[0080] Specifically, the advantages of the method and system proposed by the present invention include but are not limited to:

[0081] (1) The special color comparison card has an overall "concave" shape, and the side arms are provided with color blocks for color correction. This design structure allows the tongue to be placed directly in the middle blank area for comparison, which can overcome the differences between different photographic equipment and different photographic environments, and ensure the efficiency of color comparison of tongue images and the consistency and accuracy of color detection.

[0082] (2) The special colorimetric card adopts a fixed size, so that the specific size of features such as tooth marks, cracks and punctures can be systematically measured through comparative analysis. This quantitative method provides an objective diagnostic method for tongue feature identification, effectively reducing the error caused by subjectivity in traditional methods. At the same time, the use of the special colorimetric card is simple and intuitive, which can improve the patient's sense of participation and enhance their attention to and management of their own health, thereby promoting the improvement of overall health level.

[0083] (3) Use supervised learning algorithms to achieve tongue color correction. Compared with unsupervised learning algorithms, supervised learning algorithms are more capable of learning the changing patterns of various factors such as illumination changes, device differences, and color space conversion during the color correction process. They are suitable for shooting with mobile devices in open environments where the equipment and light sources are uncontrollable.

[0084] (4) Based on a special colorimetric card, a 3*3 matrix correction and a gamma correction algorithm are used to achieve color correction of the acquired image, which can realize standard and objective image acquisition for TCM visual diagnosis, and then perform standardized intelligent visual diagnosis recognition based on the corrected image.

[0085] (5) By cutting the image and using the pre-labeled tongue image cutting database, intelligent recognition of the tongue image is achieved, and information interference of the non-tongue image part is removed.

[0086] (6) Use deep learning technology to obtain the tongue width, tongue length, tongue area, tongue color, tongue coating color, physical constitution type and other visual diagnosis-related attribute information of the tongue image, and perform calculations based on the above visual diagnosis attribute information, as well as the tongue position, light uniformity, whether there is reflection and other related attributes to assist in correct photography, so as to achieve intelligent visual diagnosis based on mobile phone photography. Use the manually annotated database as a training set to train the deep learning model, and use the trained deep learning model for intelligent visual diagnosis.

[0087] (7) The solution proposed by the present invention relies on the widely popular smart phone, which is low-cost and can facilitate citizens to diagnose tongue images through telemedicine without having to travel. This not only facilitates patients to conduct self-examination at home, but also lowers the threshold for medical treatment. For patients in remote areas, it can effectively alleviate the problem of insufficient medical resources.

[0088] Among them, the supervised learning algorithm refers to using standard color cards or reference color plates when shooting images, learning with these color blocks under ambient light sources, and obtaining color conversion relationships. [1] In the process of image acquisition, the researchers used standard color cards or self-made color scales as references and used topological segmentation-reduction algorithms to [2] , polynomial regression algorithm [3], combination of white balance and polynomial regression [4], back propagation neural network algorithm and gray wolf optimization algorithm optimize the tongue color correction algorithm of the traditional back propagation neural network algorithm [5] to map the color value of the color card to the color value under standard lighting to achieve color correction. In supervised learning, the supervised color plate provides a known color standard, which enables the model to understand the relationship between the color observed under different lighting conditions or equipment and the standard color. This helps to correct the color deviation in the image. The more accurate the color gamut range of the supervised color plate, the better the image correction effect. The supervised color plate used for image color correction in the present invention is a color comparison card for traditional Chinese medicine and body surface diagnosis [6], which has an accurate color gamut range and greatly improves the color correction effect. References (such as patents / papers / standards):

[0089] [1] Liu Qi, Huang Xiaoyang, Wang Boliang, et al. Color cast detection and color correction method of tongue diagnosis images in natural environment [J]. Journal of Xiamen University (Natural Science Edition), 2016, 55(2): 278-284.

[0090] [2] Xu Jiatuo, Tu Liping, Zhang Zhifeng, et al. A color correction method for medical true color images based on supervised color principle [J]. Journal of Biomedical Engineering, 2010, 27(4): 721-726.

[0091] [3]Hu M, Lan K, Fang W, et al. Automated tongue diagnosis on the smartphone and its applications [J]. Computer Methods and Programs inBiomedicine, 2019, 174: 51~64.

[0092] [4] Hu Zhiyang, Tan Jiayuan, Liu Guohua. A TCM image correction algorithm based on white balance and polynomial regression[J]. Journal of Nankai University (Natural Science Edition), 2022, 55(4): 25-28, 37.

[0093] [5] Wang Jishi, Yang Junhan, Zhang Shiqi, et al. Application of GWO-BP neural network in tongue diagnosis image color correction[J]. Information on Traditional Chinese Medicine, 2024, 41(5): 31-36.

[0094] [6] Wang Junwen, Song Xiaohui, Guo Zhuang, et al. A colorimetric card: 2023.3.21, 202223093752.6[P].

[0095] Corresponding to the above method, the present invention also provides a tongue image intelligent processing system based on mobile phone photography, the system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0096] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0097] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0098] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0099] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tongue image intelligent processing method based on mobile phone photography, characterized in that: The method comprises the following steps: Acquire an original image containing a tongue image, which is taken by a mobile phone device under the condition that a special color comparison card is placed at a specific position on the face and the subject sticks out the tongue; wherein the special color comparison card includes identification codes at four corner points and a plurality of color blocks of preset specifications; Preprocessing the original image; wherein the preprocessing step at least includes: determining the position of the special colorimetric card based on the identification code contained in the special colorimetric card, performing deformation correction on the original image based on the square area surrounded by the identification codes located at the four corner points, and performing color correction on the image after deformation correction using the plurality of color blocks of preset specifications; A cutting model pre-trained based on deep learning is used to cut the pre-processed image to obtain a corresponding tongue image; wherein the cutting model is trained using a pre-processed image of a preset scale and a tongue contour image with a manually drawn contour as a training set; The tongue image obtained by cutting is input into a classification model pre-trained based on deep learning to obtain an attribute label as an output; wherein the classification model is trained using a tongue image of a preset scale and manually annotated attribute labels as a training set, and the types of attribute labels include one or more of tongue width, tongue length, tongue area, tongue color, tongue coating color and constitution type.

2. The method according to claim 1, characterized in that The step of preprocessing the original image also includes: after determining the position of the special colorimetric card based on the identification code contained in the special colorimetric card, performing color card data identification on the special colorimetric card, judging the light environment for capturing the original image according to preset standards, and if it does not meet the standards, providing feedback to adjust the light environment and recapture the original image; wherein the dimensions of the preset standards include uniformity, brightness, illuminance and light distribution.

3. The method according to claim 1, characterized in that The method further comprises: after obtaining a tongue image by cutting the preprocessed image using the cutting model, judging whether the tongue image exceeds the recognition area, and if the tongue image exceeds the recognition area, providing feedback to adjust the position and re-capture the original image.

4. The method according to claim 1, characterized in that: The type of attribute tag also includes tongue image; The step of inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning to obtain an attribute label as an output includes: inputting the tongue image obtained by cutting into a classification model pre-trained based on deep learning, the classification model extracting features contained in the tongue image, and calculating corresponding attribute labels as output.

5. The method according to claim 1, characterized in that The overall structure of the special color comparison card is a "concave"-shaped structure, and the plurality of color blocks of preset specifications are arranged on the side arms. When taking photos to collect the original image, the subject extends the tongue so that the tongue is located between the two arms without obstruction; The step of performing deformation correction on the original image based on the square area surrounded by the identification codes at the four corner points includes: performing deformation correction on the original image by using an affine transformation method to find the position and data of each color block on the special colorimetric card; The step of using the plurality of color blocks of preset specifications to perform color correction on the image after deformation correction comprises: using a 3*3 matrix correction and a gamma correction method to perform color correction on the image after deformation correction using the plurality of color blocks of preset specifications; The step of preprocessing the original image also includes: removing noise, enhancing contrast and adjusting white balance.

6. The method according to claim 1, characterized in that The method further includes the step of pre-training a cutting model based on deep learning, including: Obtaining an untrained image segmentation model and a training set consisting of pre-processed images of a preset size and manually outlined tongue contours; Based on the deep learning model training technology, the image segmentation model is trained using the training set to obtain a trained segmentation model for segmenting the tongue image.

7. The method according to claim 1, characterized in that The method also includes the step of pre-training a classification model based on deep learning, including: Obtain an untrained classification model and a training set consisting of tongue images of a preset size and manually annotated attribute labels; The classification model is trained using a training set based on deep learning model training technology to obtain a trained classification model for determining attribute label information based on tongue images.

8. A tongue image intelligent processing system based on mobile phone photography, comprising a processor, a memory and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images