Indentation classification method and device based on target detection and fine-grained classification, terminal and medium

Abstract

The invention provides a tooth mark classification method and device based on target detection and fine-grained classification, a terminal and a medium, and the method comprises the steps: determining the position of a tooth mark on a to-be-classified tongue image through a constructed tooth mark detection model, and inputting an image corresponding to each tooth mark obtained according to the position of the tooth mark into a constructed tooth mark classification model. And obtaining a tooth mark classification result corresponding to each tooth mark. According to the indentation classification method based on target detection and fine-grained classification, indentation positioning can be more accurate, and the classification accuracy is higher.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a tooth mark classification method, device, terminal and medium based on target detection and fine-grained classification. Background Art

[0002] The classification of tooth marks in traditional Chinese medicine is a method of judging the health status of the human body by observing the tooth marks on the tongue. Different types of tooth marks reflect different organ functions and pathological changes, which can provide important reference for diagnosis and treatment in traditional Chinese medicine.

[0003] With the development of modernization of traditional Chinese medicine, the demand and application of digitalization of traditional Chinese medicine are becoming more and more extensive. At present, there are several methods for tooth mark classification: Yang et al. proposed a method for classifying the severity of tooth mark tongue based on morphological feature extraction. The method first uses the MTCNN network to locate the face, then uses the Grabcut algorithm to segment the tongue, and finally uses the Douglas-Peucker algorithm to calculate the number of tooth marks. However, due to the use of the tongue segmentation algorithm, the tongue needs to be segmented very finely, which easily causes the loss of teeth mark edge pixels. Li et al. proposed the recognition of tooth mark tongue based on edge features and tongue color features, but this method needs to adjust parameters according to the data set, and the process is complicated. Gong Ke proposed a method for classifying tooth marks based on the edge features, tongue edge color change features and tongue color features of tooth mark tongue, but this method is easily affected by color and noise. Yan et al. proposed a method for classifying tooth mark tongue based on deep learning and random forest. This method first detects the tooth mark position through the target detection model, and then classifies the tooth marks through random forest. However, this method is easily affected by noise interference and causes overfitting. Rui et al. proposed using the Mask Scoring R-CNN method to segment and classify tooth marks. However, this method requires a lot of manpower to annotate tooth marks at the pixel level. Summary of the invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a tooth mark classification method, device, terminal and medium based on target detection and fine-grained classification, so as to solve the problems of inaccurate tooth mark positioning and low classification accuracy in the prior art.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a tooth mark classification method based on target detection and fine-grained classification, including: preprocessing the collected tongue image to be classified; based on the constructed tooth mark detection model, obtaining one or more tooth mark detection frames corresponding to each tooth mark according to the preprocessed tongue image to be classified; screening the optimal detection frame for each tooth mark, and performing tooth mark detection judgment on the optimal detection frame; based on the constructed tooth mark classification model, obtaining the tooth mark classification result corresponding to each tooth mark according to the optimal detection frame judged by tooth mark detection.

[0006] In some embodiments of the first aspect of the present application, the method for training the tooth mark detection model includes: marking the positions of the tooth marks on a plurality of preprocessed tongue images to obtain a target detection data set; inputting the target detection data set into an improved benchmark YOLOv7 algorithm model for training to obtain the tooth mark detection model; wherein the benchmark YOLOv7 algorithm model includes: an input end, a benchmark network, and a head output layer with a pooling layer.

[0007] In some embodiments of the first aspect of the present application, the method for obtaining the improved baseline YOLOv7 algorithm model includes: replacing the pooling layer in the head output layer in the baseline YOLOv7 algorithm model with a DCBP structure; wherein the DCBP structure includes: a 3×3 hole convolution layer, a 3×3 standard convolution layer, a batch normalization layer, an activation function layer, and a maximum pooling layer.

[0008] In some embodiments of the first aspect of the present application, the preprocessing method includes: detecting the tongue position area of ​​the tongue image and cropping the tongue position area that has been enlarged by a set magnification factor to obtain a tongue image; adjusting the pixel size and grayscale processing of the tongue image to obtain a preprocessed tongue image.

[0009] In some embodiments of the first aspect of the present application, the screening of the optimal detection frame for each tooth mark includes: performing non-maximum suppression processing and confidence filtering on one or more tooth mark detection frames corresponding to each tooth mark to obtain the optimal detection frame corresponding to the tooth mark.

[0010] In some embodiments of the first aspect of the present application, the tooth mark detection judgment of the optimal detection frame includes: if the optimal detection frame is determined to be not empty, the optimal detection frame passes the tooth mark detection judgment; if the optimal detection frame is determined to be empty, the optimal detection frame fails the tooth mark detection judgment.

[0011] In some embodiments of the first aspect of the present application, the method for training the tooth mark classification model includes: annotating the tooth mark positions and tooth mark categories of multiple acquired tongue images to obtain a tooth mark classification dataset; inputting the tooth mark classification dataset into a TransFG algorithm model for training to obtain the tooth mark classification model.

[0012] In some embodiments of the first aspect of the present application, the tooth mark classification model constructed is used to obtain the tooth mark classification result corresponding to the tongue image to be classified according to the optimal detection frame determined by tooth mark detection, including: based on the optimal detection frame, cropping and pixel size adjustment of the preprocessed tongue image to obtain the corresponding tooth mark image; inputting the tooth mark image into the tooth mark classification model to obtain the tooth mark classification result of the corresponding tooth mark.

[0013] In some embodiments of the first aspect of the present application, inputting the tooth mark image into the tooth mark classification model to obtain the tooth mark classification result corresponding to the tooth mark includes: dividing the input tooth mark image to obtain multiple local regions; performing self-attention mechanism feature encoding on each local region to obtain the spatial feature expression corresponding to each local region; filtering the spatial feature expression corresponding to each local region to obtain the filtered spatial feature expression; performing feature fusion on the filtered spatial feature expression to obtain the final image feature expression; performing softmax classification on the final image feature expression to obtain the corresponding tooth mark classification result.

[0014] In some embodiments of the first aspect of the present application, the types of tooth mark classification results include: no tooth mark results, mild tooth mark results, moderate tooth mark results and severe tooth mark results.

[0015] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a tooth mark classification device based on target detection and fine-grained classification, including: a preprocessing module, used to preprocess the collected tongue image to be classified; a tooth mark detection module, connected to the preprocessing module, used to obtain one or more tooth mark detection frames corresponding to each tooth mark based on the preprocessed tongue image to be classified based on the constructed tooth mark detection model; a screening processing module, connected to the tooth mark detection module, used to screen the optimal detection frame for each tooth mark, and perform tooth mark detection judgment on the optimal detection frame; a tooth mark detection module, connected to the screening processing module, used to obtain the tooth mark classification result corresponding to each tooth mark based on the constructed tooth mark classification model according to the optimal detection frame judged by tooth mark detection.

[0016] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a terminal, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the tooth mark classification method based on target detection and fine-grained classification.

[0017] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the tooth mark classification method based on target detection and fine-grained classification is implemented.

[0018] As described above, the tooth mark classification method, device, terminal and medium based on target detection and fine-grained classification of the present application have the following beneficial effects: the position of the tooth mark on the tongue image to be classified is determined by the constructed tooth mark detection model, and then the image corresponding to each tooth mark obtained according to the tooth mark position is input into the constructed tooth mark classification model to obtain the tooth mark classification result corresponding to each tooth mark. The tooth mark classification method based on target detection and fine-grained classification in the present application can make the tooth mark positioning more accurate and the classification accuracy higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic flow chart of a tooth mark classification method based on target detection and fine-grained classification in one embodiment of the present application.

[0020] Figure 2 Shown is a schematic diagram of the head output layer structure in the benchmark YOLOv7 algorithm model in one embodiment of the present application.

[0021] Figure 3 Shown is a schematic diagram of the head output layer structure in the improved baseline YOLOv7 algorithm model in one embodiment of the present application.

[0022] Figure 4 Shown is a structural schematic diagram of a DCBP structure in an embodiment of the present application.

[0023] Figure 5A Shown is a schematic diagram of the F1-score of the tooth mark detection model in a specific embodiment of the present application.

[0024] Figure 5B Shown is a schematic diagram of the PR curve of the tooth mark detection model in a specific embodiment of the present application.

[0025] Figure 6 Shown is a schematic diagram of the TransFG algorithm model structure in one embodiment of the present application.

[0026] Figure 7 Shown is a schematic diagram of the tooth mark classification process in a specific embodiment of the present application.

[0027] Figure 8 Shown is a schematic diagram of the structure of a tooth mark classification device based on target detection and fine-grained classification in one embodiment of the present application.

[0028] Fig. 9 Shown is a schematic diagram of the structure of a terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0031] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0032] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "comprising" and "including" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. It should be further understood that the terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition occur only when a combination of elements, functions, or operations are inherently mutually exclusive in some manner.

[0033] The present invention provides a tooth mark classification method, device, terminal and medium based on target detection and fine-grained classification, which determines the position of the tooth mark on the tongue image to be classified through a constructed tooth mark detection model, and then inputs the image corresponding to each tooth mark obtained according to the tooth mark position into the constructed tooth mark classification model to obtain the tooth mark classification result corresponding to each tooth mark. The tooth mark classification method based on target detection and fine-grained classification in the present application can make the tooth mark positioning more accurate and the classification accuracy higher.

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0035] like Figure 1 As shown, a flow chart of a tooth mark classification method based on target detection and fine-grained classification in an embodiment of the present invention is presented.

[0036] The tooth mark classification method based on target detection and fine-grained classification includes:

[0037] Step S101: preprocessing the collected tongue image to be classified;

[0038] Optionally, the preprocessing method includes: detecting the tongue position area of ​​the tongue image and cropping the tongue position area after enlarging by a set magnification factor to obtain a tongue image; and performing pixel size adjustment and grayscale processing on the tongue image to obtain a preprocessed tongue image.

[0039] Optionally, set the magnification factor to 1.25 times.

[0040] Optionally, the tongue position area is detected by a tongue detection model; the tongue position area is enlarged according to a set enlargement multiple with the center of the tongue position area, and then the enlarged tongue position area is cropped to obtain a tongue image. Grayscale processing is to convert a color RGB image into a grayscale image. Pixel size adjustment is to adjust the pixel size of the tongue image.

[0041] Optionally, the tongue detection model may adopt target detection models such as Faster RCNN, CornerNet, MaskRCNN, etc., which are not exhaustively listed here. Those skilled in the art may select a suitable tongue detection model according to actual needs.

[0042] It should be noted that the magnification factor can be set according to actual needs, and the present invention does not limit this.

[0043] Step S102: Based on the constructed tooth mark detection model, one or more tooth mark detection frames corresponding to each tooth mark are obtained according to the preprocessed tongue image to be classified.

[0044] In order to take into account both the detection speed and detection accuracy of the tooth mark detection model, in one embodiment of the present invention, a method of optimizing the benchmark YOLOv7 algorithm model based on tooth mark data is used to construct the tooth mark detection model:

[0045] The method for training the tooth mark detection model includes: marking the positions of the tooth marks on a plurality of preprocessed tongue images to obtain a target detection data set; inputting the target detection data set into an improved benchmark YOLOv7 algorithm model for training to obtain the tooth mark detection model;

[0046] The benchmark YOLOv7 algorithm model includes: an input end, a benchmark network, and a head output layer with a pooling layer.

[0047] Optionally, the labelme tool is used to mark the tooth mark position on the preprocessed tongue image, and the marked content includes the tooth mark position information. The marking process is that a technician in the field first pre-marks the preprocessed tongue image, and then two technicians review and correct the pre-marked marking results. It should be noted that the labelme tool is an image labeling software.

[0048] Optionally, after an initial target detection data set is obtained by annotating tooth mark positions on a plurality of preprocessed tongue images, the initial target detection data set is randomly shuffled according to the annotated content and divided into a training set and a validation set to obtain a target detection data set.

[0049] Optionally, in order to better test the generalization of the tooth mark detection model, an external independent dataset is used as a test set to test the tooth mark detection model. This test set is not included in the object detection dataset.

[0050] Optionally, the division ratio of the training set, the validation set, and the test set may be 8:2:2 or 7:3:3.

[0051] Optionally, the method for obtaining the improved benchmark YOLOv7 algorithm model includes: replacing a pooling layer in a head output layer in the benchmark YOLOv7 algorithm model with a DCBP structure;

[0052] Among them, Figure 4 As shown in the figure, the DCBP structure includes: a 3×3 dilated convolution layer (Dilated Conv3×3), a 3×3 standard convolution layer (Conv3×3), a batch normalization layer (Batch Normalization), an activation function layer (Relu) and a maximum pooling layer (Max pool).

[0053] Specifically, Figure 2 As shown in the figure, in the head output layer of the baseline YOLOv7 algorithm model, the result of the third downsampling first passes through the maximum pooling layer (max pool) and then performs feature concatenation (cat) with the result of the fourth downsampling to obtain the result after feature concatenation. The result after feature concatenation passes through the ELAN-H structure and then passes through the maximum pooling layer again to obtain the result after feature concatenation with the result of the fifth downsampling. The maximum pooling layer is a downsampling method that can reduce the size of the feature map, but it is easy to cause feature information loss, especially after multiple pooling operations, which has a great impact on the details of local features. In order to achieve downsampling without causing information loss, as shown in Figure 3 As shown, in this embodiment, each pooling layer in the head output layer of the benchmark YOLOv7 algorithm model is replaced by Figure 4The DCBP structure shown in the figure uses a standard convolution layer (Conv3×3) with stride=2, a 3×3 dilated convolution layer (Dilated Conv3×3), a batch normalization layer (BatchNormalization), an activation function layer (Relu), and a maximum pooling layer (Max pool) to increase the feature extraction capability. The output results of the 3×3 dilated convolution layer (Dilated Conv3×3) and the standard convolution layer (Conv3×3) are concatenated (cat) to obtain a feature map of (h / 2)×(w / 2)×(2c), and then a conv 1×1 convolution layer is used to complete the fusion between feature channels to output the fused feature map of (h / 2)×(w / 2)×c. The DCBP structure can expand the receptive field while completing downsampling.

[0054] It should be noted that compared with previous YOLO series algorithm models, such as the YOLOv5 algorithm model, the baseline YOLOv7 algorithm model can more efficiently utilize model parameters and computational complexity. Using this algorithm model can reduce 50% of model parameters while having faster reasoning speed and higher detection accuracy. In addition, the E-ELAN network structure in the YOLOv7 algorithm model can improve the learning ability of the neural network without destroying the original gradient propagation path.

[0055] Optionally, the preprocessing method includes: detecting the tongue position area of ​​the tongue image and cropping the tongue position area after enlarging by a set magnification factor to obtain a tongue image; and performing pixel size adjustment and grayscale processing on the tongue image to obtain a preprocessed tongue image.

[0056] It should be noted that the preprocessing method in this embodiment can achieve all the functions of the preprocessing method in the above embodiments, which will not be repeated here.

[0057] Optionally, each preprocessed tongue image to be classified includes one or more tooth marks; each tooth mark corresponds to one or more tooth mark detection frames.

[0058] In order to better illustrate the tooth mark detection model, a specific embodiment is now provided.

[0059] Embodiment 1: A method for training a tooth mark detection model.

[0060] In the Ubuntu 18.04 system, a Pytorch environment was created to train the tooth mark detection model. The hardware used for the training model was an Nvidia 3090 24G graphics card. The training parameters were set to epoch=500, batchSize=128, initial learning rate 0.001, and the image sizes of the model input were 320×320 and 640×640 respectively. The dataset included 806 preprocessed tongue images, the validation set included 201 preprocessed tongue images, and the test set included 400 preprocessed tongue images. The comparison of different inputs of the tooth mark detection model is shown in the following table.

[0061] Table 1. Comparison of different inputs of tooth mark detection model

[0062]

[0063] When the input image size is 640×640, the F1-score of the tooth mark detection model is as follows Figure 5A As shown, the PR (Precesion-Recall) curve is as follows Figure 5B shown.

[0064] Step S103: Filter the optimal detection frame for each tooth mark, and perform tooth mark detection judgment on the optimal detection frame.

[0065] Optionally, the screening of the optimal detection frame for each tooth mark includes: performing non-maximum suppression processing and confidence filtering on one or more tooth mark detection frames corresponding to each tooth mark to obtain the optimal detection frame corresponding to the tooth mark.

[0066] It should be noted that the non-maximum suppression processing and confidence filtering use the NMS (Non-Maximum Suppression) algorithm. The tooth mark detection model will output one or more detection frames corresponding to each tooth mark. The NMS algorithm can be used to select the optimal detection frame from multiple detection frames.

[0067] Optionally, non-maximum suppression processing and confidence filtering based on the NMS algorithm include: sorting one or more detection boxes corresponding to each tooth mark in order of confidence from high to low; selecting the detection box with the highest confidence to add to the final output list, and deleting it from the detection box list; wherein the detection box list includes: one or more detection boxes corresponding to each tooth mark; calculating the area of ​​each detection box; calculating the IoU (intersection-over-union, the intersection of two detection boxes divided by the union) of the detection box with the highest confidence and other detection boxes; deleting the detection boxes whose IoU is greater than a threshold; and repeating the above process until the detection box list is empty.

[0068] It should be noted that the threshold can be set according to actual needs, and the present invention does not limit this.

[0069] Optionally, the tooth mark detection judgment on the optimal detection frame includes: if it is determined that the optimal detection frame is not empty, the optimal detection frame passes the tooth mark detection judgment; if it is determined that the optimal detection frame is empty, the optimal detection frame fails the tooth mark detection judgment.

[0070] Further, if it is determined that the optimal detection frame is empty, the optimal detection frame fails to pass the tooth mark detection and a detection result without tooth marks is output.

[0071] Specifically, the optimal detection frame is judged by tooth mark detection to determine whether there are tooth marks on the tongue image to be classified. If the optimal detection frame is not empty, the optimal detection frame passes the tooth mark detection judgment, which means that there are tooth marks on the tongue image to be classified. The tooth marks need to be classified by the constructed tooth mark classification model to obtain the tooth mark classification result of each tooth mark. If the optimal detection frame is empty, the optimal detection frame does not pass the tooth mark detection judgment, which means that there are no tooth marks on the tongue image to be classified.

[0072] Step S104: Based on the constructed tooth mark classification model, a tooth mark classification result corresponding to each tooth mark is obtained according to an optimal detection frame determined by tooth mark detection.

[0073] Optionally, types of tooth mark classification results include: no tooth mark result, light tooth mark result, moderate tooth mark result and heavy tooth mark result.

[0074] Optionally, the method for training the tooth mark classification model includes: annotating the tooth mark positions and tooth mark categories of multiple acquired tongue images to obtain a tooth mark classification dataset; and inputting the tooth mark classification dataset into a TransFG algorithm model for training to obtain the tooth mark classification model.

[0075] Optionally, the labelme tool is used to annotate the tooth mark position and tooth mark category of the preprocessed tongue image, and the annotated content includes the tooth mark position information and the tooth mark category information. The annotation process is that a technician in the field first pre-annotates the preprocessed tongue image, and then two technicians review and correct the pre-annotated annotation results.

[0076] Optionally, after an initial tooth mark classification dataset is obtained by annotating the tooth mark positions and tooth mark categories of multiple preprocessed tongue images, the initial tooth mark classification dataset is randomly shuffled according to the annotated content and divided into a training set and a validation set to obtain a tooth mark classification dataset.

[0077] It should be noted that since the differences between the light tooth mark results, moderate tooth mark results and heavy tooth mark results in the tooth mark classification results are small, it is suitable to use the image fine-grained classification method. The TransFG algorithm model is a fine-grained image classification method based on Transformer, which can be applied to sub-classification tasks with small distinction between categories. The TransFG algorithm model mainly adopts the encoding method of spatial feature encoding based on the local area attention mechanism and channel feature encoding based on global features, and has achieved good results in the field of fine-grained image classification. The TransFG algorithm model draws on the self-attention mechanism and multi-head attention mechanism in the Transformer, which can capture the contextual relationship between different positions in the image and can effectively encode and represent the image features.

[0078] Optionally, the tooth mark classification model constructed based on the optimal detection frame determined by tooth mark detection obtains the tooth mark classification result corresponding to the tongue image to be classified, including: based on the optimal detection frame, cropping and pixel size adjustment of the preprocessed tongue image to obtain the corresponding tooth mark image; inputting the tooth mark image into the tooth mark classification model to obtain the tooth mark classification result of the corresponding tooth mark.

[0079] Optionally, cropping and pixel size adjustment of the preprocessed tongue image include: cropping the tongue image to be classified according to the optimal detection frame of each tooth mark to obtain the original tooth mark image of the tooth mark; adjusting the pixel size of the original tooth mark image to a set pixel size according to the long side size of the original tooth mark image.

[0080] Optionally, set the pixel size to 64×64.

[0081] It should be noted that the set pixel size can be set according to actual needs, and the present invention does not limit this.

[0082] Optionally, inputting the tooth mark image into the tooth mark classification model to obtain the tooth mark classification result corresponding to the tooth mark includes: dividing the input tooth mark image to obtain multiple local regions; performing self-attention mechanism feature encoding on each local region to obtain the spatial feature expression corresponding to each local region; filtering the spatial feature expression corresponding to each local region to obtain the filtered spatial feature expression; performing feature fusion on the filtered spatial feature expression to obtain the final image feature expression; performing softmax classification on the final image feature expression to obtain the corresponding tooth mark classification result.

[0083] Specifically, Figure 6As shown, the image of the input model is divided into multiple local areas, and then the local areas are sequentially combined into an input sequence. Each local area is formed into an input vector corresponding to each local area through Linear Projection. The input vector corresponding to each local area is encoded by the self-attention mechanism feature through Transformer Encoder to obtain the spatial feature expression corresponding to each local area. The spatial feature expression corresponding to each local area is filtered through the Part Selection Module to obtain the filtered spatial feature expression. Redundant features can be removed by filtering through the Part Selection Module. Then, the final image feature expression is obtained by feature fusion by globally encoding the filtered spatial feature expression. Finally, the final image feature expression is classified through the Classification Head to obtain the corresponding tooth mark classification result. The Classification Head uses a softmax classifier for classification.

[0084] Preferably, since the distance between categories should be considered when classifying categories with continuity and degree, when classifying tooth marks, the formula for calculating the standard cross-entropy loss (Cross-entropyLoss) in the TransFG algorithm model in FIG5 can be replaced by the following formula:

[0085]

[0086] Where N is the number of tooth mark images in the sample, M is the number of categories, and d c Represents the distance between categories, y ic The sign function takes the value 0 or 1, P ic It is expressed as the probability that the i-th sample is predicted to be category c.

[0087] In order to better illustrate the tooth mark classification model, a specific embodiment is now provided.

[0088] Embodiment 2: A method for training a tooth mark classification model.

[0089] The pixel size of the tooth mark classification model input image is set to 64×64, the training parameters are set to epoch=100, batchSize=8, the initial learning rate is 0.01, and the mixed precision training method is adopted. 1174 tooth mark images are selected for training and testing. The test results of the tooth mark classification model are shown in the following table.

[0090] Table 2. Test results of tooth mark classification model

[0091] category Precision Recall f1-score Mild 95.15 93.88 94.51 Moderate 94.63 94.63 95.64 Severe 93.26 91.32 92.28

[0092] In order to better illustrate the tooth mark classification method based on target detection and fine-grained classification, a specific embodiment is now provided.

[0093] Embodiment 3: A tooth mark classification method based on target detection and fine-grained classification.

[0094] like Figure 7 As shown, a collected tongue image to be classified is preprocessed to obtain a preprocessed tongue image. The preprocessed tongue image is input into the tooth mark detection model to obtain the model detection result; wherein the model detection result is one or more tooth mark detection boxes corresponding to each tooth mark. The model detection result is processed by NMS and confidence filtering to obtain the tooth mark bbox (optimal detection box) corresponding to each tooth mark. Determine whether the tooth mark bbox is empty; if the tooth mark bbox is empty, output the tooth mark classification result without tooth marks; if the tooth mark bbox is not empty, then according to the tooth mark bbox of each tooth mark, the tooth mark image corresponding to the tooth mark is cropped on the tongue image to be classified, and the tooth mark image of each tooth mark is input into the tooth mark classification model to obtain the tooth mark classification result corresponding to each tooth mark.

[0095] Similar to the principle of the above embodiment, the present invention provides a tooth mark classification device based on target detection and fine-grained classification.

[0096] The following provides specific embodiments in conjunction with the accompanying drawings:

[0097] like Figure 8 As shown, a tooth mark classification device based on target detection and fine-grained classification in an embodiment of the present invention is demonstrated.

[0098] The tooth mark classification device 8 based on target detection and fine-grained classification comprises:

[0099] A preprocessing module 81 is used to preprocess the collected tongue image to be classified;

[0100] A tooth mark detection module 82, connected to the preprocessing module 81, is used to obtain one or more tooth mark detection frames corresponding to each tooth mark according to the preprocessed tongue image to be classified based on the constructed tooth mark detection model;

[0101] A screening processing module 83, connected to the tooth mark detection module 82, is used to screen the optimal detection frame of each tooth mark and perform tooth mark detection judgment on the optimal detection frame;

[0102] The tooth mark detection module 84 is connected to the screening processing module 83, and is used to obtain the tooth mark classification result corresponding to each tooth mark according to the constructed tooth mark classification model and the optimal detection frame determined by the tooth mark detection.

[0103] Since the device in this embodiment can realize all the functions of the above embodiments, they will not be repeated here.

[0104] It should be noted that the modules provided in this embodiment are similar to the methods provided above in the implementation mode, so they will not be described in detail. It should also be noted that it should be understood that the division of the modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the preprocessing module 81 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the preprocessing module 81. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0105] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0106] like Fig. 9 As shown, it is a schematic diagram of the structure of the terminal in an embodiment of the present invention.

[0107] The terminal 9 comprises: a processor 92 and a memory 91; the memory 91 is used to store a computer program; the processor 92 is used to execute the computer program stored in the memory, so that the terminal 9 performs the following steps: Figure 1 The tooth mark classification method based on target detection and fine-grained classification.

[0108] Optionally, the number of the memory 91 can be one or more, the number of the processor 92 can be one or more, and Fig. 9 Take one as an example.

[0109] Optionally, the processor 92 in the control device will Figure 1 In the above steps, one or more instructions corresponding to the process of the application are loaded into the memory 91, and the processor 92 runs the application stored in the first memory, thereby achieving the following Figure 1 Various functions in the tooth mark classification method based on target detection and fine-grained classification.

[0110] Optionally, the memory 91 may include but is not limited to high-speed random access memory and non-volatile memory. For example, one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices; the processor 92 may include but is not limited to a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0111] Optionally, the processor 92 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0112] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed as follows: Figure 1 The tooth mark classification method based on target detection and fine-grained classification. The computer-readable storage medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (read-only disk memory), a magneto-optical disk, a ROM (read-only memory), a RAM (random access memory), an EPROM (erasable programmable read-only memory), an EEPROM (electrically erasable programmable read-only memory), a magnetic card or an optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product that is not connected to a computer device, or a component that is connected to a computer device for use.

[0113] In some embodiments of the present invention, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, server or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0114] In summary, the present application provides a tooth mark classification method, device, terminal and medium based on target detection and fine-grained classification, which determines the position of the tooth mark on the tongue image to be classified through a constructed tooth mark detection model, and then inputs the image corresponding to each tooth mark obtained according to the tooth mark position into the constructed tooth mark classification model to obtain the tooth mark classification result corresponding to each tooth mark. The tooth mark classification method based on target detection and fine-grained classification in the present application can make the tooth mark positioning more accurate and the classification accuracy higher.

[0115] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A tooth mark classification method based on target detection and fine-grained classification, characterized in that: include: Preprocessing the collected tongue images to be classified; Based on the constructed tooth mark detection model, one or more tooth mark detection frames corresponding to each tooth mark are obtained according to the preprocessed tongue image to be classified; Filter the optimal detection frame for each tooth mark, and perform tooth mark detection judgment on the optimal detection frame; Based on the constructed tooth mark classification model, the tooth mark classification result corresponding to each tooth mark is obtained according to the optimal detection frame determined by tooth mark detection.

2. The method according to claim 1, characterized in that The method for training the tooth mark detection model includes: Annotating the tooth mark positions on multiple pre-processed tongue images to obtain a target detection data set; The target detection data set is input into an improved benchmark YOLOv7 algorithm model for training to obtain the tooth mark detection model; wherein the benchmark YOLOv7 algorithm model includes: an input end, a benchmark network, and a head output layer with a pooling layer.

3. The method according to claim 2, characterized in that The method for obtaining the improved baseline YOLOv7 algorithm model includes: The pooling layer in the head output layer of the benchmark YOLOv7 algorithm model is replaced with a DCBP structure; The DCBP structure includes: a 3×3 hole convolution layer, a 3×3 standard convolution layer, a batch normalization layer, an activation function layer and a maximum pooling layer.

4. The method according to claim 1 or 2, characterized in that: Methods for preprocessing include: Detecting the tongue position area of ​​the tongue image and cropping the tongue position area after enlarging the tongue by a set magnification to obtain a tongue image; The pixel size of the tongue image is adjusted and the grayscale is processed to obtain a preprocessed tongue image.

5. The method according to claim 1, characterized in that The optimal detection frame for screening each tooth mark includes: Non-maximum suppression processing and confidence filtering are performed on one or more tooth mark detection frames corresponding to each tooth mark to obtain the optimal detection frame corresponding to the tooth mark.

6. The method according to claim 1, characterized in that The tooth mark detection and judgment of the optimal detection frame includes: If it is determined that the optimal detection frame is not empty, the optimal detection frame is determined by tooth mark detection; If the optimal detection frame is determined to be empty, the optimal detection frame fails to pass the tooth mark detection.

7. The method according to claim 1, characterized in that The method for training the tooth mark classification model includes: Annotating the tooth mark positions and tooth mark categories of the acquired multiple tongue images to obtain a tooth mark classification dataset; The tooth mark classification data set is input into the TransFG algorithm model for training to obtain the tooth mark classification model.

8. The method according to claim 7, characterized in that The tooth mark classification model constructed based on the tooth mark classification result corresponding to the tongue image to be classified is obtained according to the optimal detection frame determined by tooth mark detection, including: Based on the optimal detection frame, the preprocessed tongue image is cropped and the pixel size is adjusted to obtain a corresponding tooth mark image; The tooth mark image is input into the tooth mark classification model to obtain a tooth mark classification result corresponding to the tooth mark.

9. The method according to claim 8, characterized in that Inputting the tooth mark image into the tooth mark classification model to obtain a tooth mark classification result corresponding to the tooth mark includes: Divide the input tooth mark image to obtain multiple local areas; Perform self-attention mechanism feature encoding on each local area to obtain the spatial feature expression corresponding to each local area; Filter the spatial feature expression corresponding to each local area to obtain the filtered spatial feature expression; Perform feature fusion on the filtered spatial feature expression to obtain the final image feature expression; The final image feature expression is subjected to softmax classification to obtain a corresponding tooth mark classification result.

10. The method according to claim 1, characterized in that The types of tooth mark classification results include: no tooth mark result, light tooth mark result, moderate tooth mark result and heavy tooth mark result.

11. A tooth mark classification device based on target detection and fine-grained classification, characterized in that: include: A preprocessing module, used for preprocessing the collected tongue images to be classified; A tooth mark detection module, connected to the preprocessing module, for obtaining one or more tooth mark detection frames corresponding to each tooth mark according to the preprocessed tongue image to be classified based on the constructed tooth mark detection model; A screening processing module, connected to the tooth mark detection module, for screening the optimal detection frame of each tooth mark and performing tooth mark detection judgment on the optimal detection frame; The tooth mark detection module is connected to the screening and processing module and is used to obtain the tooth mark classification result corresponding to each tooth mark based on the constructed tooth mark classification model and the optimal detection frame determined by tooth mark detection.

12. A terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so as to enable the terminal to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 10 is implemented.

Images