A few-sample graph convolution tongue image classification method, device, equipment and storage medium

Abstract

The present invention provides a few-sample graph convolution tongue image classification method, device, equipment and storage medium, firstly, by obtaining a query tongue image, and calling a local spatial attention module to extract features from the query tongue image and support samples to generate a query feature matrix and a support feature matrix; then, calling a pre-trained independent graph convolution network to perform graph convolution operations on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation; finally, calculating the distance between the query feature vector and the support feature vector, and taking the category of the support sample corresponding to the minimum distance as the classification result of the query tongue image. The problem that the prior art cannot classify tongue image pictures with a small number of samples is solved.

Description

Technical Field

[0001] The present invention relates to the field of tongue image classification, and in particular to a few-sample graph convolution tongue image classification method, device, equipment and storage medium. Background Art

[0002] In the current tongue image feature classification technology, computer vision tasks mainly rely on convolutional neural networks (CNN) for processing and analysis. Initially, researchers proposed using AlexNet for feature classification. With the continuous development of convolutional neural networks, ResNet has also been introduced into current tasks. Some scholars, such as Sun Meng, etc., proposed a tongue image classification method that combines the ResNet network and the Inception module. They adopted the metric classification method of TripletLoss to achieve image classification. This method not only increases the depth and width of the network, but also enables the network to better extract features.

[0003] In recent years, the most commonly used attention mechanism SENet (Squeeze-and-Excitation Network) in tongue image classification tasks is the SENet. Niu Fuquan et al. proposed a tongue area image classification method based on the ResNet-101 network and SE block. They first used image enhancement technology to process the input tongue area image, and then used the ResNet-101 network to extract image features. Then, the SE block was introduced to adaptively adjust the weights of different feature channels to further improve the accuracy of feature expression. Finally, they used a fully connected layer to classify the features and achieve classification and recognition of the tongue area.

[0004] However, compared with conventional medical images, it is more difficult to obtain tongue images. At the same time, the status of various internal diseases reflected on the tongue needs to be judged by professional doctors in order to accurately give disease labels. Therefore, the method of training classification models through large-scale data is time-consuming and labor-intensive.

[0005] In view of this, this application is filed. Summary of the invention

[0006] The present invention discloses a few-sample graph convolution tongue image classification method, device, equipment and storage medium, aiming to solve the problem that the prior art cannot classify tongue image pictures with a small number of samples.

[0007] A first embodiment of the present invention provides a few-sample graph convolution tongue image classification method, comprising:

[0008] Acquire a query tongue image, and call a local spatial attention module to extract features from the query tongue image and supporting samples to generate a query feature matrix and a supporting feature matrix;

[0009] Calling a pre-trained independent graph convolution network to perform a graph convolution operation on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation;

[0010] The distance between the query feature vector and the support feature vector is calculated, and the category of the support sample corresponding to the minimum distance is used as the classification result of the query tongue image.

[0011] Preferably, the calling of the local spatial attention module to extract features from the query tongue image to generate a query feature matrix is ​​specifically:

[0012] Performing local perception on the query tongue image to differentiate and generate multiple regions;

[0013] A composite feature extraction network of CNN and Transformer is used to extract features from the query tongue image to generate feature representations of each region, wherein the CNN can deeply extract detailed features of each region, and the Transformer can perceive features of the entire region using self-attention;

[0014] The feature representations of each region are stacked to generate a query feature matrix of the query tongue image.

[0015] Preferably, local perception is performed on the query tongue image to differentiate and generate a tongue tip area, a tongue surface area, a tongue root area, a left tongue edge area, and a right tongue edge area.

[0016] Preferably, the training of the independent graph convolutional network includes:

[0017] Sample query sets and support sets from the training set, perform data augmentation on the sampled query sets and support sets, and then input them into the classification model for feature extraction to generate query set feature vectors and the support set eigenvector , where N is the number of categories in the support set, K is the number of images in each category, and d is the final feature dimension;

[0018] Call the cross entropy loss function to calculate the first loss between the query set feature vector and the support set feature vector , the loss formula is:

[0019] ;

[0020] in, For query samples The category label of is the i-th support sample The category label of For indexing and traversing support set samples, For the support set features, It is used to determine whether the support set feature and the query set feature category are the same, if they are not the same, it is recorded as 0, and if they are the same, it is recorded as 1;

[0021] Maximizing rank, rank loss when running a standalone graph convolutional network The formula is:

[0022]

[0023] in, Used to index and traverse all samples, is the final feature matrix of the jth sample, It is used to calculate the sum of the singular values ​​of the current feature matrix. The number of singular values ​​is the rank of the matrix.

[0024] The first loss and the rank loss are summed to generate a final loss, and the parameters of the independent graph convolutional network are updated according to the final loss until the training iteration is completed.

[0025] Preferably, the expression of the graph convolution operation is:

[0026]

[0027] in, is the activation function, is the diagonal matrix of the graph, is the adjacency matrix, is the node feature matrix of the lth layer, is the weight matrix of the lth layer, is the node feature matrix of the l+1th layer.

[0028] A second embodiment of the present invention provides a tongue image classification device using a few-sample graph convolution method, comprising:

[0029] A feature matrix generating unit, used for acquiring a query tongue image, and calling a local spatial attention module to extract features from the query tongue image and supporting samples, so as to generate a query feature matrix and a supporting feature matrix;

[0030] A feature vector generating unit, configured to call a pre-trained independent graph convolutional network to perform a graph convolution operation on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolutional network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation;

[0031] The classification result generating unit is used to calculate the distance between the query feature vector and the support feature vector, and take the category of the support sample corresponding to the minimum distance as the classification result of the query tongue image.

[0032] Preferably, the calling of the local spatial attention module to extract features from the query tongue image to generate a query feature matrix is ​​specifically:

[0033] Performing local perception on the query tongue image to differentiate and generate multiple regions;

[0034] A composite feature extraction network of CNN and Transformer is used to extract features from the query tongue image to generate feature representations of each region, wherein the CNN can deeply extract detailed features of each region, and the Transformer can perceive features of the entire region using self-attention;

[0035] The feature representations of each region are stacked to generate a query feature matrix of the query tongue image.

[0036] Preferably, local perception is performed on the query tongue image to differentiate and generate a tongue tip area, a tongue surface area, a tongue root area, a left tongue edge area, and a right tongue edge area.

[0037] A few-sample graph convolution tongue image classification device provided in the third embodiment of the present invention is characterized in that it includes a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to implement a few-sample graph convolution tongue image classification method as described in any one of the above.

[0038] A computer-readable storage medium provided in the fourth embodiment of the present invention is characterized in that a computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a few-sample graph convolution tongue image classification method as described in any one of the above items.

[0039] Based on a few-sample graph convolution tongue classification method, device, equipment and storage medium provided by the present invention, firstly, by acquiring a query tongue image, and calling a local spatial attention module to extract features from the query tongue image and the support sample, a query feature matrix and a support feature matrix are generated; then, a pre-trained independent graph convolution network is called to perform graph convolution operations on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation; finally, the distance between the query feature vector and the support feature vector is calculated, and the category of the support sample corresponding to the minimum distance is used as the classification result of the query tongue image. The problem that the prior art cannot classify tongue images with a small number of samples is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a few-sample graph convolution tongue image classification method provided by the first embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the main framework flow chart provided by the present invention;

[0042] Figure 3 It is a schematic diagram of tongue characteristic partitioning provided by the present invention;

[0043] Figure 4 is a schematic diagram of a local spatial attention module provided by the present invention;

[0044] Figure 5 The tongue image diagram composed of nodes provided by the present invention;

[0045] Figure 6 The present invention provides a matrix of angles of tongue images;

[0046] Figure 7 is an adjacency matrix of a tongue image diagram provided by the present invention;

[0047] Figure 8 It is a module schematic diagram of a few-sample graph convolution tongue image classification device provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0050] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0052] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0053] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0054] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0055] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] The present invention discloses a few-sample graph convolution tongue image classification method, device, equipment and storage medium, aiming to solve the problem that the prior art cannot classify tongue image pictures with a small number of samples.

[0057] See also Figure 1 and Figure 2The first embodiment of the present invention provides a few-sample graph convolution tongue image classification method, which can be performed by a few-sample graph convolution tongue image classification device (hereinafter referred to as the classification device), and in particular, is performed by one or more processors in the classification device to at least implement the following steps:

[0058] S101, obtaining a query tongue image, and calling a local spatial attention module to extract features from the query tongue image and supporting samples to generate a query feature matrix and a supporting feature matrix;

[0059] In this embodiment, the classification device may be a terminal with data processing capability such as a laptop computer, a desktop computer, a server, a workstation, etc., and the classification device may be installed with a corresponding operating system and application software, and the functions required by this embodiment are realized through the combination of the operating system and the application software;

[0060] It should be noted that since tongue image annotation requires professional doctor experience, it is difficult to obtain a large number of annotated samples. The local spatial attention module can maximize the use of limited support samples and extract rich regional feature information, thereby improving the generalization ability of few-sample classification.

[0061] In this embodiment, the calling of the local spatial attention module performs feature extraction on the query tongue image to generate a query feature matrix, specifically:

[0062] The query tongue image is locally sensed to differentiate and generate multiple regions (such as Figure 3 As shown), it can be differentiated into a tongue tip region n5, a tongue surface region n3, a tongue root region n1, a left tongue edge region n2, and a right tongue edge region n4;

[0063] Please combine Figure 4 , a composite feature extraction network of CNN and Transformer is used to extract features from the query tongue image to generate a feature representation of each region , , wherein the CNN can deeply extract the detailed features of each region, and the Transformer can perceive the features of the entire region using self-attention;

[0064] The feature representations of each region are stacked to generate a query feature matrix of the query tongue image.

[0065] It should be noted that by calling the local spatial attention module to extract features from the query tongue image, local perception of the tongue image can be achieved, and the image can be differentiated into multiple regions. This facilitates the extraction of features from each region, allowing the model to more accurately capture feature information from different regions in the tongue image, thereby improving the accuracy of feature representation.

[0066] Furthermore, in the feature extraction process, a composite feature extraction network of CNN and Transformer is used. CNN can deeply extract the detailed features of each region, while Transformer can perceive the features of the entire region using the self-attention mechanism. In order to ensure that the feature representation of each region is relatively independent and avoid the mutual influence between the features of different regions, the feature representation of each region is then stacked to generate a query feature matrix for querying the tongue image. The query feature matrix contains the feature information of all regions and can comprehensively reflect the feature representation of the entire tongue image. Through this comprehensive feature representation, the features of the tongue image can be described more comprehensively, which helps to improve the classification accuracy of the model for the tongue image.

[0067] S102, calling a pre-trained independent graph convolution network to perform a graph convolution operation on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation;

[0068] It should be noted that the Graph Convolutional Network is a deep learning model for processing graph data. Unlike the traditional Convolutional Neural Network (CNN) which is used to process image data with regular grid structures, GCN is used to process irregular data, such as graph data in the fields of social networks, recommendation systems, bioinformatics, etc. In this embodiment, the five regions of the tongue image can be regarded as five nodes of the graph, thus forming the following Figure 5 Then, the angle matrix of the tongue image can be obtained (such as Figure 6 ) and the adjacency matrix (as Figure 7 As shown in Figure 2). Therefore, we can update and iterate our tongue image features through the graph convolution formula. The update formula is:

[0069]

[0070] in, is the activation function, is the diagonal matrix of the graph, is the adjacency matrix, is the node feature matrix of the lth layer, is the weight matrix of the lth layer, is the node feature matrix of the l+1th layer.

[0071] It should be noted that a common problem when using graph convolutional networks is over-smoothing. Over-smoothing refers to the tendency of the representation of nodes to aggregate into similar values ​​after passing through multiple layers of graph convolutional layers, resulting in a loss of distinction between nodes. This phenomenon may affect the model's learning of the representation of nodes in the graph, especially when the graph structure is large or the number of layers is deep. Although increasing the amount of model training data can alleviate this phenomenon, this method is not applicable to few-sample tasks.

[0072] Therefore, in this embodiment, an independent graph convolutional network is used, which additionally designs a node-independent module based on the graph convolutional network. This module calculates the rank of the updated graph matrix in each parameter iteration and maximizes the rank. The rank of a matrix refers to the maximum number of linearly independent columns (or rows) in the matrix. The larger the rank, the more linearly independent rows there are in the matrix. Through this operation, the features of each node can be made more independent, thereby reducing the impact of over-smoothing on the tongue image features. Finally, in this embodiment, an activation function is used to convert the multi-node feature matrix into a tongue image feature vector.

[0073] S103, calculating the distance between the query feature vector and the support feature vector, and taking the category of the support sample corresponding to the minimum distance as the classification result of the query tongue image.

[0074] It should be noted that in this embodiment, by comparing the distance between the query feature vector and the support feature vector, the query tongue image can be matched with samples of similar features in the support set. According to the minimum distance principle, the category of the support sample with the closest distance is used as the classification result of the query tongue image. Fully utilizing the limited support sample information In the few-sample learning scenario, in this embodiment, only a very small number of support samples are available. By calculating the distance between the query sample and each support sample, the category information carried by these valuable support samples is fully utilized to maximize their value.

[0075] In a possible embodiment of the present invention, the training of the independent graph convolutional network includes:

[0076] Sample query sets and support sets from the training set, perform data augmentation on the sampled query sets and support sets, and then input them into the classification model for feature extraction to generate query set feature vectors and the support set eigenvector , where N is the number of categories in the support set, K is the number of images in each category, and d is the final feature dimension;

[0077] The query set is a sample set used to test the performance of the model, and contains new samples that the model has not seen during training. In this embodiment, the query set is obtained by random sampling from the entire training set. Specifically, some tongue images can be randomly selected from the training set as query samples to construct the query set. These query samples represent new and unknown inputs that the model needs to classify for a small number of samples.

[0078] The support set is a set of samples used to train the model. In meta-learning, the support set is used for the model to learn how to adapt to new classification tasks. In this embodiment, the support set is constructed by sampling from the training set in the form of "N-way K-shot": N represents the number of categories contained in the support set, and K represents the number of support samples contained in each category; the construction process can be: 1) randomly select N tongue image categories from the entire training set, 2) for these N categories, randomly select K tongue images from each category as support samples for that category, 3) combine the N*K support samples of these N categories to form an "N-way K-shot" support set. For example, if N=5 and K=3, then the support set contains 5 categories, 3 support samples for each category, and a total of 15 tongue images. This support set will be used for the model to learn how to classify new categories based on limited samples. By continuously sampling different support sets from the training set, the model can learn to better adapt to various new few-sample classification tasks.

[0079] Call the cross entropy loss function to calculate the first loss between the query set feature vector and the support set feature vector , the loss formula is:

[0080] ;

[0081] in, To query the sample The category label of is the i-th support sample The category label of For indexing and traversing support set samples, For the support set features, It is used to determine whether the support set feature and the query set feature category are the same, if they are not the same, it is recorded as 0, and if they are the same, it is recorded as 1;

[0082] Maximizing rank, rank loss when running a standalone graph convolutional network The formula is:

[0083]

[0084] in, Used to index and traverse all samples, is the final feature matrix of the jth sample, It is used to calculate the sum of the singular values ​​of the current feature matrix. The number of singular values ​​is the rank of the matrix.

[0085] The first loss and the rank loss are summed to generate a final loss, and the parameters of the independent graph convolutional network are updated according to the final loss until the training iteration is completed.

[0086] It should be noted that in this embodiment, in order to make full use of data enhancement to improve the generalization ability of the model, the training process performs data enhancement processing on the sampled query set and support set before inputting the model. Data enhancement can provide more diverse training data for the model, enhance the model's adaptability to image transformation, and thus improve the generalization ability of the model during testing, wherein data enhancement includes but is not limited to operations such as rotation, translation, scaling, and mirroring.

[0087] Furthermore, in this embodiment, the feature distance and feature diversity are optimized simultaneously. The training process optimizes two important goals: minimizing the distance between the query feature and the support feature (first loss) and maximizing the rank of the feature matrix (rank loss). The former ensures that the distance between similar samples is the closest, while the latter avoids features that are too smooth or too similar. The combination of the two can learn high-quality feature representations that are discriminative.

[0088] See also Figure 7 A second embodiment of the present invention provides a tongue image classification device using a few-sample graph convolution method, comprising:

[0089] The feature matrix generating unit 201 is used to obtain a query tongue image and call a local spatial attention module to extract features from the query tongue image and supporting samples to generate a query feature matrix and a supporting feature matrix;

[0090] A feature vector generating unit 202 is used to call a pre-trained independent graph convolution network to perform a graph convolution operation on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix when running;

[0091] The classification result generating unit 203 is used to calculate the distance between the query feature vector and the support feature vector, and take the category of the support sample corresponding to the minimum distance as the classification result of the query tongue image.

[0092] Preferably, the calling of the local spatial attention module to extract features from the query tongue image to generate a query feature matrix is ​​specifically:

[0093] Performing local perception on the query tongue image to differentiate and generate multiple regions;

[0094] A composite feature extraction network of CNN and Transformer is used to extract features from the query tongue image to generate feature representations of each region, wherein the CNN can deeply extract detailed features of each region, and the Transformer can perceive features of the entire region using self-attention;

[0095] The feature representations of each region are stacked to generate a query feature matrix of the query tongue image.

[0096] Preferably, local perception is performed on the query tongue image to differentiate and generate a tongue tip area, a tongue surface area, a tongue root area, a left tongue edge area, and a right tongue edge area.

[0097] A few-sample graph convolution tongue image classification device provided in the third embodiment of the present invention is characterized in that it includes a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to implement a few-sample graph convolution tongue image classification method as described in any one of the above.

[0098] A computer-readable storage medium provided in the fourth embodiment of the present invention is characterized in that a computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a few-sample graph convolution tongue image classification method as described in any one of the above items.

[0099] Based on a few-sample graph convolution tongue classification method, device, equipment and storage medium provided by the present invention, firstly, by acquiring a query tongue image, and calling a local spatial attention module to extract features from the query tongue image and the support sample, a query feature matrix and a support feature matrix are generated; then, a pre-trained independent graph convolution network is called to perform graph convolution operations on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation; finally, the distance between the query feature vector and the support feature vector is calculated, and the category of the support sample corresponding to the minimum distance is used as the classification result of the query tongue image. The problem that the prior art cannot classify tongue images with a small number of samples is solved.

[0100] Exemplarily, the computer program described in the third and fourth embodiments of the present invention may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device for implementing a few-sample graph convolution tongue image classification. For example, the device described in the second embodiment of the present invention.

[0101] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the few-sample graph convolution tongue image classification method, and uses various interfaces and lines to connect the various parts of the implementation of the few-sample graph convolution tongue image classification method.

[0102] The memory can be used to store the computer program and / or module, and the processor realizes various functions of a few-sample graph convolution tongue image classification method by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0103] Wherein, if the implemented module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0104] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0105] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A few-sample graph convolution tongue image classification method, characterized in that: include: Acquire a query tongue image, and call a local spatial attention module to perform feature extraction on the query tongue image and support samples to generate a query feature matrix and a support feature matrix; the calling of the local spatial attention module to perform feature extraction on the query tongue image to generate a query feature matrix is ​​specifically as follows: perform local perception on the query tongue image to differentiate and generate a tongue tip region, a tongue surface region, a tongue root region, a left tongue edge region, and a right tongue edge region; use a CNN and Transformer composite feature extraction network to perform feature extraction on the query tongue image to generate a feature representation of each region, wherein the CNN can perform in-depth extraction of detailed features of each region, and the Transformer can use self-attention to perceive features of the entire region; the feature representations of each region are stacked to generate a query feature matrix of the query tongue image; Calling a pre-trained independent graph convolution network to perform a graph convolution operation on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix during operation; wherein the training of the independent graph convolution network includes: Sample query sets and support sets from the training set, perform data augmentation on the sampled query sets and support sets, and then input them into the classification model for feature extraction to generate query set feature vectors and the support set eigenvector , where N is the number of categories in the support set, K is the number of images in each category, and d is the final feature dimension; Call the cross entropy loss function to calculate the first loss between the query set feature vector and the support set feature vector , the loss formula is: ； in, For query samples The category label of is the i-th support sample The category label of For indexing and traversing support set samples, It is used to determine whether the support set feature and the query set feature category are the same, if they are not the same, it is recorded as 0, and if they are the same, it is recorded as 1; Maximizing rank, rank loss when running a standalone graph convolutional network The formula is: in, Used to index and traverse all samples, is the final feature matrix of the jth sample, It is used to calculate the sum of the singular values ​​of the current feature matrix. The number of singular values ​​is the rank of the matrix. Summing the first loss and the rank loss to generate a final loss, and updating the parameters of the independent graph convolutional network according to the final loss until the training iteration is completed; The distance between the query feature vector and the support feature vector is calculated, and the category of the support sample corresponding to the minimum distance is used as the classification result of the query tongue image.

2. The method for tongue image classification using few-sample graph convolution according to claim 1, characterized in that: The expression of the graph convolution operation is: in, is the activation function, is the diagonal matrix of the graph, is the adjacency matrix, is the node feature matrix of the lth layer, is the weight matrix of the lth layer, is the node feature matrix of the l+1th layer.

3. A tongue image classification device using few-sample graph convolution, characterized in that: include: A feature matrix generating unit is used to obtain a query tongue image, and call a local spatial attention module to perform feature extraction on the query tongue image and supporting samples to generate a query feature matrix and a supporting feature matrix. The calling of the local spatial attention module to perform feature extraction on the query tongue image to generate a query feature matrix is ​​specifically as follows: performing local perception on the query tongue image to differentiate and generate a tongue tip region, a tongue surface region, a tongue root region, a left tongue edge region, and a right tongue edge region; using a CNN and Transformer composite feature extraction network to perform feature extraction on the query tongue image to generate a feature representation of each region, wherein the CNN can perform in-depth extraction of detailed features of each region, and the Transformer can use self-attention to perceive features of the entire region; and stacking the feature representations of each region to generate a query feature matrix of the query tongue image; A feature vector generating unit is used to call a pre-trained independent graph convolution network to perform a graph convolution operation on the query feature matrix and the support feature matrix to generate a query feature vector and a support feature vector, wherein the independent graph convolution network maximizes the rank of the query feature matrix and the rank of the support feature matrix when running, wherein, The training of the independent graph convolutional network includes: Sample query sets and support sets from the training set, perform data augmentation on the sampled query sets and support sets, and then input them into the classification model for feature extraction to generate query set feature vectors and the support set eigenvector , where N is the number of categories in the support set, K is the number of images in each category, and d is the final feature dimension; Call the cross entropy loss function to calculate the first loss between the query set feature vector and the support set feature vector , the loss formula is: ； in, To query the sample The category label of is the i-th support sample The category label of For indexing and traversing support set samples, It is used to determine whether the support set feature and the query set feature category are the same, if they are not the same, it is recorded as 0, and if they are the same, it is recorded as 1; Maximizing rank, rank loss when running a standalone graph convolutional network The formula is: in, Used to index and traverse all samples, is the final feature matrix of the jth sample, It is used to calculate the sum of the singular values ​​of the current feature matrix. The number of singular values ​​is the rank of the matrix. Summing the first loss and the rank loss to generate a final loss, and updating the parameters of the independent graph convolutional network according to the final loss until the training iteration is completed; The classification result generating unit is used to calculate the distance between the query feature vector and the support feature vector, and take the category of the support sample corresponding to the minimum distance as the classification result of the query tongue image.

4. A few-sample graph convolution tongue image classification device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a few-sample graph convolution tongue image classification method as described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a few-sample graph convolution tongue image classification method as described in any one of claims 1 to 2.

Images