Constitution multi-label classification method, device and medium

Abstract

The embodiment of the application provides a body constitution multi-label classification method, equipment and medium, feature extraction is performed on a preprocessed image of a tongue image; global features are obtained by a first feature processing branch according to first image features and embedded features of labels, local features are obtained by a second feature processing branch according to the first image features, feature interaction is performed by transferring the local features to the first feature processing branch and transferring the global features to the second feature processing branch, and the global features and the local features are fused to obtain fused features; classification is performed by using the fused features; dynamic interaction features enable the model to better understand the correlation and dependence between different features, enable the second feature processing branch to extract features from bottom to top for the first feature processing branch, and enable the first feature processing branch to guide feature extraction of the second feature processing branch through a top-down attention mechanism, thereby improving the generalization capability of classification.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to, but are not limited to, the field of information processing, and in particular to a constitution multi-label classification method, device and medium. BACKGROUND

[0002] The current constitution classification usually adopts single-label classification, that is, each individual is divided into a unique constitution type. In single-label classification, each individual can only be classified into a specific constitution type, and cannot belong to multiple types at the same time. The multi-label classification method can better consider the mutual relationship between multiple labels, and improve the accuracy and individualization degree of constitution classification. However, the multi-label classification method also puts forward higher requirements for the classification model, and the features concerned by the model need to be more comprehensive. SUMMARY

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of the present application is to at least partially solve one of the technical problems existing in the related art. The constitution multi-label classification method, device and medium provided by the embodiments of the present application improve the model generalization ability through interaction features.

[0005] In an embodiment of the first aspect of the present application, a constitution multi-label classification method comprises:

[0006] obtaining a tongue image, the tongue image being labeled with a label;

[0007] performing tongue body segmentation processing and tongue fur and tongue substance separation processing on the tongue image to obtain a preprocessed image;

[0008] performing feature extraction on the preprocessed image to obtain a first image feature;

[0009] obtaining a global feature from the first image feature and an embedding feature of the label by a first feature processing branch, obtaining a local feature by convolution on the first image feature by a second feature processing branch, performing feature interaction by transferring the local feature to the first feature processing branch and transferring the global feature to the second feature processing branch, and fusing the global feature and the local feature to obtain a fused feature;

[0010] establishing a mapping relationship between the fused feature and a plurality of constitution type labels of a class to obtain constitution data;

[0011] training a classification model according to the constitution data;

[0012] inputting an image to be classified into the classification model for classification to obtain a classification result.

[0013] According to certain embodiments of the first aspect of the application, the obtaining the global feature according to the first image feature and the embedding feature of the label comprises:

[0014] obtaining a self-attention value according to the embedding vector of the label;

[0015] obtaining a cross-attention value according to the first image feature and the self-attention value;

[0016] performing a multi-scale pooling operation on the first image feature and the embedding vector of the label to obtain a plurality of features of different scales, and performing linear combination on the plurality of features of different scales to obtain a multi-scale pooled feature;

[0017] performing full connection on the self-attention value, the cross-attention value and the multi-scale pooled feature to obtain the global feature.

[0018] According to certain embodiments of the first aspect of the application, the obtaining the self-attention value according to the embedding vector of the label comprises:

[0019] taking the embedding vector of the label as keys K, values V and queries Q of a multi-head attention module, and obtaining the self-attention value according to a multi-head attention mechanism.

[0020] According to certain embodiments of the first aspect of the application, the obtaining the cross-attention value according to the first image feature and the self-attention value comprises:

[0021] taking the first image feature as keys K and values V of a multi-head attention module, taking the self-attention value as queries Q of the multi-head attention module, and obtaining the cross-attention value according to a multi-head attention mechanism.

[0022] According to certain embodiments of the first aspect of the application, the performing the multi-scale pooling operation on the first image feature and the embedding vector of the label to obtain a plurality of features of different scales, and performing linear combination on the plurality of features of different scales to obtain a multi-scale pooled feature comprises:

[0023] taking the embedding vector of the label as keys K, values V and queries Q of a multi-head attention module, performing different size pooling operations on the keys K, values V and queries Q to obtain a first pooled feature of the keys K, a second pooled feature of the values V and a third pooled feature of the queries Q;

[0024] cross-multiplying the first pooled feature of the keys K and the third pooled feature of the queries Q to obtain a fourth pooled feature and a fifth pooled feature;

[0025] adding the second pooled feature of the values V one by one to obtain a sixth pooled feature;

[0026] normalizing the fourth pooled feature, the fifth pooled feature and the sixth pooled feature to obtain normalized values;

[0027] multiplying the normalized values and the sixth pooled feature as weights, adding the fifth pooled feature as a residual, performing linear combination to obtain a multi-scale pooled feature.

[0028] According to some embodiments of the first aspect of the application, the convolution according to the first image feature to obtain a local feature comprises:

[0029] projective convolution is performed on the first image feature to obtain a first convolution value;

[0030] spatial convolution is performed on the first convolution value to obtain a second convolution value;

[0031] projective convolution is performed on the second convolution value to obtain a third convolution value;

[0032] the third convolution value and the first image feature are connected in residual to obtain a local feature.

[0033] According to some embodiments of the first aspect of the application, the passing of the local feature to the first feature processing branch comprises:

[0034] convolution is performed on the local feature to convert dimensions to obtain a first local feature;

[0035] different channels of the first local feature are combined to obtain a second local feature;

[0036] average pooling is performed on the second local feature to downsample to obtain a third local feature;

[0037] the third local feature is processed by a first activation function and layer normalization to obtain a feature passed to the first feature processing branch.

[0038] According to some embodiments of the first aspect of the application, the passing of the global feature to the second feature processing branch comprises:

[0039] the global feature is processed by a second activation function and batch normalization to obtain a first global feature;

[0040] the first global feature is upsampled by bilinear interpolation to obtain a second global feature;

[0041] the first global feature and the second global feature are connected in residual to obtain a feature passed to the second feature processing branch.

[0042] In an embodiment of the second aspect of the present application, an electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the constitution method of constitution multi-label classification when executing the computer program.

[0043] In an embodiment of the third aspect of the present application, a computer storage medium stores computer executable instructions for executing the constitution method of constitution multi-label classification.

[0044] The above scheme has at least the following beneficial effects: feature extraction is performed on the preprocessed image of the tongue image; global features are obtained from the first image features and the embedding features of the labels by the first feature processing branch, local features are obtained by convolution of the first image features by the second feature processing branch, feature interaction is performed by passing the local features to the first feature processing branch and passing the global features to the second feature processing branch, fusion features are obtained by fusing the global features and the local features; classification is performed using the fusion features; dynamic interaction of features enables the model to better understand the correlation and dependence between different features, enables the second feature processing branch to extract features for the first feature processing branch from bottom to top, and enables the first feature processing branch to guide the feature extraction of the second feature processing branch through a top-down attention mechanism, thereby improving the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and together with the embodiments of the present application are used to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0046] Figure 1 is a step diagram of the constitution method of constitution multi-label classification provided by the embodiments of the present application;

[0047] Figure 2 is a sub-step diagram of obtaining global features from the first image features and the embedding features of the labels;

[0048] Figure 3 is a sub-step diagram of step S413;

[0049] Figure 4 is a sub-step diagram of obtaining local features by convolution of the first image features;

[0050] Figure 5 is a sub-step diagram of passing the local features to the first feature processing branch;

[0051] Figure 6 is a sub-step diagram of passing the global features to the second feature processing branch;

[0052] Figure 7 is a schematic diagram of a model using a body constitution multi-label classification method;

[0053] Figure 8 is a schematic diagram of a multi-scale pooling network;

[0054] Figure 9 is a schematic diagram of a C2T network;

[0055] Figure 10 is a schematic diagram of a T2C network. DETAILED DESCRIPTION

[0056] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0057] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0058] The embodiments of the present application will be further described below with reference to the drawings.

[0059] The embodiments of the present application provide a body constitution multi-label classification method.

[0060] Referring to Figure 1 , the body constitution multi-label classification method includes but is not limited to the following steps:

[0061] Step S100, acquiring a tongue image;

[0062] Step S200, performing tongue body segmentation processing and tongue fur and tongue body separation processing on the tongue image to obtain a preprocessed image;

[0063] Step S300, performing feature extraction on the preprocessed image to obtain a first image feature;

[0064] Step S400, obtaining a global feature according to the first image feature and an embedding feature of a label through a first feature processing branch, obtaining a local feature through a second feature processing branch according to the first image feature, performing feature interaction by transferring the local feature to the first feature processing branch and transferring the global feature to the second feature processing branch, and fusing the global feature and the local feature to obtain a fused feature;

[0065] Step S500, a mapping relationship between the fusion features and the body constitution type labels of the plurality of categories is established to obtain body constitution data;

[0066] Step S600, a classification model is trained according to the body constitution data;

[0067] Step S700, the image to be classified is input into the classification model for classification to obtain a classification result.

[0068] For step S100, the original tongue image usually includes a face part, a lip part and a tongue part. The tongue image can be directly obtained from a large image database. The tongue images in the large image database have been artificially labeled with labels.

[0069] The tongue image can also be obtained by photographing the tongues of different people by a photographing device. The tongue image is an image containing a tongue obtained by photographing the tongues of different people in different environments by a photographing device such as a mobile phone, a camera or a tongue image instrument. Different environments include different lighting conditions, different time periods and other environmental conditions. These video data are used to train the model so as to improve the generalization and robustness of the model to changes in various factors such as lighting conditions, video image size and background information.

[0070] The tongue images in the data set are labeled using the LabelMe tool. The labeling information of the tongue images is converted to obtain a labeled image corresponding to each tongue image. The background area and the tongue area in the labeled image are distinguished by different pixel values.

[0071] It can be understood that the label is a description of the constitution corresponding to the tongue image.

[0072] For step S200, the tongue image is preprocessed to obtain a preprocessed image. The preprocessing can include image cropping, image rotation, tongue segmentation processing and tongue fur and tongue substance separation processing, so as to realize the standardization processing of the original tongue image.

[0073] The tongue segmentation processing and the tongue fur and tongue substance separation processing of the tongue image can be performed by a basic feature extraction network of a UNet structure. The basic feature extraction network includes a plurality of cascaded encoders and a plurality of cascaded decoders, and the encoders and the decoders are one-to-one jump connected.

[0074] Referring to Figure 7 , the Stem module is a basic feature extraction network.

[0075] For step S300, the preprocessed image is input into the basic model for simple feature extraction to obtain rough first image features.

[0076] Specifically, the preprocessed image is subjected to 7x7 convolution and 3x3 max pooling for feature extraction to obtain the first image features.

[0077] For step S400, refer to Figure 2 , the global feature is obtained by the first feature processing branch according to the first image feature and the embedding feature of the label, including but not limited to the following steps:

[0078] Step S411, obtaining a self-attention value according to the embedding vector of the label;

[0079] Step S412, obtaining a cross-attention value according to the first image feature and the self-attention value;

[0080] Step S413, performing a multi-scale pooling operation on the first image feature and the embedding vector of the label to obtain a plurality of features of different scales, and performing linear combination on the plurality of features of different scales to obtain a multi-scale pooling feature;

[0081] Step S414, performing full connection on the self-attention value, the cross-attention value and the multi-scale pooling feature to obtain the global feature.

[0082] The first feature processing branch is a Transformer architecture.

[0083] The embedding vector of the label (Label embedding) can be obtained by embedding the tongue image. The embedding vector is a representation of an entity, which can also be regarded as a set of features extracted from the entity. Entities such as words, objects and labels in the sample can be converted into embedding vectors and then sent to the subsequent part of the network. In the embedding layer, the embedding vector of the label is retrieved from a large embedding vector table according to the id of the label and output to the subsequent network.

[0084] For step S411, the self-attention value is obtained according to the embedding vector of the label, specifically:

[0085] The embedding vector of the label is used as the key K, value V and query Q of the multi-head attention module, and the self-attention value is obtained according to the multi-head attention mechanism.

[0086] The key K, value V and query Q of the multi-head attention module are all embedding vectors of the label, and the key K, value V and query Q are respectively represented as: Q′ i-1 , Q″ i-1 and Q i-1 ; then the self-attention value is represented as:

[0087] When using the self-attention module, the embedding vector of the label is converted into a vector representation of the label so that the computer can better understand and process them. By embedding the label into the first feature processing branch, the model can effectively automatically capture the semantics of the label and make more accurate predictions for multiple labels related to the input sample.

[0088] For step S412, cross attention values are obtained according to the first image feature and the self-attention value, specifically:

[0089] The first image feature is taken as the key K and the value V of the multi-head attention module, and the embedding vector of the label is taken as the query Q of the multi-head attention module, and the cross attention value is obtained according to the multi-head attention mechanism.

[0090] The key K and the value V of the multi-head attention module are the first image feature, and the query Q of the multi-head attention module is the self-attention value, so the cross attention value is represented as:

[0091] The relationship between the visual feature and the semantic vector of the label is mined by extracting the self-attention value and the cross attention value.

[0092] Referring to Figure 8 , Figure 8 is a schematic diagram of a multi-scale pooling network (MSP).

[0093] Referring to Figure 3 For step S413, multi-scale pooling operations are performed on the first image feature and the embedding vector of the label to obtain a plurality of features of different scales, and linear combination is performed on the plurality of features of different scales to obtain a multi-scale pooling feature, including but not limited to the following steps:

[0094] Step S4131, the embedding vector of the label is taken as the key K, the value V and the query Q of the multi-head attention module, and different size pooling operations are performed on the key K, the value V and the query Q to obtain the first pooling feature of the key K, the second pooling feature of the value V and the third pooling feature of the query Q.

[0095] Step S4132, the first pooling feature of the key K and the third pooling feature of the query Q are cross-multiplied to obtain the fourth pooling feature and the fifth pooling feature:

[0096] Step S4133, the second pooling feature of the value V is added one by one to obtain the sixth pooling feature.

[0097] Step S4134, the fourth pooling feature, the fifth pooling feature and the sixth pooling feature are normalized to obtain a normalized value.

[0098] Step S4135, the normalized value is multiplied as a weight with the sixth pooling feature, and the fifth pooling feature is added as a residual to perform linear combination to obtain the multi-scale pooling feature.

[0099] In this embodiment, the query Q is subjected to a 2x2 pooling operation to obtain the third pooling feature Pool qA 3x3 pooling operation is performed on the key K to obtain a first pooling feature Pool of the key K k A 4x4 pooling operation is performed on the value V to obtain a second pooling feature Pool of the value V v Each pooling layer is allowed to extract different levels of features, and then the features are fused together.

[0100] Different scales of pooling, such as the first pooling feature Pool of the key K k and the third pooling feature Pool of the query Q q are cross-multiplied to obtain a fourth pooling feature Pool' q and a fifth pooling feature Pool' k The second pooling feature Pool of the value V v is added one by one to obtain a sixth pooling feature Pool' v Then

[0101] The fourth pooling feature, the fifth pooling feature, and the sixth pooling feature are normalized by a Softmax function to obtain normalized values. The normalized values are multiplied by the sixth pooling feature as weights, and the fifth pooling feature is added as a residual to obtain a multi-scale pooling feature through linear combination. Then where d k represents the dimension.

[0102] For step S414, the self-attention value, the cross-attention value, and the multi-scale pooling feature are subjected to information correlation full connection (position-wise feed-forward network, FFN) to obtain a global feature.

[0103] The Transformer model can model the automatic learning of label relationships in the input sequence in the self-attention mechanism, and can better understand the association and dependence between different features through cross-attention.

[0104] Referring to Figure 4 , the local feature is obtained by convolving the first image feature according to the second feature processing branch, including but not limited to the following steps:

[0105] Step S421, projection convolution is performed on the first image feature to obtain a first convolution value;

[0106] Step S422, spatial convolution is performed on the first convolution value to obtain a second convolution value;

[0107] Step S423, projection convolution is performed on the second convolution value to obtain a third convolution value;

[0108] Step S424, the third convolution value is connected with the first image feature to obtain a local feature.

[0109] The second feature processing branch is a CNN architecture with an inner branch (IB) structure, in which the resolution of the feature map decreases with the increase of the network depth.

[0110] Firstly, the basic bottleneck block of ResNet network is adopted, which usually consists of three convolution layers. The first convolution layer uses a small 1x1 projection convolution to reduce the dimension of the feature map. The second convolution layer uses a larger 3x3 spatial convolution to extract features. The third convolution layer again uses a small 1x1 projection convolution to further reduce the dimension of the feature map. It also includes a residual connection between the input and output. The IB structure modifies the residual block by introducing an additional nested branch, replacing the main 3x3 convolution with a residual block, which greatly enhances the feature representation ability of the model.

[0111] Referring to Figure 9 , Figure 9 is a schematic diagram of the C2T network.

[0112] Referring to Figure 5 , the local feature is transmitted to the first feature processing branch, including but not limited to the following steps:

[0113] Step S431, the local feature is convolved to convert the dimension to obtain a first local feature;

[0114] Step S432, combining different channels of the first local feature to obtain a second local feature;

[0115] Step S433, performing average pooling on the second local feature to downsample to obtain a third local feature;

[0116] Step S434, processing the third local feature through a first activation function and layer normalization to obtain a feature transmitted to the first feature processing branch.

[0117] The format of the local feature is [B, C, H, W], where B represents the batch size, C represents the number of channels, H represents the image height, and W represents the image width.

[0118] In this embodiment, 1x1 convolution is used to convert the dimension of the feature map. In addition, the features of different channels are combined to improve the expressiveness of the features. Average pooling and reshaping are used to downsample the feature map, which reduces its spatial size while preserving the basic information. GELU activation function is used to achieve fast convergence and reduce training time, thereby improving the efficiency of model training. Layer normalization is used for feature regularization.

[0119] Referring to Figure 10, Figure 10 is a schematic diagram of a T2C network.

[0120] Referring to Figure 6 , the global feature is delivered to the second feature processing branch, including but not limited to the following steps:

[0121] Step S441, processing the global feature through a second activation function and batch normalization to obtain a first global feature;

[0122] Step S442, upsampling the first global feature through bilinear interpolation to obtain a second global feature;

[0123] Step S443, performing residual connection on the first global feature and the second global feature to obtain a feature delivered to the second feature processing branch.

[0124] The format of the global feature is [B,_,C], wherein '_' represents the sum of the number of image patches and the number of class labels, which is usually equal to HxW+1.

[0125] The global feature is inverted and reshaped before step S441 to form a format of [B,C,H,W].

[0126] In this embodiment, the spatial scale is adjusted by using appropriate upsampling techniques. Batch normalization is used to normalize the features. The commonly used ReLU activation function is used in the convolution operation. Bilinear interpolation is used to upsample the feature map to enhance the spatial resolution and capture finer details. Multiple 1x1 convolution operations are used to optimize feature information exchange. After 1x1 convolution, ReLU and Sigmoid activation are included in a cross manner to improve the nonlinear fitting ability. Finally, a residual connection is introduced between the output of bilinear interpolation and the output after a series of operations to preserve important information and improve the performance of the model, and an embedded residual structure is used to learn more detailed information.

[0127] Delivering the local feature to the first feature processing branch and the global feature to the second feature processing branch, the CNN can provide richer input for the Transformer through bottom-up feature extraction, and the Transformer can guide the feature extraction of the CNN through top-down attention mechanism, and gradually integrate the feature maps through interactive way.

[0128] Both CNN and Transformer have strong generalization ability and can process input data of different sizes and scales. In traditional Chinese constitution classification, individual data samples may have different feature representations and scales. CNN can effectively process spatial local features, while Transformer can process sequence features. Therefore, combining CNN and Transformer can better adapt to and process different scales and types of traditional Chinese constitution data, and improve the generalization ability of classification.

[0129] For step S500, a mapping relationship between the fusion features and the constitution type labels of multiple categories is established to obtain constitution data. The fusion features include tongue color, tongue image features, etc.

[0130] For step S600, a classification model is trained according to the constitution data. The classification model can be based on a deep learning model.

[0131] For step S700, a real-time image of the tongue image of the human body is obtained by a photographic device as a to-be-classified image, and the to-be-classified image is input into the classification model for classification to obtain a classification result.

[0132] The classification result is one or more of the following constitutions: normal constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi depression constitution, and special constitution.

[0133] Embodiments of the present application provide an electronic device. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the constitution multi-label classification method described above.

[0134] In general, for the hardware structure of the electronic device, the processor can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0135] The memory can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory and called and executed by the processor to implement the interface information processing method of the embodiments of the present application.

[0136] The input / output interface is configured to realize information input and output.

[0137] The communication interface is configured to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0138] The bus transmits information between various components (for example, the processor, the memory, the input / output interface and the communication interface) of the device. The processor, the memory, the input / output interface and the communication interface are connected to each other through the bus to realize the communication connection between them in the device.

[0139] Embodiments of the present application provide a computer storage medium. The computer storage medium stores computer executable instructions. The computer executable instructions are used to execute the constitution multiple label classification method.

[0140] Those of ordinary skill in the art will appreciate that all or some steps, systems of the above disclosed methods can be implemented as software, firmware, hardware, or appropriate combinations of them. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it should be appreciated by those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. In the foregoing description of the present specification, reference has been made to descriptive terms such as "one embodiment / implementation", "another embodiment / implementation", or "some embodiments / implementations" etc. It is appreciated that such descriptive terms are not intended to necessarily limit or restrict the scope of the embodiments or examples to the described embodiments / examples, but are merely used to connect a feature described in connection with one embodiment or example with one or more other embodiments or examples. Furthermore, it is also understood that the described features are not necessarily limited to a single embodiment or example, but can be combined with each other or with other features of one or more embodiments or examples.

[0141] Those of ordinary skill in the art will appreciate that all or some steps, systems, functional modules / units in the above disclosed methods can be implemented as software, firmware, hardware, or appropriate combinations of them.

[0142] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0143] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or software function units.

[0144] When the integrated unit is implemented in the form of software function units and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or parts contributing to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program storage media.

[0145] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other ways. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0146] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method of body mass multi-label classification, the method comprising: The method comprises the following steps: acquiring a tongue image, wherein the tongue image is labeled with a label; performing tongue body segmentation processing and tongue fur and tongue body separation processing on the tongue image to obtain a pretreated image; extracting features from the pretreated image to obtain first image features; obtaining global features from the first image features and embedded features of the label through a first feature processing branch, obtaining local features through convolution of the first image features through a second feature processing branch, performing feature interaction by transferring the local features to the first feature processing branch and transferring the global features to the second feature processing branch, and fusing the global features and the local features to obtain fused features; establishing a mapping relationship between the fused features and a plurality of category constitution type labels to obtain constitution data; training a classification model according to the constitution data; inputting an image to be classified into the classification model for classification to obtain a classification result; wherein the global features are obtained from the first image features and the embedded features of the label, comprising: obtaining a self-attention value from an embedded vector of the label; obtaining a cross-attention value from the first image features and the self-attention value; performing multi-scale pooling operation on the first image features and the embedded vector of the label to obtain a plurality of features of different scales, and linearly combining the plurality of features of different scales to obtain multi-scale pooling features; fully connecting the self-attention value, the cross-attention value and the multi-scale pooling features to obtain global features; the embedded vector of the label is taken as the key K, the value V and the query Q of the multi-head attention module, and the self-attention value is obtained according to the multi-head attention mechanism; the first image features are taken as the key K and the value V of the multi-head attention module, the self-attention value is taken as the query Q of the multi-head attention module, and the cross-attention value is obtained according to the multi-head attention mechanism; the embedded vector of the label is taken as the key K, the value V and the query Q of the multi-head attention module, and different size pooling operations are performed on the key K, the value V and the query Q to obtain the first pooling feature of the key K, the second pooling feature of the value V and the third pooling feature of the query Q; the first pooling feature of the key K and the third pooling feature of the query Q are cross-multiplied to obtain the fourth pooling feature and the fifth pooling feature; the second pooling features of the value V are added one by one to obtain the sixth pooling feature; the fourth pooling feature, the fifth pooling feature and the sixth pooling feature are normalized to obtain a normalized value; the normalized value is taken as a weight and multiplied by the sixth pooling feature, and the fifth pooling feature is taken as a residual and added, linearly combined to obtain multi-scale pooling features; the first image features are projected and convolved to obtain a first convolution value; ​ ​ ​ ​ performing spatial convolution on the first convolution value to obtain a second convolution value; performing projection convolution on the second convolution value to obtain a third convolution value; performing residual connection on the third convolution value and the first image feature to obtain a local feature.

2. The method of claim 1, wherein, The transmitting the local feature to the first feature processing branch includes: performing convolution on the local feature to convert dimensions to obtain a first local feature; combining different channels of the first local feature to obtain a second local feature; performing average pooling on the second local feature to down-sample to obtain a third local feature; processing the third local feature through a first activation function and layer normalization to obtain a feature transmitted to the first feature processing branch.

3. The method of claim 1, wherein, The transmitting the global feature to the second feature processing branch includes: processing the global feature through a second activation function and batch normalization to obtain a first global feature; performing up-sampling on the first global feature through bilinear interpolation to obtain a second global feature; performing residual connection on the first global feature and the second global feature to obtain a feature transmitted to the second feature processing branch.

4. An electronic device, comprising: The method includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the body constitution multi-label classification method according to any one of claims 1 to 3.

5. A computer storage medium, characterized in that a computer executable instruction for executing the body constitution multi-label classification method according to any one of claims 1 to 3.

Images