Tongue image attribute recognition method and device based on constraint graph, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a tongue appearance attribute recognition method and device based on a constraint graph, equipment and a storage medium. The method comprises the following steps: inputting a constraint graph of a tongue appearance and training samples into a deep learning model, generating each predicted attribute label of a preset tongue appearance image through the deep learning model; generating the final probability of each predicted attribute label of the preset tongue appearance image through a pre-defined probability model and the suppression intensity of each predicted attribute label of the preset tongue appearance image; generating the total loss of the deep learning model on the training samples through a loss model, the suppression intensity of each predicted attribute label of the preset tongue appearance image and the final probability of each predicted attribute label of the preset tongue appearance image; determining the trained deep learning model based on the total loss; and generating the attribute recognition result of a current tongue appearance image through the trained deep learning model. The application can improve the generation efficiency of the attribute recognition result of the current tongue appearance image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for tongue image attribute recognition based on constraint graphs. Background Technology

[0002] In data processing technology, the current tongue image, as raw input data, exists in the form of a pixel matrix and cannot be directly understood by the data processing system. To enable the data processing system to use the information in the current tongue image, attribute recognition of the current tongue image is required to obtain the attribute recognition results.

[0003] However, in existing technologies, the attribute recognition results of current tongue images are obtained manually. Specifically, traditional Chinese medicine practitioners or trained professionals observe the current tongue image with the naked eye, determine its attributes, and manually record the attribute recognition results. This manual recognition method has the following technical drawbacks: First, it consumes significant human and time resources, significantly increasing the generation time of the attribute recognition results and hindering efficiency. Second, manual recognition suffers from poor consistency; different individuals may use different terms or have semantic biases in their attribute recognition results, and even the same individual may make inconsistent judgments at different times, thus compromising the stability of the attribute recognition results. Therefore, manual recognition is neither efficient nor stable in generating attribute recognition results for current tongue images. Thus, how to generate attribute recognition results for current tongue images is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for tongue image attribute recognition based on constraint graphs, in order to solve the aforementioned technical problem of how to generate attribute recognition results for the current tongue image.

[0005] In a first aspect, embodiments of this application provide a tongue image attribute recognition method based on constraint diagrams, applied to electronic devices, the tongue image attribute recognition method comprising: Obtain the preset tongue image and the attribute recognition results of the preset tongue image, and combine the preset tongue image and the attribute recognition results corresponding to the preset tongue image into training samples; Obtain the constraint map of the tongue image, input the constraint map of the tongue image and training samples into the deep learning model, and generate each predicted attribute label of the preset tongue image through the deep learning model. The inhibition strength of each predicted attribute label in a preset tongue image is generated using a predefined inhibition strength model. The final probability of each predicted attribute label of the preset tongue image is generated by using a predefined probability model and the suppression strength of each predicted attribute label of the preset tongue image. The total loss of the deep learning model on the training samples is generated by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model and a predefined method.

[0006] In one possible implementation of the first aspect, a constraint map of the tongue image is obtained, and the constraint map and training samples are input into a deep learning model. The deep learning model then generates a predicted attribute label for each pre-defined tongue image, including: Obtain multiple attribute dimensions of the tongue image, define each attribute value under each attribute dimension as a label node, and combine all label nodes into a label node set; The forbidden rules are extracted from the knowledge base. These rules are used to record information about tag nodes that cannot be active at the same time. Within the set of label nodes, two or more label nodes involved in the forbidden matching rule are combined into a forbidden matching combination. The two or more label nodes involved in the forbidden matching rule are connected in the form of a super edge to generate the super edge of the forbidden matching combination. All the super edges of the forbidden matching combinations are combined into a super edge set. The set of label nodes and the set of super edges are combined to generate the constraint graph of the tongue image. The constraint graph of the tongue image and the training samples are input into the deep learning model.

[0007] In one possible implementation of the first aspect, the total loss of the deep learning model on the training samples is generated using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. Then, using the trained deep learning model and a predefined method, the attribute recognition result of the current tongue image is generated, including: The total loss of the deep learning model on the training samples is generated by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. With the goal of reducing the total loss, train the deep learning model until the total loss is less than the preset loss value, then stop training the deep learning model and save the trained deep learning model. The current tongue image is acquired and input into the trained deep learning model. The trained deep learning model then generates the attribute recognition results of the current tongue image.

[0008] In one possible implementation of the first aspect, a current tongue image is acquired, and the current tongue image is input into a trained deep learning model. The trained deep learning model then generates attribute recognition results for the current tongue image, including: Send a connection request to the image acquisition device, receive the response message returned by the image acquisition device based on the connection request, parse the response message, and generate the connection result of the image acquisition device; When the connection result of the image acquisition device is successful, a acquisition command is sent to the image acquisition device, the current tongue image returned by the image acquisition device according to the acquisition command is received, the current tongue image is input into the trained deep learning model, and the attribute recognition result of the current tongue image is generated through the trained deep learning model.

[0009] In one possible implementation of the first aspect, the suppression strength model is defined as follows:

[0010] This represents the suppression strength of the i-th predicted attribute label of the preset tongue image. The greater the suppression strength of the i-th predicted attribute label of the preset tongue image, the lower the probability of the i-th predicted attribute label appearing in the preset tongue image; the smaller the suppression strength of the i-th predicted attribute label of the preset tongue image, the higher the probability of the i-th predicted attribute label appearing in the preset tongue image. This represents a preset tongue image; The set of superedges of the constraint graph includes the superedges of all forbidden combinations. A forbidden combination is a combination of multiple predicted attribute labels that should not appear at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... One forbidden combination; Indicates the first The superedges of a forbidden combination belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute labels other than the i-th predicted attribute label; This represents the initial probability of the predicted attribute labels other than the i-th predicted attribute label in the forbidden combination; Indicates multiplication.

[0011] In one possible implementation of the first aspect, the probabilistic model is defined as follows: ; This represents the final probability of the i-th predicted attribute label of the preset tongue image. The higher the final probability of the i-th predicted attribute label of the preset tongue image, the higher the probability that the preset tongue image has the i-th predicted attribute label. The lower the final probability of the i-th predicted attribute label of the preset tongue image, the lower the probability that the preset tongue image has the i-th predicted attribute label. This represents a preset tongue image; Represents the Sigmoid function; This represents the initial probability of the i-th predicted attribute label of the preset tongue image; This represents the correction amount of the i-th predicted attribute label of the preset tongue image. The correction amount of the i-th predicted attribute label of the preset tongue image is learned by the attention module and is used to adjust the initial probability of the i-th predicted attribute label of the preset tongue image. This indicates the preset strength coefficient; This represents the suppression strength of the i-th predicted attribute label of the preset tongue image.

[0012] In one possible implementation of the first aspect, the loss model is defined as follows: ; The value represents the total loss of the deep learning model on the training samples; N represents the total number of predicted attribute labels. This represents the observation mask for the i-th predicted attribute label; represents the confidence level of the i-th predicted attribute label; BCE represents the binary cross-entropy. This represents the annotation value of the true attribute label corresponding to the i-th predicted attribute label. The annotation value is the correct answer of manual annotation. The annotation value is 0 or 1. The annotation value of 1 indicates that the true attribute label corresponding to the i-th predicted attribute label exists. The annotation value of 0 indicates that the true attribute label corresponding to the i-th predicted attribute label does not exist. This represents the model's prediction probability for the i-th predicted attribute label; This represents a constant to prevent the denominator from being zero; , , These represent the preset first weighting coefficient, second weighting coefficient, and third weighting coefficient, respectively. This indicates a prohibited combination penalty, which is used to penalize combinations of predicted attribute labels that should not appear simultaneously. The set of hyperedges of the constraint graph includes the hyperedges of all forbidden combinations. A forbidden combination is a set of label nodes selected from the set of label nodes according to the forbidden combination rules, which should not be effective in the same sample at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... One forbidden combination; Indicates the first The superedges of a forbidden combination belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute label loss of the deep learning model. The smaller the predicted attribute label loss of the deep learning model, the closer the deviation between the joint distribution of predicted attribute labels predicted by the deep learning model and the preset joint distribution of predicted attribute labels is. The larger the predicted attribute label loss of the deep learning model, the farther the deviation between the joint distribution of predicted attribute labels predicted by the deep learning model and the preset joint distribution of predicted attribute labels is. This represents the consistency loss of the deep learning model. The greater the consistency loss, the more inconsistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The smaller the consistency loss, the more consistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The enhanced tongue image refers to the preset tongue image that has undergone image enhancement operations.

[0013] Secondly, embodiments of this application provide a tongue image attribute recognition device based on constraint diagrams, applied to electronic devices, including: The acquisition module is used to acquire the preset tongue image and the attribute recognition results of the preset tongue image, and to form training samples by combining the preset tongue image and the attribute recognition results corresponding to the preset tongue image. The input module is used to obtain the constraint map of the tongue image. The constraint map of the tongue image and the training samples are input into the deep learning model, and the deep learning model generates the predicted attribute label of each preset tongue image image. The first generation module is used to generate the suppression intensity of each predicted attribute label of a preset tongue image through a predefined suppression intensity model; The second generation module is used to generate the final probability of each predicted attribute label of the preset tongue image by using a predefined probability model and the suppression strength of each predicted attribute label of the preset tongue image. The recognition module is used to generate the total loss of the deep learning model on the training samples by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model and a predefined method.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the tongue image attribute recognition method described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the tongue image attribute recognition method described in the first aspect above.

[0016] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the tongue image attribute recognition method described in the first aspect.

[0017] The beneficial effects of the embodiments of this application are as follows: Firstly, by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image, the total loss of the deep learning model on the training samples is generated. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by the trained deep learning model and the predefined method. Since there is no need for manual recognition of the current tongue image, the generation time of the attribute recognition result of the current tongue image is reduced, which is conducive to improving the generation efficiency of the attribute recognition result of the current tongue image. Secondly, since the deep learning model, under the condition of fixed parameters, performs inference on the current tongue image at different time points and outputs the same attribute recognition result for the current tongue image, it is beneficial to improve the stability of the attribute recognition result of the current tongue image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application scenario diagram of the tongue image attribute recognition method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the tongue image attribute recognition method provided in the embodiments of this application; Figure 3 A flowchart for generating attribute recognition results provided in an embodiment of this application; Figure 4 A schematic block diagram of the tongue image attribute recognition device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0023] Furthermore, the technical solutions of the various embodiments can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0024] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0025] The tongue image attribute recognition method provided in this application embodiment can be applied to electronic devices, including but not limited to servers, mobile phones, tablets, wearable devices, vehicle-mounted devices, and laptops. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0026] Please see Figure 1 , Figure 1 The application scenario diagram of the tongue image attribute recognition method provided in the embodiments of this application is described in detail below: The electronic device connects to an object storage system and retrieves a preset tongue image and the attribute recognition results of the preset tongue image from the object storage system.

[0027] In this embodiment, the preset tongue image and its attribute recognition results are obtained from the object storage system without reading different data sources, which effectively simplifies the process of obtaining the preset tongue image and its attribute recognition results and improves the efficiency of obtaining the preset tongue image and its attribute recognition results.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating the tongue image attribute recognition method provided in this application embodiment, which can be applied to electronic devices.

[0029] like Figure 2 As shown, the tongue image attribute recognition method provided in this application includes the following steps, detailed below: S201, Obtain the preset tongue image and the attribute recognition results of the preset tongue image, and combine the preset tongue image and the attribute recognition results corresponding to the preset tongue image into training samples. The attribute recognition results of the preset tongue image include multiple real attribute labels of the preset tongue image. The preset tongue image is a tongue image that has already been recognized.

[0030] Tongue images, in particular, refer to digital images obtained by capturing images of the tongue using an image acquisition device. These images are stored as a pixel matrix and include the tongue region and surrounding oral tissues.

[0031] S202, Obtain the constraint map of the tongue image, input the constraint map of the tongue image and the training samples into the deep learning model, and generate each predicted attribute label of the preset tongue image through the deep learning model. This process involves obtaining a constraint map of the tongue image, inputting the constraint map and training samples into a deep learning model, and generating a predicted attribute label for each pre-defined tongue image through the deep learning model, including: Obtain multiple attribute dimensions of the tongue image, define each attribute value under each attribute dimension as a label node, and combine all label nodes into a label node set; The forbidden rules are extracted from the knowledge base. These rules are used to record information about tag nodes that cannot be active at the same time. Within the set of label nodes, two or more label nodes involved in the forbidden matching rule are combined into a forbidden matching combination. The two or more label nodes involved in the forbidden matching rule are connected in the form of a super edge to generate the super edge of the forbidden matching combination. All the super edges of the forbidden matching combinations are combined into a super edge set. The set of label nodes and the set of super edges are combined to generate the constraint graph of the tongue image. The constraint graph of the tongue image and the training samples are input into the deep learning model.

[0032] Among them, multiple attribute dimensions include tongue color, tongue shape, coating color, coating texture, tongue posture, saliva, and sublingual veins.

[0033] Among them, the attributes of tongue color include light red, red, crimson, purple, blue and light white; the attributes of tongue shape include large, thin, cracked, teeth marks and prickles; the attributes of tongue coating color include white, yellow, gray and black; the attributes of tongue coating texture include thick, thin, greasy, putrid, dry and moist; the attributes of tongue posture include stiff, weak, trembling, crooked and shortened; the attributes of body fluid include abundant body fluid, normal body fluid and scarce body fluid; and the attributes of sublingual veins include normal, thick, varicose and abnormal color.

[0034] S203, using a predefined inhibition strength model, generates the inhibition strength of each predicted attribute label of a preset tongue image; The inhibition strength model is defined as follows: ; This represents the suppression strength of the i-th predicted attribute label of the preset tongue image. The greater the suppression strength of the i-th predicted attribute label of the preset tongue image, the lower the probability of the i-th predicted attribute label appearing in the preset tongue image; the smaller the suppression strength of the i-th predicted attribute label of the preset tongue image, the higher the probability of the i-th predicted attribute label appearing in the preset tongue image. This represents a preset tongue image; The set of superedges of the constraint graph includes the superedges of all forbidden combinations. A forbidden combination is a combination of multiple predicted attribute labels that should not appear at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... One forbidden combination; Indicates the first The superedges of a forbidden combination belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute labels other than the i-th predicted attribute label; This represents the initial probability of the predicted attribute labels other than the i-th predicted attribute label in the forbidden combination; Indicates multiplication.

[0035] S204, using a predefined probability model and the suppression strength of each predicted attribute label of the preset tongue image, generate the final probability of each predicted attribute label of the preset tongue image; The probabilistic model is defined as follows: ; This represents the final probability of the i-th predicted attribute label of the preset tongue image. The higher the final probability of the i-th predicted attribute label of the preset tongue image, the higher the probability that the preset tongue image has the i-th predicted attribute label. The lower the final probability of the i-th predicted attribute label of the preset tongue image, the lower the probability that the preset tongue image has the i-th predicted attribute label. This represents a preset tongue image; Represents the Sigmoid function; This represents the initial probability of the i-th predicted attribute label of the preset tongue image; This represents the correction amount of the i-th predicted attribute label of the preset tongue image. The correction amount of the i-th predicted attribute label of the preset tongue image is learned by the attention module and is used to adjust the initial probability of the i-th predicted attribute label of the preset tongue image. This indicates the preset strength coefficient; This represents the suppression strength of the i-th predicted attribute label of the preset tongue image.

[0036] S205: By using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image, the total loss of the deep learning model on the training samples is generated. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model and a predefined method.

[0037] Specifically, the process involves generating the total loss of the deep learning model on the training samples using a predefined loss model, the suppression strength of each predicted attribute label in the preset tongue image, and the final probability of each predicted attribute label in the preset tongue image. Based on the total loss, the trained deep learning model is determined. Then, using the trained deep learning model and a predefined method, the attribute recognition result of the current tongue image is generated, including: The total loss of the deep learning model on the training samples is generated by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. With the goal of reducing the total loss, train the deep learning model until the total loss is less than the preset loss value, then stop training the deep learning model and save the trained deep learning model. The current tongue image is acquired and input into the trained deep learning model. The trained deep learning model then generates the attribute recognition results of the current tongue image.

[0038] The loss model is defined as follows: ; The value represents the total loss of the deep learning model on the training samples; N represents the total number of predicted attribute labels. This represents the observation mask for the i-th predicted attribute label; represents the confidence level of the i-th predicted attribute label; BCE represents the binary cross-entropy. This represents the annotation value of the true attribute label corresponding to the i-th predicted attribute label. The annotation value is the correct answer of manual annotation. The annotation value is 0 or 1. The annotation value of 1 indicates that the true attribute label corresponding to the i-th predicted attribute label exists. The annotation value of 0 indicates that the true attribute label corresponding to the i-th predicted attribute label does not exist. This represents the model's prediction probability for the i-th predicted attribute label; This represents a constant to prevent the denominator from being zero; , , These represent the preset first weighting coefficient, second weighting coefficient, and third weighting coefficient, respectively. This indicates a prohibited combination penalty, which is used to penalize combinations of predicted attribute labels that should not appear simultaneously. The set of hyperedges of the constraint graph includes the hyperedges of all forbidden combinations. A forbidden combination is a set of label nodes selected from the set of label nodes according to the forbidden combination rules, which should not be effective in the same sample at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... One forbidden combination; Indicates the first The superedges of a forbidden combination belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute label loss of the deep learning model. The smaller the predicted attribute label loss of the deep learning model, the closer the deviation between the joint distribution of predicted attribute labels predicted by the deep learning model and the preset joint distribution of predicted attribute labels is. The larger the predicted attribute label loss of the deep learning model, the farther the deviation between the joint distribution of predicted attribute labels predicted by the deep learning model and the preset joint distribution of predicted attribute labels is. This represents the consistency loss of the deep learning model. The greater the consistency loss, the more inconsistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The smaller the consistency loss, the more consistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The enhanced tongue image refers to the preset tongue image that has undergone image enhancement operations.

[0039] The predicted attribute label loss of the deep learning model is obtained through the mean squared error loss function. The process of obtaining the predicted attribute label loss of the deep learning model is detailed below: A preset tongue image is fed into a deep learning model. After forward propagation, the model outputs a prediction vector, which includes the predicted value for each dimension, with each value ranging from 0 to 1. Simultaneously, multiple ground truth attribute labels corresponding to the preset tongue image are acquired and encoded into a ground truth vector identical to the prediction vector, where each dimension has a value of 0 or 1. The prediction vector and the ground truth vector are input into a mean squared error loss function. This function calculates the squared difference between the predicted and ground truth values ​​for each dimension, sums the squared differences across all dimensions, and divides by the total number of dimensions to obtain a scalar value. This scalar value serves as the prediction attribute label loss for the deep learning model. A smaller prediction attribute label loss indicates a closer deviation between the joint distribution of predicted attribute labels and the preset joint distribution; a larger loss indicates a greater deviation.

[0040] The consistency loss of the deep learning model is obtained through the KL divergence loss function. The process of obtaining the consistency loss of the deep learning model is detailed below: A preset tongue image is generated, and image enhancement operations are performed on the preset tongue image to obtain an enhanced tongue image. The enhanced tongue image refers to the preset tongue image after image enhancement operations, which include one or more combinations of random rotation, random cropping, random brightness adjustment, random contrast adjustment, and random addition of Gaussian noise. The preset tongue image and the enhanced tongue image are used together as input to the deep learning model. A preset tongue image is input into a deep learning model. After forward propagation, the deep learning model outputs a probability distribution, denoted as the first distribution. An enhanced tongue image is then input into the deep learning model. After forward propagation, the deep learning model outputs another probability distribution, denoted as the second distribution. The first and second distributions have the same dimension, and the sum of all elements in their respective vectors is 1.

[0041] The first and second distributions are input into the KL divergence loss function. For each possible predicted attribute label, the function performs the following calculations: Calculate the logarithm of the probability values ​​of the second distribution on the predicted attribute labels, and the logarithm of the probability values ​​of the first distribution on the predicted attribute labels; subtract the logarithm of the probability values ​​of the first distribution on the predicted attribute labels from the logarithm of the second distribution on the predicted attribute labels to obtain the difference corresponding to the predicted attribute labels; multiply the probability value of the first distribution on the predicted attribute labels by the difference corresponding to the predicted attribute labels to obtain the loss component of the predicted attribute labels; sum the loss components of all predicted attribute labels and output a scalar value as the consistency loss of the deep learning model. The larger the consistency loss of the deep learning model, the more inconsistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image; the smaller the consistency loss of the deep learning model, the more consistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The enhanced tongue image refers to the preset tongue image that has undergone image enhancement operation.

[0042] The attribute recognition results of the current tongue image include multiple predicted attribute labels for the current tongue image. The multiple predicted attribute labels for the current tongue image are as follows: tongue color is pale red, tongue shape is swollen, tongue shape has teeth marks, tongue coating color is white, tongue coating texture is thick and greasy, and saliva is excessive.

[0043] The attribute recognition results of the current tongue image are used to convert the visual features in the image into structured attribute label data, which is then used by the data processing system for subsequent automated processing. Specifically, the attribute recognition results of the current tongue image have one or more of the following functions: First, it is used for structured storage. The attribute tags obtained from the identification, such as tongue color, tongue shape, tongue coating color, tongue coating texture, and body fluid, are written into the electronic medical record system or tongue image database, realizing the standardized recording of the attribute identification results of the current tongue image, replacing free text description.

[0044] Secondly, it is used for image classification and retrieval. Based on the attribute recognition results, the current tongue image is automatically classified and indexed, and it supports quick filtering by single attribute or attribute combination. For example, it can query all image records with a pale red tongue color and a thick, greasy coating.

[0045] The beneficial effects of the embodiments of this application are as follows: Firstly, by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image, the total loss of the deep learning model on the training samples is generated. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by the trained deep learning model and the predefined method. Since there is no need for manual recognition of the current tongue image, the generation time of the attribute recognition result of the current tongue image is reduced, which is conducive to improving the generation efficiency of the attribute recognition result of the current tongue image. Secondly, since the deep learning model, under the condition of fixed parameters, performs inference on the current tongue image at different time points and outputs the same attribute recognition result for the current tongue image, it is beneficial to improve the stability of the attribute recognition result of the current tongue image.

[0046] Please see Figure 3 , Figure 3 The flowchart for generating attribute recognition results provided in the embodiments of this application is described in detail below: S301, send a connection request to the image acquisition device, receive the response message returned by the image acquisition device according to the connection request, parse the response message, and generate the connection result of the image acquisition device; Image acquisition equipment refers to hardware devices used to acquire images of the tongue. Image acquisition equipment includes cameras and digital cameras.

[0047] S302, when the connection result of the image acquisition device is successful, send an acquisition command to the image acquisition device, receive the current tongue image returned by the image acquisition device according to the acquisition command, input the current tongue image into the trained deep learning model, and generate the attribute recognition result of the current tongue image through the trained deep learning model.

[0048] In this embodiment, the current tongue image is input into a trained deep learning model, and the trained deep learning model generates the attribute recognition result of the current tongue image, avoiding the bias caused by human experience and improving the consistency of the attribute recognition result of the current tongue image.

[0049] For the tongue image attribute recognition method described in the above embodiments, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic block diagram of the tongue image attribute recognition device provided in the embodiments of this application. Figure 4 The tongue image attribute recognition device 400 shown can be applied to, for example... Figure 1 The application scenario diagram shows electronic devices. The following section uses electronic devices as an example to illustrate this. Figure 4The tongue image attribute recognition device 400 shown will be described in detail. The tongue image attribute recognition device 400 may include an acquisition module 401, an input module 402, a first generation module 403, a second generation module 404, and a recognition module 405.

[0050] The acquisition module 401 is used to acquire the preset tongue image and the attribute recognition result of the preset tongue image, and to form a training sample by combining the preset tongue image and the attribute recognition result corresponding to the preset tongue image. The input module 402 is used to obtain the constraint map of the tongue image, input the constraint map of the tongue image and training samples into the deep learning model, and generate each predicted attribute label of the preset tongue image through the deep learning model. The first generation module 403 is used to generate the suppression intensity of each predicted attribute label of a preset tongue image through a predefined suppression intensity model; The second generation module 404 is used to generate the final probability of each predicted attribute label of the preset tongue image by using a predefined probability model and the suppression strength of each predicted attribute label of the preset tongue image. The recognition module 405 is used to generate the total loss of the deep learning model on the training samples by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model and a predefined method.

[0051] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0052] The beneficial effects of the embodiments of this application are as follows: Firstly, by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image, the total loss of the deep learning model on the training samples is generated. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by the trained deep learning model and the predefined method. Since there is no need for manual recognition of the current tongue image, the generation time of the attribute recognition result of the current tongue image is reduced, which is conducive to improving the generation efficiency of the attribute recognition result of the current tongue image. Secondly, since the deep learning model, under the condition of fixed parameters, performs inference on the current tongue image at different time points and outputs the same attribute recognition result for the current tongue image, it is beneficial to improve the stability of the attribute recognition result of the current tongue image.

[0053] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0054] like Figure 5 As shown, Figure 5 The electronic device includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 executes the computer program 22 to implement the steps in any of the above method embodiments.

[0055] The electronic device may include, but is not limited to, processor 20 and memory 21. Those skilled in the art will understand that... Figure 5 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0056] The processor 20 is used to run a computer program 22 stored in the memory 21, and performs the following steps when executing the computer program 22: Obtain the preset tongue image and the attribute recognition results of the preset tongue image, and combine the preset tongue image and the attribute recognition results corresponding to the preset tongue image into training samples; Obtain the constraint map of the tongue image, input the constraint map of the tongue image and training samples into the deep learning model, and generate each predicted attribute label of the preset tongue image through the deep learning model. The inhibition strength of each predicted attribute label in a preset tongue image is generated using a predefined inhibition strength model. The final probability of each predicted attribute label of the preset tongue image is generated by using a predefined probability model and the suppression strength of each predicted attribute label of the preset tongue image. The total loss of the deep learning model on the training samples is generated by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model and a predefined method.

[0057] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors, field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0058] In some embodiments, the memory 21 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 21 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device.

[0059] Furthermore, the memory 21 may include both internal storage units and external storage devices of the electronic device. The memory 21 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0060] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0061] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0062] The computer-readable storage medium may also be an external storage device of the tongue image attribute recognition device or electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, or non-transitory computer-readable storage medium equipped on the tongue image attribute recognition device or electronic device.

[0063] Since the computer program stored in the computer-readable storage medium can execute any of the constraint diagram-based tongue image attribute recognition methods provided in the embodiments of this application, the computer-readable storage medium can achieve the beneficial effects that any of the constraint diagram-based tongue image attribute recognition methods provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0064] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the above-described tongue image attribute recognition method.

[0065] When a computer program is loaded into an electronic device, it can perform the following steps: Obtain the preset tongue image and the attribute recognition results of the preset tongue image, and combine the preset tongue image and the attribute recognition results corresponding to the preset tongue image into training samples; Obtain the constraint map of the tongue image, input the constraint map of the tongue image and training samples into the deep learning model, and generate each predicted attribute label of the preset tongue image through the deep learning model. The inhibition strength of each predicted attribute label in a preset tongue image is generated using a predefined inhibition strength model. The final probability of each predicted attribute label of the preset tongue image is generated by using a predefined probability model and the suppression strength of each predicted attribute label of the preset tongue image. The total loss of the deep learning model on the training samples is generated by using a predefined loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model and a predefined method.

[0066] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0067] Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium includes: an entity or device for carrying computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium.

[0068] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0069] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for tongue image attribute recognition based on constraint graphs, characterized in that, The tongue image attribute recognition method, applied to electronic devices, includes: Obtain the preset tongue image and the attribute recognition results of the preset tongue image, and combine the preset tongue image and the attribute recognition results corresponding to the preset tongue image into training samples; Obtain the constraint map of the tongue image, input the constraint map of the tongue image and training samples into the deep learning model, and generate each predicted attribute label of the preset tongue image through the deep learning model. The suppression intensity model is used to generate the suppression intensity of each predicted attribute label of a preset tongue image. The final probability of each predicted attribute label of the preset tongue image is generated by using a probability model and the suppression strength of each predicted attribute label of the preset tongue image. The total loss of the deep learning model on the training samples is generated by using the loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. The attribute recognition result of the current tongue image is generated by using the trained deep learning model. The inhibition strength model is defined as follows: ； This represents the suppression strength of the i-th predicted attribute label of the preset tongue image. The greater the suppression strength of the i-th predicted attribute label of the preset tongue image, the lower the probability of the i-th predicted attribute label appearing in the preset tongue image; the smaller the suppression strength of the i-th predicted attribute label of the preset tongue image, the higher the probability of the i-th predicted attribute label appearing in the preset tongue image. This represents a preset tongue image; The set of superedges of the constraint graph includes the superedges of all forbidden combinations. A forbidden combination is a combination of multiple predicted attribute labels that should not appear at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... Forbidden combinations; Indicates the first The superedges of forbidden combinations belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute labels other than the i-th predicted attribute label; This represents the initial probability of the predicted attribute labels other than the i-th predicted attribute label in the forbidden combination; Indicates multiplication.

2. The tongue image attribute recognition method according to claim 1, characterized in that, Obtain the constraint map of the tongue image, input the constraint map and training samples into the deep learning model, and generate a predicted attribute label for each of the preset tongue image images through the deep learning model, including: Obtain multiple attribute dimensions of the tongue image, define each attribute value under each attribute dimension as a label node, and combine all label nodes into a label node set; The forbidden rules are extracted from the knowledge base. These rules are used to record information about tag nodes that cannot be active at the same time. Within the set of label nodes, two or more label nodes involved in the forbidden matching rule are combined into a forbidden matching combination. The two or more label nodes involved in the forbidden matching rule are connected in the form of a super edge to generate the super edge of the forbidden matching combination. All the super edges of the forbidden matching combinations are combined into a super edge set. The set of label nodes and the set of super edges are combined to generate the constraint graph of the tongue image. The constraint graph of the tongue image and the training samples are input into the deep learning model.

3. The tongue image attribute recognition method according to claim 1, characterized in that, The total loss of the deep learning model on the training samples is generated by using the loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined. Using the trained deep learning model, the attribute recognition results of the current tongue image are generated, including: The total loss of the deep learning model on the training samples is generated by using the loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. With the goal of reducing the total loss, train the deep learning model until the total loss is less than the preset loss value, then stop training the deep learning model and save the trained deep learning model. The current tongue image is acquired and input into the trained deep learning model. The trained deep learning model then generates the attribute recognition results of the current tongue image.

4. The tongue image attribute recognition method according to claim 3, characterized in that, The current tongue image is acquired and input into a trained deep learning model. The trained deep learning model then generates attribute recognition results for the current tongue image, including: Send a connection request to the image acquisition device, receive the response message returned by the image acquisition device based on the connection request, parse the response message, and generate the connection result of the image acquisition device; When the connection result of the image acquisition device is successful, a acquisition command is sent to the image acquisition device, the current tongue image returned by the image acquisition device according to the acquisition command is received, the current tongue image is input into the trained deep learning model, and the attribute recognition result of the current tongue image is generated through the trained deep learning model.

5. The tongue image attribute recognition method according to claim 1, characterized in that, The probabilistic model is defined as follows: ； This represents the final probability of the i-th predicted attribute label of the preset tongue image. The higher the final probability of the i-th predicted attribute label of the preset tongue image, the higher the probability that the preset tongue image has the i-th predicted attribute label. The lower the final probability of the i-th predicted attribute label of the preset tongue image, the lower the probability that the preset tongue image has the i-th predicted attribute label. This represents a preset tongue image; Represents the Sigmoid function; This represents the initial probability of the i-th predicted attribute label of the preset tongue image; This represents the correction amount of the i-th predicted attribute label of the preset tongue image. The correction amount of the i-th predicted attribute label of the preset tongue image is learned by the attention module and is used to adjust the initial probability of the i-th predicted attribute label of the preset tongue image. This indicates the preset strength coefficient; This represents the suppression strength of the i-th predicted attribute label of the preset tongue image.

6. The tongue image attribute recognition method according to claim 1, characterized in that, The loss model is defined as follows: ； The value represents the total loss of the deep learning model on the training samples; N represents the total number of predicted attribute labels. Represents the observation mask for the i-th predicted attribute label; represents the confidence level of the i-th predicted attribute label; BCE represents the binary cross-entropy. This represents the annotation value of the true attribute label corresponding to the i-th predicted attribute label. The annotation value is the correct answer of manual annotation. The annotation value is 0 or 1. The annotation value of 1 indicates that the true attribute label corresponding to the i-th predicted attribute label exists. The annotation value of 0 indicates that the true attribute label corresponding to the i-th predicted attribute label does not exist. This represents the model's prediction probability for the i-th predicted attribute label; This represents a constant to prevent the denominator from being zero; , , These represent the preset first weighting coefficient, second weighting coefficient, and third weighting coefficient, respectively. This indicates a prohibited combination penalty, which is used to penalize combinations of predicted attribute labels that should not appear simultaneously. The set of hyperedges of the constraint graph includes the hyperedges of all forbidden combinations. A forbidden combination is a set of label nodes selected from the set of label nodes according to the forbidden combination rules, which should not be effective in the same sample at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... Forbidden combinations; Indicates the first The superedges of forbidden combinations belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute label loss of the deep learning model. The smaller the predicted attribute label loss of the deep learning model, the closer the deviation between the joint distribution of predicted attribute labels predicted by the deep learning model and the preset joint distribution of predicted attribute labels is. The larger the predicted attribute label loss of the deep learning model, the farther the deviation between the joint distribution of predicted attribute labels predicted by the deep learning model and the preset joint distribution of predicted attribute labels is. This represents the consistency loss of the deep learning model. The greater the consistency loss, the more inconsistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The smaller the consistency loss, the more consistent the output of the deep learning model is on the preset tongue image and the enhanced tongue image. The enhanced tongue image refers to the preset tongue image that has undergone image enhancement operations.

7. A tongue image attribute recognition device based on constraint graphs, characterized in that, Applied to electronic devices, including: The acquisition module is used to acquire the preset tongue image and the attribute recognition results of the preset tongue image, and to form training samples by combining the preset tongue image and the attribute recognition results corresponding to the preset tongue image. The input module is used to obtain the constraint map of the tongue image. The constraint map of the tongue image and the training samples are input into the deep learning model, and the deep learning model generates the predicted attribute label of each preset tongue image image. The first generation module is used to generate the suppression intensity of each predicted attribute label of the preset tongue image through the suppression intensity model; The second generation module is used to generate the final probability of each predicted attribute label of the preset tongue image by using a probability model and the suppression strength of each predicted attribute label of the preset tongue image. The recognition module is used to generate the total loss of the deep learning model on the training samples by using the loss model, the suppression strength of each predicted attribute label of the preset tongue image, and the final probability of each predicted attribute label of the preset tongue image. Based on the total loss, the trained deep learning model is determined, and the attribute recognition result of the current tongue image is generated by the trained deep learning model. The inhibition strength model is defined as follows: ； This represents the suppression strength of the i-th predicted attribute label of the preset tongue image. The greater the suppression strength of the i-th predicted attribute label of the preset tongue image, the lower the probability of the i-th predicted attribute label appearing in the preset tongue image; the smaller the suppression strength of the i-th predicted attribute label of the preset tongue image, the higher the probability of the i-th predicted attribute label appearing in the preset tongue image. This represents a preset tongue image; The set of superedges of the constraint graph includes the superedges of all forbidden combinations. A forbidden combination is a combination of multiple predicted attribute labels that should not appear at the same time. Indicates the sequence number of the forbidden combination; This indicates that the i-th predicted attribute label belongs to the i-th... Forbidden combinations; Indicates the first The superedges of forbidden combinations belong to the set of superedges of the superedge-constrained graph; Indicates the first The constraint strength of each forbidden combination; Indicates the first The activation coefficients of each forbidden combination on a preset tongue image; This represents the predicted attribute labels other than the i-th predicted attribute label; This represents the initial probability of the predicted attribute labels other than the i-th predicted attribute label in the forbidden combination; Indicates multiplication.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the tongue image attribute recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the tongue image attribute recognition method as described in any one of claims 1 to 6.

Images