Model training method and apparatus, and storage medium

Abstract

The application discloses a model training method and device and a storage medium. The method comprises the following steps: establishing a first model by using a neural network; acquiring first sample picture data; performing resampling processing and weight increasing processing on specific data in the first sample picture data to obtain second sample picture data and third sample picture data; and training the first model by using the first sample picture data, the second sample picture data and the third sample picture data. According to the scheme provided in the application, the contribution of specific data in sample picture data is enhanced by performing resampling and weight increasing processing on the sample picture data and training the first model by using the obtained sample picture data, the resampled sample picture data and the sample picture data with increased weight, so that the classification performance of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and deep learning, and in particular to a model training method and device and a storage medium. BACKGROUND

[0002] Tongue diagnosis is a simple and effective method of observing the color and shape of the tongue to assist in diagnosis and differentiation. Therefore, in order to promote the development of tongue diagnosis, the related art combines traditional Chinese medicine diagnosis and treatment with image analysis technology, and realizes the objectification, standardization and quantification of tongue diagnosis through quantitative analysis of tongue appearance.

[0003] However, in the process of identifying and analyzing tongue appearance, the common deep learning sublingual plexus classification model has the problem of low classification accuracy in the model training process. SUMMARY

[0004] To solve the problems in the related art, the embodiments of the present application provide a model training method, device and storage medium.

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a model training method, comprising:

[0007] establishing a first model using a neural network;

[0008] obtaining first sample picture data;

[0009] performing resampling processing and increasing weight processing on specific data in the first sample picture data to obtain second sample picture data and third sample picture data, respectively;

[0010] training the first model using the first sample picture data, the second sample picture data and the third sample picture data.

[0011] In the above method, the resampling processing on the specific data in the first sample picture data to obtain the second sample picture data comprises:

[0012] repeatedly sampling the sample picture data in the first sample picture data that satisfies a first condition to obtain fourth sample picture data; the first condition represents that the data amount of the sample picture data is lower than a first threshold;

[0013] and / or,

[0014] undersampling the sample picture data in the first sample picture data that does not satisfy the first condition to obtain fifth sample picture data;

[0015] The fourth sample picture data and / or the fifth sample picture data are used as the second sample picture data.

[0016] In the method, in each training, for each type of sample picture data in the at least one type of sample picture data, sample picture data satisfying a preset condition in similarity is selected from the corresponding type of sample picture data; the data quantity of each type of sample picture data in the at least one type of sample picture data is lower than the first threshold; and the preset condition is determined according to a result of last training.

[0017] The sample picture data satisfying the preset condition in similarity in each type of the at least one type is used as sample picture data satisfying the first condition.

[0018] In the method, the increasing weight processing of the specific data in the first sample picture data to obtain third sample picture data includes:

[0019] The sample picture data satisfying the first condition in the first sample picture data is subjected to increasing weight processing to obtain sixth sample picture data; the first condition represents that the data quantity of the sample picture data is lower than a first threshold.

[0020] And / or,

[0021] The sample picture data not satisfying the first condition in the first sample picture data is subjected to decreasing weight processing to obtain seventh sample picture data.

[0022] The sixth sample picture data and / or the seventh sample picture data are used as the third sample picture data.

[0023] In the method, when the sample picture data satisfying the first condition in the first sample picture data is subjected to increasing weight processing, the method includes:

[0024] In each training, according to a value of a misclassification cost, a weight increasing value of the sample picture data satisfying the first condition in the first sample picture data is determined; the misclassification cost represents a factor for calculating the weight increasing value; and the value of the misclassification cost is determined according to a result of last training.

[0025] The sample picture data satisfying the first condition in the first sample picture data is subjected to increasing weight processing by using the weight increasing value.

[0026] In the method, the first sample picture data, the second sample picture data and the third sample picture data are respectively input to three branch networks of the first model, a first submodule of one branch network is connected with a second submodule of another two branch networks respectively, and the output of the first submodule is connected with the second submodule.

[0027] In the method, the loss value corresponding to the training result of the first model is determined by using the cross-entropy loss function of the first model.

[0028] The loss value is used to determine whether the first model is trained again, and the loss value at least includes a loss value corresponding to an error classification in the training result of the first model.

[0029] In the method, the cross-entropy loss function of the first model is represented as:

[0030]

[0031]

[0032]

[0033]

[0034] N represents the number of pictures corresponding to the first sample picture data, K represents the number of categories of the pictures corresponding to the first sample picture data, j represents the number of categories of the predicted classification result in the training result of the first model, g represents the number of categories of the actual classification result in the training result of the first model, y (i)[j] represents the matching condition of the i-th picture and the predicted label j; represents the actual probability of the i-th picture with the real category j and the predicted classification result j; represents a normalization factor, and a', b', and g' represent the normalization parameters of the first model after training;

[0035] wherein, is used to calculate the loss value corresponding to the difference between the predicted classification result and the actual classification result in the training result of the first model; is used to calculate the loss value corresponding to the error classification result in the training result of the first model.

[0036] The embodiments of the present application also provide a model training device, which comprises:

[0037] A first processing unit is configured to establish a first model by using a neural network.

[0038] A first processing unit is configured to establish a first model by using a neural network.

[0039] A second processing unit is configured to perform resampling processing and weight increasing processing on the specific data in the first sample picture data to obtain second sample picture data and third sample picture data, respectively.

[0040] a training unit configured to train the first model using the first sample picture data, the second sample picture data, and the third sample picture data.

[0041] The embodiments of the present application further provide a model training apparatus, comprising a processor and a memory for storing a computer program capable of running on the processor,

[0042] The processor is configured to run the computer program, and the computer program is executed by the processor to implement the steps of any of the model training methods.

[0043] The embodiments of the present application further provide a storage medium having a computer program stored thereon, and the computer program is executed by the processor to implement the steps of any of the model training methods.

[0044] The model training method, apparatus and storage medium provided by the embodiments of the present application are used to establish a first model by using a neural network, obtain first sample picture data, and perform resampling processing and weight increasing processing on specific data in the first sample picture data to obtain second sample picture data and third sample picture data respectively, and train the first model by using the first sample picture data, the second sample picture data and the third sample picture data. The technical solution provided by the embodiments of the present application is used to perform resampling processing and weight increasing processing on specific data in the first sample picture data after establishing the first model by using the neural network, and train the first model by using the first sample picture data and the processed sample picture data. In this way, the contribution rate of the picture features of the specific data to training the first model can be increased without affecting the picture features of the first sample picture data, and the problem that the classification and recognition rate of the first model for specific data is low due to the existence of unbalanced samples in the first sample picture data is avoided, so that the classification performance is damaged, and the classification and recognition rate is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A first model training method flowchart of the embodiments of the present application;

[0046] Figure 2 An architecture diagram of the first model in the application embodiments of the present application;

[0047] Figure 3 An architecture diagram of a sub-module of the first model in the application embodiments of the present application;

[0048] Figure 4 A first model training apparatus structure diagram of the embodiments of the present application;

[0049] Figure 5 A model training apparatus structure diagram of the embodiments of the present application. DETAILED DESCRIPTION

[0050] The application will be described in further detail below with reference to the drawings and embodiments.

[0051] Tongue diagnosis is a simple and effective method of observing the color and shape of the tongue to assist in diagnosis and differentiation. The tongue is the sprout of the heart and the external manifestation of the spleen. The coating is generated by stomach qi. Tongue diagnosis mainly observes the shape, color, and moisture of the tongue and tongue coating to determine the nature of the disease, the depth of the disease, the abundance of qi and blood, the balance of body fluid, and the deficiency and excess of organs. However, for a long time, the results of tongue diagnosis have completely relied on the subjective judgment of doctors, and the correctness of the diagnosis information is only related to the experience accumulation of doctors and environmental factors, resulting in a lack of objective diagnosis methods and standards for tongue diagnosis. In addition, since most tongue diagnosis experience is not easy to transmit and preserve, it also hinders the development of tongue diagnosis to some extent. Therefore, based on traditional Chinese medicine theory, combining traditional Chinese medicine diagnosis and treatment with image analysis technology to quantitatively analyze tongue images and realize the objectification, standardization and quantification of tongue diagnosis has become the only way for the development of traditional Chinese medicine tongue diagnosis.

[0052] In the related art, a deep learning sublingual plexus classification model generally only uses one convolutional neural network (CNN) network classifier, without considering the case of uneven distribution of image samples in the actual image training process, i.e., there are small class samples and large class samples. Specifically, in the data collection process of six classification of sublingual plexus pictures, there are more pictures of normal plexus, a little plexus, and tortuous plexus, while there are very few pictures of thickened plexus, purple plexus, and filled plexus, and the maximum number of classes / the minimum number of classes is even less than 5%. Among them, the data samples corresponding to the tongue coating categories of normal plexus, a little plexus, and tortuous plexus with more pictures can be referred to as large class samples, and the data samples corresponding to the tongue coating categories of thickened plexus, purple plexus, and filled plexus with fewer pictures can be referred to as small class samples. That is, the class sample with more sample quantity is referred to as a large class sample, and the class sample with less sample quantity is referred to as a small class sample. Since the sample picture data corresponding to the small class sample is small, for example, the sample picture data of thickened plexus and purple plexus in the sublingual plexus accounts for only 1% of the total sample picture data, and accordingly, the contribution in the classification model training process is also small, which reduces the recognition rate of the sublingual plexus classification model for small class samples, that is, in the related art, the sample distribution imbalance is not considered in the model training process, resulting in low classification accuracy.

[0053] Based on this, in various embodiments of the present application, after the model is established by using the neural network, the specific data in the sample picture data is respectively processed by resampling and weight increasing, and then the model is trained by using the picture sample data, the resampled picture sample data and the picture sample data after weight increasing. In this way, the contribution of the specific data in the model training process can be strengthened, so as to enhance the classification accuracy of the model for the specific data, and thus improve the overall performance of the model.

[0054] The embodiments of the present application provide a model training method, as shown in the following Figure 1 The method comprises the following steps:

[0055] Step 101: establishing a first model by using a neural network;

[0056] Step 102: obtaining first sample picture data;

[0057] Step 103: performing resampling and weight increasing on specific data in the first sample picture data to obtain second sample picture data and third sample picture data respectively;

[0058] Step 104: training the first model by using the first sample picture data, the second sample picture data and the third sample picture data.

[0059] In step 101, in actual application, the CNN can be used to establish the first model, the deep convolutional neural network (DCNN) can also be used to establish the first model, and other types of neural networks can also be used to establish the first model. The type of the neural network is not limited in the embodiments of the present application.

[0060] In step 102, in actual application, after the first model is established, the first sample picture data can be obtained from a sample picture database.

[0061] In actual application, the first model established by using the neural network comprises at least one convolution layer (Conv), at least one global average pooling layer (AvgPooling), at least one maximum pooling layer (MaxPooling), at least one fully connected layer (Fc), batch normalization (BN), rectified linear unit (Relu), logistic regression layer (Softmax) and loss function (Loss).

[0062] The convolutional layer is used to obtain local feature data in sample picture data; the global average pooling layer is used to regularize the structure of the first model to prevent overfitting; the max pooling layer is used to reduce the dimension of the local feature data by using the maximum value of the local feature data; the fully connected layer is used to combine multiple local feature data into global feature data; the batch normalization is used to normalize the local feature data; the rectified linear unit is used to retain the local feature data; the logistic regression layer is used to output a classification result corresponding to the sample picture data; and the loss function is used to calculate a loss value of a training result of the first model.

[0063] Specifically, the first model can include three branch networks, and each branch network can include a residual network unit (resnet block) and a fully connected layer. In actual application, multiple resnet blocks and corresponding fully connected layers can be set in one branch network according to needs. For example, as shown in FIG. 1, the first model includes three branch networks, and each branch network further includes two resnet blocks and one fully connected layer. Figure 2 A residual network unit is composed of two convolutional layers, one batch normalization unit, one rectified linear unit, and one pooling layer. Then, the three branch networks are connected to the logistic regression layer through a weighting function, and the logistic regression layer is connected to the loss function, thereby forming a network architecture of the first model.

[0064] In actual application, after the first model is established, first sample picture data needs to be obtained to input into the first model to achieve the purpose of training the first model; the first sample picture data refers to sample picture data classified in advance. For example, the first sample picture data can be sample picture data associated with tongue fur, and can include sample picture data categories such as normal choroid, a little choroid, tortuous choroid, thickened choroid, purple and dim choroid, and filled choroid. In this way, after the first model is trained by using the first sample picture data, the first model can be used to identify the category of sample picture data associated with tongue fur.

[0065] In actual application, in order to resample specific data in the first sample picture data to obtain second sample picture data, oversampling processing can be performed on sample picture data with small data volume in the first sample picture data, or undersampling processing can be performed on sample picture data with large data volume in the first sample picture data.

[0066] Based on this, in an embodiment, the resampling processing on the specific data in the first sample picture data to obtain second sample picture data includes:

[0067] The sample picture data in the first sample picture data that meets the first condition is resampled to obtain fourth sample picture data; the first condition represents that the data amount of the sample picture data is lower than a first threshold value;

[0068] and / or,

[0069] The sample picture data in the first sample picture data that does not meet the first condition is undersampled to obtain fifth sample picture data.

[0070] The fourth sample picture data and / or the fifth sample picture data is used as the second sample picture data.

[0071] The data amount can be understood as the actual number of the sample picture data in the first sample picture data, or as the percentage of the number of the sample picture data in the first sample picture data.

[0072] In actual application, the first condition is used to judge the specific data in the first sample picture data, and the specific data is resampled. The first condition represents that the data amount of the sample picture data is lower than a first threshold value. Here, the value of the first threshold value can be set as a fixed value, or can be dynamically set by the user as needed, and the value of the first threshold value is not limited in the embodiment of the application.

[0073] In actual application, the sample picture data in the first sample picture data that meets the first condition, i.e., the sample picture data with a data amount lower than the first threshold value (also referred to as small category sample), is resampled to generate fourth sample picture data. The data amount of the fourth sample picture data can be greater than or equal to the first threshold value.

[0074] The sample picture data in the first sample picture data that does not meet the first condition, i.e., the sample picture data with a data amount greater than the first threshold value (also referred to as large category sample), is undersampled to obtain fifth sample picture data. The data amount of the fifth sample picture data can be less than or equal to the first threshold value, or the data amount of the fifth sample picture data can be less than or equal to a second threshold value. Exemplarily, the fifth sample picture data is obtained by randomly selecting sample picture data in the large category sample.

[0075] Here, in actual application, when the fourth sample picture data is obtained, the fourth sample picture data can be used as the second sample picture data.

[0076] When the fifth sample picture data is obtained, the fifth sample picture data is used as the second sample picture data.

[0077] When the fourth sample picture data and the fifth sample picture data are obtained, the fourth sample picture data and the fifth sample picture data can be taken together as the second sample picture data.

[0078] In actual application, in the process of repeatedly sampling the small-class samples, a plurality of samples in the small-class samples need to be selected to construct a new small-class sample.

[0079] Based on this, in an embodiment, the following steps are performed:

[0080] In each training, for each class of sample picture data in the at least one class of sample picture data, sample picture data with a similarity satisfying a preset condition is selected from the corresponding class of sample picture data; the data quantity of each class of sample picture data in the at least one class is lower than the first threshold; and the preset condition is determined according to the result of the last training.

[0081] The sample picture data with the similarity satisfying the preset condition in each class of the at least one class is taken as sample picture data satisfying the first condition.

[0082] Here, the similarity of the corresponding class of sample picture data can be determined based on Euclidean distance, and then the sample picture data with the similarity satisfying the preset condition is taken as a sample satisfying the first condition, and the sample satisfying the first condition is subjected to repeated sampling processing.

[0083] In actual application, when the preset condition is determined based on the result of the last training, the preset condition can be determined according to experience.

[0084] Here, in actual application, in the process of repeated sampling, the sampling dimension of the sample needs to be determined, and if the sample has a plurality of sampling dimensions, repeated sampling processing needs to be performed on each dimension. For example, if the picture color corresponding to the sample is grayscale, there is only one sampling dimension in the sample, and repeated sampling needs to be performed on the sample picture data of this dimension. If the picture color corresponding to the sample picture data to be repeatedly sampled is color, the sample has three sampling dimensions, for example, the sample is composed of a three-dimensional array, and is represented as 3*H*W, and the three sampling dimensions can be 1*H*W, 2*H*W, and 3*H*W. If repeated sampling processing needs to be performed on the sample picture data in three dimensions, one dimension is processed at a time.

[0085] In actual application, in order to increase the weight of the specific data in the first sample picture data to obtain third sample data, the weight of the sample picture data with a small data quantity in the first sample picture data can be increased, or the weight of the sample picture data with a large data quantity in the first sample picture data can be reduced.

[0086] Based on this, in an embodiment, the increasing weight processing on the specific data in the first sample picture data to obtain third sample picture data comprises:

[0087] The sample picture data in the first sample picture data satisfying a first condition is subjected to increasing weight processing to obtain sixth sample picture data; the first condition represents that the data amount of the sample picture data is lower than a first threshold value;

[0088] and / or,

[0089] The sample picture data in the first sample picture data not satisfying the first condition is subjected to decreasing weight processing to obtain seventh sample picture data;

[0090] The sixth sample picture data and / or the seventh sample picture data is / are taken as the third sample picture data.

[0091] In actual application, when the sixth sample picture data is obtained, the sixth sample picture data can be taken as the third sample picture data; when the seventh sample picture data is obtained, the seventh sample picture data is taken as the third sample picture data; when the sixth sample picture data and the seventh sample picture data are obtained, the sixth sample picture data and the seventh sample picture data are taken together as the third sample picture data.

[0092] Here, in the process of training the first model, the weight of the sample picture data can be understood as the contribution rate of the picture feature corresponding to the sample picture data to the network model. The greater the weight of the sample picture data, the more attention will be paid to the data feature corresponding to the sample picture data when training the network model. Correspondingly, the smaller the weight of the sample picture data, the less attention will be paid to the data feature corresponding to the sample picture data when training the network model; that is, some data features in the sample picture data can be ignored.

[0093] In actual application, the weight of the sample picture data (i.e. small-class sample) in the first sample picture data whose data amount is lower than the set threshold value can be increased to increase the contribution rate of the small-class sample to the first model, or the weight of the sample picture data (i.e. large-class sample) in the first sample picture data whose data amount is greater than or equal to the set threshold value can be reduced to reduce the contribution rate of the large-class sample to the first model, thereby avoiding the problem that the imbalance of the sample picture data affects the performance of the first model. Exemplarily, for the sample picture data in the first sample picture data whose data amount is greater than or equal to the first threshold value, the decreasing weight processing can be realized by setting a weight reduction value; wherein the weight reduction value can be set as a fixed value, or can be dynamically set by the first model according to the training result.

[0094] Here, in actual application, each time the first model is trained by using the first sample picture data, the second sample picture data and the third sample picture data, it can be considered that a classifier is output after each training; the classifier can be used to classify the input sample picture data and output a classification result.

[0095] Since the sample picture data input in the process of training the first model is pre-classified data, according to the classification result output by the first model and the actual category of the sample picture data, it can be determined whether the classification result of the current first model is correct.

[0096] In this case, for the sample picture data with small data volume in the first sample picture data, when the classification of the classifier output by the first model is wrong, the weight increasing processing can be performed by setting a wrong classification cost.

[0097] Based on this, in an embodiment, when the sample picture data satisfying the first condition in the first sample picture data is subjected to the weight increasing processing, the method comprises:

[0098] In each training, according to the value of the wrong classification cost, the weight increasing value of the sample picture data satisfying the first condition in the first sample picture data is determined; the wrong classification cost represents a factor for calculating the weight increasing value; the value of the wrong classification cost is determined according to the last training result;

[0099] The sample picture data satisfying the first condition in the first sample picture data is subjected to the weight increasing processing by using the weight increasing value.

[0100] In actual application, each time the first model is trained, it can be considered that a classifier is output, and when it is determined that there is an error classification according to the classification result of the classifier, a wrong classification cost can be added to the sample picture data corresponding to the error classification. In this way, according to the value of the wrong classification cost, the weight increasing value of the sample data corresponding to the error classification can be determined, so as to realize the weight increasing processing.

[0101] Here, when the value of the wrong classification cost is determined based on the last training result, the value of the wrong classification cost can be set according to experience.

[0102] In actual application, in each training, the corresponding weight increase value can be determined by setting the correlation between the misclassification cost and the weight increase value. Illustratively, when the classifier misclassifies the sample picture data with small data volume in the first sample picture data, the corresponding weight increase value is determined based on the value of the misclassification cost determined based on the last training result, so that the weight of the sample picture data with small data volume in the first sample picture data is updated according to the weight increase value. Illustratively, a table of the corresponding relationship between the value of the misclassification cost and the weight increase value can be set, and the weight increase value corresponding to the value of the misclassification cost is found according to the table; the weight increase value of the sample picture data can also be determined by setting a function, and when the classifier misclassifies the sample picture data with small data volume in the first sample picture data, the corresponding weight increase value is obtained by inputting the misclassification cost into the function.

[0103] Then, the weight of the sample picture data with small data volume in the first sample picture data is updated by using the weight increase value.

[0104] In the process of training the first model by using the first sample picture data, the second sample picture data and the third sample picture data, in order to reduce the coupling, the first sample picture data, the second sample picture data and the third sample picture data can be input into three branch networks of the first model to realize decoupling.

[0105] Meanwhile, since the second sample picture data and the third sample picture data are obtained by resampling processing and weight increasing processing, the resampling processing changes the data volume of the sample picture data, and the weight increasing processing changes the picture feature distribution of the sample picture data, so that the features of the sample picture data extracted by the first model through training may be affected to a certain extent. Therefore, the branch networks need to be connected across branches, so as to maximize the retention of the picture data feature distribution of the original sample picture data.

[0106] Based on this, in an embodiment, the first sample picture data, the second sample picture data and the third sample picture data are input into three branch networks of the first model, the first sub-module of one branch network is connected with the second sub-module of the other two branch networks respectively, and the output of the first sub-module is connected with the second sub-module.

[0107] In actual application, the first sub-module can also be referred to as a first residual network unit, and the second sub-module can also be referred to as a second residual network unit.

[0108] Here, in order to connect the branch networks across branches, the first sub-module and the second sub-module can be densely connected and fused. Specifically, the first sub-module of one branch network is connected with the second sub-module of the other two branch networks respectively, and the output of the first sub-module is connected with the second sub-module.

[0109] In practical applications, after the sample image data is input into the first submodule of a branch network, feature data is extracted through the first submodule. The extracted feature data is then input into the second submodule. This method of extracting feature data through the first and second submodules improves the accuracy of feature data extraction, allowing for an accurate representation of the original image feature distribution of the sample image data based on a limited amount of feature data.

[0110] Here, as Figure 3 As shown, after the sample image data is input into the first submodule, if the data dimensions of the sample image data differ, it will first pass through the first convolutional layer; the convolutional kernel corresponding to the first convolutional layer can be a 3*3 kernel. After extracting the feature data of the sample image data using the convolutional kernel corresponding to the first convolutional layer, the extracted feature data is input into the batch normalization unit to normalize the extracted feature data. Then, the normalized feature data is input into the second convolutional layer, and the convolutional kernel corresponding to the second convolutional layer is used to extract feature data again to reduce the dimension and number of feature data. Here, the size of the convolutional kernel corresponding to the second convolutional layer can be the same as or different from the size of the convolutional kernel corresponding to the first convolutional layer. Next, the feature data output by the second convolutional layer is input into the corrected linear unit. Through the processing of the corrected linear unit, the sparsity and expressive power of the first model can be increased while retaining the feature data, that is, the features of the sample image data can be accurately expressed using a small amount of feature data. Then, a pooling layer is used to extract the feature data after the modified linear unit (MRU) processing, thereby reducing the dimensionality and volume of the feature data. For example, a max pooling layer is used to extract the maximum value of all feature data in the pooling window, which is then used as the pooled feature data.

[0111] If the sample image data has the same data dimension, after extracting feature data through the first convolutional layer, the extracted feature data can be directly input into the pooling layer to reduce the data dimension and the amount of feature data, thereby improving the efficiency of training the first model.

[0112] Here, for the second submodule of a branch network, after receiving the feature data output from the first submodule of its own branch network and the first submodules of the other two branch networks, it needs to combine the feature data output from the three first submodules by concatenating them along the channel dimension. Then, the second submodule performs feature extraction again on the combined feature data to reduce the amount of feature data.

[0113] In actual application, the first sample picture data can be input to the first branch network, the second sample picture data can be input to the second branch network, and the third sample picture data can be input to the third branch network. Then, for the outputs of the second sub-modules in the three branch networks, the feature data output by the three branch networks can be combined through a weighting function to obtain combined feature data. Exemplarily, the weighting function can be represented as:

[0114]

[0115] wherein W ce , W rs , and W rw respectively represent the weights corresponding to the first sample picture data, the second sample picture data, and the third sample picture data; and α, β, and γ respectively represent the normalization parameters of the first sample picture data, the second sample picture data, and the third sample picture data, with α+β+γ=1.

[0116] In actual application, W ce , W rs , and W rw can be understood as weight matrices corresponding to the first sample picture data, the second sample picture data, and the third sample picture data. When the weighting function is used, W ce , W rs , and W rw can be directly used to combine the feature data output by the three branch networks to obtain the combined feature data; or W ce , W rs , and W rw can be transposed first, and then the transposed W ce , W rs , and W rw are used to combine the feature data output by the three branch networks to obtain the combined feature data.

[0117] Here, when the feature data of the three branch networks needs to be combined, since the feature data extracted by each branch network can be different, the feature data with less data amount can be easily ignored. Therefore, the feature data of the three branch networks needs to be normalized to ensure that the first model can be trained based on all the feature data, so as to facilitate training of the first model with optimal performance.

[0118] In actual application, after the feature data of the three branch networks is combined, the combined feature data can be output through a Softmax layer to obtain a training result corresponding to the combined feature data. Specifically, the output of the Softmax layer can be:

[0119]

[0120] wherein k represents the number of categories corresponding to the first sample picture data; j represents the number of prediction categories corresponding to the first sample picture; Z i Z represents the weighted result of the i-th picture obtained through the weighting function.

[0121] The model training method provided by the embodiments of the present application can extract feature data by decoupling the three branch networks and connecting the branch networks through dense connection. In this way, the original feature distribution of the sample picture data can be retained, so that the first model achieves better training effect.

[0122] In actual application, after the training result is output through the Softmax layer, a loss function is needed to determine the degree of inconsistency between the prediction classification result and the actual classification result in the training result of the first model, so as to determine whether the parameters in the first model need to be adjusted and retrained.

[0123] In actual application, due to the uneven distribution of sample picture data in the first sample picture data, the generalization ability of the first model is limited. Although the resampling processing and the weight increasing processing can inhibit the uneven distribution of sample picture data to a certain extent, there may still be dirty data or noise interference in the sample picture data with small data volume. In addition, in the related art, when the training result output by the Softmax layer is input to the loss function, all the results of the wrong classification are set to 0, that is, the influence of the wrong classification in the training result is not considered. This will gradually accumulate errors, thereby reducing the classification performance of the first model. Therefore, the influence of the wrong classification can be determined by improving the loss function, and at the same time, the influence of the interference in the sample picture data with small data volume can be reduced.

[0124] In an embodiment, the cross-entropy loss function of the first model is used to determine the loss value corresponding to the training result of the first model.

[0125] The loss value is used to determine whether the first model needs to be retrained; and the loss value at least includes the loss value corresponding to the wrong classification in the training result of the first model.

[0126] The cross-entropy loss function in the first model is represented as

[0127]

[0128]

[0129]

[0130] wherein, N represents the number of pictures corresponding to the first sample picture data; K represents the number of categories of the pictures corresponding to the first sample picture data; j represents the number of categories of the predicted classification results in the training result of the first model; g represents the number of categories of the actual classification results; y (i)[j] represents the matching condition of the real label and the predicted label j of the ith picture; represents the actual probability of the ith picture with the real category j and the predicted classification result j; represents the normalization factor; and a', β', γ' represent the normalization parameters of the first model after training;

[0131] wherein, is used for calculating the loss value corresponding to the difference between the predicted classification result and the actual classification result in the training result of the first model; is used for calculating the loss value corresponding to the misclassified result in the training result of the first model.

[0132] In actual application, in the process of calculating the loss value corresponding to the training result by using the cross-entropy loss function, the predicted classification result and the corresponding actual classification result in the training result of the first model need to be determined respectively. Then, the loss value corresponding to the training result of the first model can be calculated according to the predicted classification result and the actual classification result. The loss value includes the loss value corresponding to the predicted classification result and the loss value corresponding to the misclassified result.

[0133] In actual application, after obtaining the loss value corresponding to the training result of the first model by using the cross-entropy loss function of the first model, the loss value can be compared with the pre-set loss value threshold. If the current loss value is lower than the loss value threshold, it means that the consistency between the predicted classification result and the actual classification result in the training result of the first model is high, that is, the classification accuracy of the first model is high. Then, the training of the first model can be ended, and the first model can be used to classify the picture sample data to be classified.

[0134] If the current loss value is greater than the loss value threshold, it means that the difference between the predicted classification result and the actual classification result in the training result of the first model is large. In this case, the related parameters in the first model need to be adjusted, such as the weights corresponding to the three branch networks, and the first model is trained again by using the first picture sample data, the second picture sample data and the third picture sample data, until the loss value corresponding to the training result is less than the loss value threshold, the training of the first model is ended.

[0135] In actual application, when the trained first model is used for classification, for the multi-class labels of the same sublingual plexus picture, each class can be judged by the first model to output the multi-class coincident label of the picture. For example, for a sublingual plexus picture, the trained first model can be used to judge the picture for the plexus thickening class, so as to determine whether the picture has the label of plexus thickening. Then, the trained first model can be used to judge the picture for the plexus purple dim class, to determine whether the picture has the label of plexus purple dim. In this way, the trained first model can identify the multi-class coincident label of a sublingual plexus picture.

[0136] The model training method provided in the embodiments of the present application, after the first model is established by the neural network, obtains first sample picture data, and performs resampling processing and reweighting processing on specific data in the first sample picture data, to obtain second sample picture data and third sample picture data; then, the first model is trained by using the first sample picture data, the second sample picture data and the third sample picture data. The technical solution provided in the embodiments of the present application, after the first model is established by the neural network, performs resampling processing and reweighting processing on specific data in the first sample picture data, and trains the first model by using the first sample picture data and the processed sample picture data. In this way, the contribution rate of the specific data in the sample picture data can be adjusted without affecting the picture feature distribution of the sample picture data, the problem of low recognition rate of the first model for specific data caused by unbalanced sample picture data is avoided, and the overall performance of the first model is improved, and the classification recognition rate is improved.

[0137] In addition, the first sample picture data, the second sample picture data and the third sample picture data are respectively input into the three branch networks of the first model to realize decoupling of the branches. At the same time, the first sub-modules of the branch networks are connected with the second sub-modules of the other two branch networks in a dense connection manner, that is, cross-branch connection is realized, so that the original features of the sample picture data can be retained, the optimal convergence effect of the first model is achieved, and the features of the sample picture data learned by the model in the training process are avoided from being damaged.

[0138] In addition, the loss value corresponding to the training result of the first model is calculated by improving the cross-entropy loss function, so that the first model can adjust the weight values of each branch network according to the loss value corresponding to the error classification in the training process. In this way, when the small class samples are misclassified, the accuracy of the training result of the model is judged by using the loss function, so that the small class samples can obtain better training opportunities without increasing the sample picture data and the complexity of the first model, thereby improving the robustness of the model, that is, providing the ability that the classification performance of the model is not affected in the case of unbalanced sample classification. At the same time, in the process of calculating the loss value by using the improved cross-entropy loss function, the loss values corresponding to the actual classification result and the error classification result in the sample picture data are calculated respectively. For the sample picture data associated with the tongue fur, there are multiple categories in the actual classification result, in this case, the first model trained by using the actual classification result corresponding to each category can identify the multi-label classification of the tongue fur context picture, so as to accurately judge the category of the tongue fur context picture.

[0139] To achieve the scheme of the embodiments of the present application, the embodiments of the present application further provide a model training device, as shown in the following Figure 4 The device comprises:

[0140] The first processing unit 401 is configured to establish a first model by using a neural network.

[0141] The acquisition unit 402 is configured to acquire first sample picture data.

[0142] The second processing unit 403 is configured to perform resampling processing and weight increasing processing on specific data in the first sample picture data to obtain second sample picture data and third sample picture data respectively.

[0143] The training unit 404 is configured to train the first model by using the first sample picture data, the second sample picture data and the third sample picture data.

[0144] In an embodiment, the second processing unit 403 is configured to:

[0145] The first sample picture data satisfying a first condition is resampled to obtain fourth sample picture data, wherein the first condition represents that the data amount of the sample picture data is lower than a first threshold; and / or the first sample picture data not satisfying the first condition is undersampled to obtain fifth sample picture data; and the fourth sample picture data and / or the fifth sample picture data are used as the second sample picture data.

[0146] In an embodiment, the second processing unit 403 is further configured to:

[0147] In each training, for each type of sample picture data in the at least one type of sample picture data of the first sample picture data, sample picture data satisfying a preset condition in similarity is selected from the corresponding type of sample picture data; the data amount of each type of sample picture data in the at least one type of sample picture data is lower than the first threshold; and the preset condition is determined according to the result of the last training.

[0148] The sample picture data satisfying the preset condition in similarity in each type of the at least one type is taken as sample picture data satisfying the first condition.

[0149] In an embodiment, the second processing unit 403 is configured to:

[0150] The sample picture data satisfying the first condition in the first sample picture data is subjected to weight increasing processing to obtain sixth sample picture data; the first condition represents that the data amount of the sample picture data is lower than the first threshold; and / or the sample picture data not satisfying the first condition in the first sample picture data is subjected to weight decreasing processing to obtain seventh sample picture data; and the sixth sample picture data and / or the seventh sample picture data is taken as the third sample picture data.

[0151] In an embodiment, the second processing unit 403 is configured to:

[0152] In each training, a weight increasing value of the sample picture data satisfying the first condition in the first sample picture data is determined according to the value of the misclassification cost; the misclassification cost represents a factor for calculating the weight increasing value; and the value of the misclassification cost is determined according to the result of the last training; and the sample picture data satisfying the first condition in the first sample picture data is subjected to weight increasing processing by using the weight increasing value.

[0153] In an embodiment, the first sample picture data, the second sample picture data and the third sample picture data are respectively input to three branch networks of the first model, a first sub-module of one branch network is connected with a second sub-module of another two branch networks respectively; and the output of the first sub-module is connected with the second sub-module.

[0154] In an embodiment, the training unit 404 is configured to:

[0155] A loss value corresponding to the training result of the first model is determined by using a cross-entropy loss function of the first model.

[0156] The loss value is used for judging whether the first model is trained again or not; and the loss value at least contains a loss value corresponding to an error classification in the training result of the first model.

[0157] In actual application, the acquisition unit 402 can be implemented by a processor in the model training apparatus in combination with a communication interface; the first processing unit 401, the second processing unit 403 and the training unit 404 can be implemented by the processor in the model training apparatus.

[0158] It should be noted that the model training apparatus provided in the above embodiments is only used for example to illustrate the division of the above program units, and in actual application, the above processing can be completed by different program units according to needs, that is, the internal structure of the apparatus is divided into different program units to complete all or part of the above-described processing. In addition, the model training apparatus and the model training method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0159] Based on the hardware implementation of the above program modules, and in order to implement the method of model training of the embodiments of the present application, the embodiments of the present application also provide a model training apparatus, as shown in Figure 5 The model training apparatus 500 includes:

[0160] a communication interface 501 capable of interacting with other devices;

[0161] a processor 502 connected with the first communication interface 501 to realize interaction with other devices, for running a computer program, and executing the method provided by one or more technical solutions of the above model training apparatus;

[0162] a memory 503, wherein the computer program is stored in the memory 503.

[0163] Specifically, the first processor 502 is configured to:

[0164] acquire first sample picture data;

[0165] establish a first model by using a neural network; perform resampling processing and weight increasing processing on specific data in the first sample picture data to obtain second sample picture data and third sample picture data respectively; and train the first model by using the first sample picture data, the second sample picture data and the third sample picture data.

[0166] In an embodiment, the first processor 502 is configured to perform resampling processing on sample picture data satisfying a first condition in the first sample picture data to obtain fourth sample picture data; the first condition represents that the data amount of the sample picture data is lower than a first threshold.

[0167] and / or,

[0168] undersample sample picture data in the first sample picture data that does not satisfy the first condition to obtain fourth sample picture data;

[0169] use the fourth sample picture data and / or the fifth sample picture data as the second sample picture data.

[0170] In an embodiment, the first processor 502 is configured to:

[0171] In each training, for each type of sample picture data in the at least one type of sample picture data, select sample picture data that satisfies a preset condition in similarity from the respective type of sample picture data; the data amount of each type of sample picture data in the at least one type of sample picture data is lower than the first threshold value; and the preset condition is determined according to a result of last training.

[0172] use the sample picture data that satisfies the preset condition in similarity in each type of the at least one type as sample picture data that satisfies the first condition.

[0173] In an embodiment, the first processor 502 is configured to:

[0174] perform weight increasing processing on sample picture data that satisfies a first condition in the first sample picture data to obtain sixth sample picture data; the first condition represents that the data amount of the sample picture data is lower than a first threshold value;

[0175] and / or,

[0176] perform weight decreasing processing on sample picture data that does not satisfy the first condition in the first sample picture data to obtain seventh sample picture data;

[0177] use the sixth sample picture data and / or the seventh sample picture data as the third sample picture data.

[0178] In an embodiment, the first processor 502 is configured to:

[0179] In each training, determine a weight increasing value of sample picture data that satisfies the first condition in the first sample picture data according to a value of a misclassification cost; the misclassification cost represents a factor used to calculate the weight increasing value; and the value of the misclassification cost is determined according to a result of last training.

[0180] perform weight increasing processing on the sample picture data that satisfies the first condition in the first sample picture data by using the weight increasing value.

[0181] In an embodiment, the first sample picture data, the second sample picture data and the third sample picture data are respectively input to three branch networks of the first model, a first submodule of one branch network is connected with a second submodule of another two branch networks respectively; and an output of the first submodule is connected with the second submodule.

[0182] In an embodiment, the first processor 502 is further configured to:

[0183] determine a loss value corresponding to the training result of the first model by using a cross-entropy loss function of the first model; wherein the loss value is used to determine whether the first model is trained again; and the loss value at least includes a loss value corresponding to an error classification in the training result of the first model.

[0184] It should be noted that the specific processing process of the first processor 502 can be understood with reference to the above method.

[0185] Of course, in actual application, each component in the model training apparatus is coupled together through the bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a state signal bus. However, in order to clearly illustrate, all kinds of buses are marked as the bus system 504 in the Figure 5 .

[0186] The first memory 503 in the embodiment of the present application is used to store various types of data to support the operation of the model training apparatus 500. Examples of these data include: any computer programs used to operate on the model training apparatus 500.

[0187] The method disclosed by the embodiments of the present application can be applied to the first processor 502 or implemented by the first processor 502. The first processor 502 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the first processor 502 or instructions in the form of software. The first processor 502 described above can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The first processor 502 can implement or execute each method, step and logic block disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the hardware decoding processor can be directly embodied to execute the steps of the method, or the combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a storage medium, which is located in the first memory 503, and the first processor 502 reads the information in the first memory 503 to combine the hardware to complete the steps of the above method.

[0188] In the exemplary embodiments, the model training apparatus 500 can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, micro controller units (MCUs), microprocessors, or other electronic elements, for executing the above method.

[0189] In the example embodiment, the embodiment of the present application also provides a storage medium, i.e., a computer storage medium, specifically a computer readable storage medium, for example, including a first memory 503 storing a computer program, which can be executed by a first processor 502 of the model training apparatus 500 to complete the steps of the aforementioned model training method. The computer readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory.

[0190] It should be noted that "first", "second", and the like are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence.

[0191] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0192] The above is only a preferred embodiment of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that, include: The first model is built using a neural network; Obtain the first sample image data; The first sample image data includes sample image data associated with the tongue coating veins; By resampling and weighting specific data from the first sample image data, second and third sample image data are obtained respectively; the specific data is determined based on a first condition, which indicates that the amount of data in the sample image data is lower than a first threshold. The first model is trained using the first sample image data, the second sample image data, and the third sample image data. The first model is used to identify multiple category labels of tongue coating vein images. The first sample image data, the second sample image data, and the third sample image data are respectively input into three branch networks of the first model. The first submodule of one branch network is connected to the second submodule of the other two branch networks. The output of the first submodule is connected to the second submodule.

2. The method according to claim 1, characterized in that, The step of resampling specific data from the first sample image data to obtain the second sample image data includes: The sample image data that meets the first condition in the first sample image data is repeatedly sampled to obtain the fourth sample image data. And / or, The sample image data that does not meet the first condition in the first sample image data is undersampled to obtain the fifth sample image data; The fourth sample image data and / or the fifth sample image data are used as the second sample image data.

3. The method according to claim 2, characterized in that, In each training session, for each class of sample image data in at least one class of the first sample image data, sample image data whose similarity meets the preset conditions are selected from the sample image data of the corresponding class. The amount of data for each sample image in at least one class is less than the first threshold; The preset conditions are determined based on the results of the previous training. Sample image data that satisfy the preset conditions for similarity in at least one class are used as sample image data that satisfy the first condition.

4. The method according to claim 1, characterized in that, The step of using specific data from the first sample image data to perform weighting processing to obtain the third sample image data includes: The sample image data that meets the first condition in the first sample image data is weighted to obtain the sixth sample image data. And / or, The sample image data that does not meet the first condition in the first sample image data is weighted down to obtain the seventh sample image data. The sixth sample image data and / or the seventh sample image data are used as the third sample image data.

5. The method according to claim 4, characterized in that, When adding weights to the sample image data that meet the first condition in the first sample image data, the method includes: In each training session, the weight increment of sample image data that meets the first condition in the first sample image data is determined based on the value of the misclassification cost; the misclassification cost represents the factor used to calculate the weight increment; the value of the misclassification cost is determined based on the previous training result; Using the weight increment value, the sample image data in the first sample image data that meets the first condition is weighted.

6. The method according to any one of claims 1 to 5, characterized in that, The loss value corresponding to the training result of the first model is determined by using the cross-entropy loss function of the first model. The loss value is used to determine whether the first model should be trained again; the loss value includes at least the loss value corresponding to misclassification in the training results of the first model.

7. The method according to claim 6, characterized in that, The cross-entropy loss function in the first model is characterized as follows: Where N represents the number of images corresponding to the first sample image data; K represents the number of categories of the images corresponding to the first sample image data; j represents the number of categories of the predicted classification results in the training results of the first model; g represents the number of categories of the actual classification results of the first sample image data; y (i)[j] This represents the matching status between the true label and the predicted label j of the i-th image; Given the i-th image whose true category is j, predict the actual probability that the classification result is j. The normalization factor is represented by α', β', and γ'; these represent the normalization parameters after training the first model. in, Used to calculate the loss value corresponding to the difference between the predicted classification result and the actual classification result in the training results of the first model; Used to calculate the loss value corresponding to the misclassification result in the training results of the first model.

8. A model training device, characterized in that, The device includes: The first processing unit is used to build a first model using a neural network; The acquisition unit is used to acquire first sample image data; the first sample image data includes sample image data associated with tongue coating veins; The second processing unit is used to perform resampling and weighting processing on specific data in the first sample image data to obtain second sample image data and third sample image data, respectively; the specific data is determined based on a first condition, which indicates that the amount of sample image data is lower than a first threshold. The training unit is used to train the first model using the first sample image data, the second sample image data, and the third sample image data. The first model is used to identify multiple category labels of tongue coating vein images. The first sample image data, the second sample image data, and the third sample image data are respectively input into three branch networks of the first model. The first submodule of one branch network is connected to the second submodule of the other two branch networks. The output of the first submodule is connected to the second submodule.

9. A model training device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the model training method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the model training method according to any one of claims 1 to 7.

Images