An information recommendation method, device, storage medium and electronic device

Abstract

This specification discloses an information recommendation method, apparatus, storage medium, and electronic device. After obtaining an image collected for a user's lesion site, the embodiments of this specification input the image into a classification model, determine various feature data through the classification model, and determine the probability that each feature data belongs to each symptom. Based on the probability that each feature data belongs to each symptom and each feature data, determine the symptom features corresponding to each symptom. Concatenate the symptom features corresponding to each symptom to obtain a comprehensive symptom feature, and identify the disease suffered by the user based on the comprehensive symptom feature. In this method, the various symptoms at the lesion site can be decoupled to obtain the symptom features of each symptom, and then the disease suffered by the patient can be determined based on the symptom features of each symptom, which can effectively improve the accuracy of classifying diseases with similar symptoms.

Description

Technical Field

[0001] This specification relates to the medical field, and particularly to an information recommendation method, apparatus, storage medium, and electronic device. Background Art

[0002] With the continuous progress of artificial intelligence technology, the combination of artificial intelligence and medicine has achieved long-term development and extensive attention.

[0003] In the prior art, a traditional neural network can be used to classify a patient's medical images to diagnose the disease suffered by the patient.

[0004] However, in the prior art, the traditional neural network can only classify diseases with obvious symptoms or independent symptoms. For example, bone abnormalities and breast cancer abnormalities are diseases with obvious symptoms or independent symptoms. However, in diseases such as keratitis, more detailed disease types can be classified, such as fungal keratitis and bacterial keratitis. The symptoms exhibited by these more detailed disease types are relatively similar, and multiple symptoms are coupled at the lesion site, resulting in the inability of the prior art to effectively classify more detailed disease types. Summary of the Invention

[0005] Embodiments of this specification provide an information recommendation method, apparatus, storage medium, and electronic device to partially solve the problems existing in the prior art.

[0006] Embodiments of this specification adopt the following technical solutions:

[0007] An information recommendation method provided in this specification includes:

[0008] Obtain an image collected for the lesion site of a user;

[0009] Input the image into a pre-trained classification model to output respective feature data through a first fully connected layer in the classification model;

[0010] Input the respective feature data into a symptom classification layer in the classification model to determine, for each symptom, the probability that each feature data belongs to the symptom through the symptom classification layer;

[0011] Input the respective feature data and the probability that each feature data belongs to each symptom into a fusion layer in the classification model to determine, for each symptom, a fusion feature corresponding to the symptom based on the probability that each feature data belongs to the symptom and the respective feature data through the fusion layer;

[0012] Input the fusion feature corresponding to each symptom into a second fully connected layer in the classification model to output a symptom feature corresponding to each symptom through the second fully connected layer;

[0013] Input the symptom features corresponding to each symptom into the splicing layer in the classification model, so as to splice the symptom features corresponding to each symptom through the splicing layer to obtain comprehensive symptom features;

[0014] Input the comprehensive symptom features into the disease classification layer in the classification model, so as to identify the disease suffered by the user based on the comprehensive symptom features through the disease classification layer;

[0015] Perform information recommendation based on the identified disease suffered by the user.

[0016] Optionally, input the image into a pre-trained classification model to output each feature data through the first fully connected layer in the classification model, specifically including:

[0017] Input the image into a pre-trained classification model to extract features from the image through the encoding layer in the classification model to obtain image features;

[0018] Input the image features into the pooling layer in the classification model to output the pooled features through the pooling layer;

[0019] Input the pooled features into the first fully connected layer in the classification model to output each feature data represented by the lesion site through the first fully connected layer.

[0020] Optionally, determine the fusion feature corresponding to this symptom based on the probability that each feature data belongs to this symptom and the feature data, specifically including:

[0021] For each feature data, multiply the probability that the feature data belongs to this symptom by the feature data to obtain the local feature corresponding to the feature data;

[0022] Accumulate the local features corresponding to each feature data to obtain the fusion feature corresponding to this symptom.

[0023] Optionally, obtain the image collected for the lesion site of the user, specifically including:

[0024] Obtain the global image collected for the lesion site of the user and obtain the local image collected for the lesion site of the user;

[0025] Input the image into a pre-trained classification model to output each feature data through the first fully connected layer in the classification model, specifically including:

[0026] Input the global image and the local image into a pre-trained classification model, so as to output, through a first fully connected layer in the classification model, each feature data corresponding to the global image as each first feature data, and output each feature data corresponding to the local image as each second feature data;

[0027] Input the feature data into a symptom classification layer in the classification model, so as to, through the symptom classification layer, determine, for each symptom, the probability that each feature data belongs to the symptom, specifically including:

[0028] Input the first feature data into a first symptom classification layer in the classification model, so as to, through the first symptom classification layer, determine, for each symptom, the probability that each first feature data belongs to the symptom; input the second feature data into a second symptom classification layer in the classification model, so as to, through the second symptom classification layer, determine, for each symptom, the probability that each second feature data belongs to the symptom;

[0029] Input the feature data and the probability that each feature data belongs to each symptom into a fusion layer in the classification model, so as to, through the fusion layer, determine, for each symptom, a fusion feature corresponding to the symptom based on the probability that each feature data belongs to the symptom and the feature data, specifically including:

[0030] Input the first feature data and the probability that each first feature data belongs to each symptom into a first fusion layer in the classification model, so as to, through the first fusion layer, determine, for each symptom, a first fusion feature corresponding to the symptom based on the probability that each first feature data belongs to the symptom and the first feature data;

[0031] Input the second feature data and the probability that each second feature data belongs to each symptom into a second fusion layer in the classification model, so as to, through the second fusion layer, determine, for each symptom, a second fusion feature corresponding to the symptom based on the probability that each second feature data belongs to the symptom and the second feature data;

[0032] Input the first fusion feature corresponding to the symptom and the second fusion feature corresponding to the symptom into a third fusion layer in the classification model, so as to, through the third fusion layer, fuse the first fusion feature corresponding to the symptom and the second fusion feature corresponding to the symptom to obtain a comprehensive fusion feature corresponding to the symptom;

[0033] Input the fusion feature corresponding to each symptom into a second fully connected layer in the classification model, so as to, through the second fully connected layer, output a symptom feature corresponding to each symptom, specifically including:

[0034] Input the comprehensive fusion features corresponding to each symptom into the second fully connected layer in the classification model, so as to output the symptom features corresponding to each symptom through the second fully connected layer.

[0035] Optionally, obtain the local images collected for the lesion site of the user, specifically including:

[0036] Input the global image into a pre-trained attention mechanism model to determine, through the attention mechanism model, the set of pixels located in the lesion part in the global image as the first set; input the global image into a pre-trained segmentation model to determine, through the segmentation model, the set of pixels located in the lesion part in the global image as the second set; wherein, the attention mechanism model and the segmentation model are trained using different training methods;

[0037] Determine the common pixel points in the first set and the second set;

[0038] Determine, from the global image, the image region of the minimum bounding rectangle containing the common pixel points as the local image.

[0039] Optionally, the first fully connected layer includes: a first sub-fully connected layer and a second sub-fully connected layer;

[0040] Input the global image and the local image into a pre-trained classification model to output, through the first fully connected layer in the classification model, the respective feature data corresponding to the global image as the respective first feature data, and output the respective feature data corresponding to the local image as the respective second feature data, specifically including:

[0041] Input the global image and the local image into a pre-trained classification model to extract features from the global image through the first encoding layer in the classification model to obtain a first image feature; extract features from the local image through the second encoding layer in the classification model to obtain a second image feature;

[0042] Input the first image feature into the pooling layer in the classification model to output a first pooled feature through the pooling layer; input the second image feature into the pooling layer in the classification model to output a second pooled feature through the pooling layer;

[0043] Input the first pooled feature into the first sub fully-connected layer in the classification model to output, through the first sub fully-connected layer, the respective feature data corresponding to the global image as respective first feature data; input the second pooled feature into the second sub fully-connected layer in the classification model to output, through the second sub fully-connected layer, the respective feature data corresponding to the local image as respective second feature data.

[0044] Optionally, concatenate the symptom features corresponding to each symptom to obtain a comprehensive symptom feature, specifically including:

[0045] Concatenate the symptom features corresponding to each symptom to obtain a concatenated feature;

[0046] Concatenate the concatenated feature and the first pooled feature to obtain a comprehensive symptom feature.

[0047] Optionally, before obtaining the image collected for the lesion site of the user, the method further includes:

[0048] Obtain a sample image collected for the lesion site of the patient;

[0049] Input the sample image into the classification model to be trained to output, through the first fully-connected layer in the classification model, respective feature data to be optimized;

[0050] Input the respective feature data to be optimized into the symptom classification layer in the classification model to determine, through the symptom classification layer, for each symptom, the probability that each feature data to be optimized belongs to that symptom;

[0051] Input the respective feature data to be optimized and the probability that each feature data to be optimized belongs to each symptom into the fusion layer in the classification model to determine, through the fusion layer, for each symptom, based on the probability that each feature data to be optimized belongs to that symptom and the respective feature data to be optimized, the fusion feature to be optimized corresponding to that symptom;

[0052] Input the fusion feature to be optimized corresponding to each symptom into the second fully-connected layer in the classification model to output, through the second fully-connected layer, the symptom feature to be optimized corresponding to each symptom;

[0053] Input the symptom feature to be optimized corresponding to each symptom into the concatenation layer in the classification model to concatenate, through the concatenation layer, the symptom features to be optimized corresponding to each symptom to obtain a comprehensive symptom feature to be optimized;

[0054] Input the comprehensive symptom feature to be optimized into the disease classification layer in the classification model to identify, through the disease classification layer, the disease type suffered by the patient based on the comprehensive symptom feature to be optimized.

[0055] The classification model is trained with the optimization objective of minimizing the difference between the disease type and the true disease type corresponding to the sample image.

[0056] Optionally, based on the probability that each feature data to be optimized belongs to this symptom and the feature data to be optimized, the fused feature to be optimized corresponding to this symptom is determined, specifically including:

[0057] For each feature data to be optimized, the probability that the feature data to be optimized belongs to this symptom is multiplied by the feature data to be optimized to obtain the local feature corresponding to the feature data to be optimized;

[0058] The local features corresponding to each feature data to be optimized are accumulated to obtain the fused feature to be optimized corresponding to this symptom.

[0059] Optionally, the classification model is trained with the optimization objective of minimizing the difference between the disease type and the true disease type corresponding to the sample image, specifically including:

[0060] The fused symptom features to be optimized corresponding to each symptom are input into a normalization layer for symptom feature classification, so as to classify the attributes of the fused symptom features to be optimized corresponding to each symptom through the normalization layer, and obtain the optimized attribute category corresponding to each symptom;

[0061] According to the difference between the optimized attribute category corresponding to each symptom and its true attribute category, the first loss is determined;

[0062] According to the difference between the disease type and the true disease type corresponding to the sample image, the second loss is determined;

[0063] According to the first loss and the second loss, the comprehensive loss is determined;

[0064] The classification model is trained with the optimization objective of minimizing the comprehensive loss.

[0065] Optionally, the sample image collected for the lesion site of the patient is obtained, specifically including:

[0066] The global sample image and the local sample image collected for the lesion site of the patient are obtained;

[0067] The sample image is input into the classification model to be trained, so as to output each feature data to be optimized through the first fully connected layer in the classification model, specifically including:

[0068] Input the global sample image and the local sample image into the classification model to be trained, so as to output, through the first fully connected layer in the classification model, the to-be-optimized feature data corresponding to the global sample image as each first to-be-optimized feature data, and output the to-be-optimized feature data corresponding to the local sample image as each second to-be-optimized feature data;

[0069] Input the to-be-optimized feature data into the symptom classification layer in the classification model, so as to, through the symptom classification layer, determine the probability that each to-be-optimized feature data belongs to each symptom for each symptom, specifically including:

[0070] Input the first to-be-optimized feature data into the first symptom classification layer in the classification model, so as to, through the first symptom classification layer, determine the probability that each first to-be-optimized feature data belongs to each symptom for each symptom; input the second to-be-optimized feature data into the second symptom classification layer in the classification model, so as to, through the second symptom classification layer, determine the probability that each second to-be-optimized feature data belongs to each symptom for each symptom;

[0071] Input the to-be-optimized feature data and the probability that each to-be-optimized feature data belongs to each symptom into the fusion layer in the classification model, so as to, through the fusion layer, for each symptom, determine the to-be-optimized fusion feature corresponding to the symptom based on the probability that each to-be-optimized feature data belongs to the symptom and the to-be-optimized feature data, specifically including:

[0072] Input the first to-be-optimized feature data and the probability that each first to-be-optimized feature data belongs to each symptom into the first fusion layer in the classification model, so as to, through the first fusion layer, for each symptom, determine the first to-be-optimized fusion feature corresponding to the symptom based on the probability that each first to-be-optimized feature data belongs to the symptom and the first to-be-optimized feature data;

[0073] Input the second to-be-optimized feature data and the probability that each second to-be-optimized feature data belongs to each symptom into the second fusion layer in the classification model, so as to, through the second fusion layer, for each symptom, determine the second to-be-optimized fusion feature corresponding to the symptom based on the probability that each second to-be-optimized feature data belongs to the symptom and the second to-be-optimized feature data;

[0074] Input the first to-be-optimized fusion feature corresponding to the symptom and the second to-be-optimized fusion feature corresponding to the symptom into the third fusion layer in the classification model, so as to, through the third fusion layer, fuse the first to-be-optimized fusion feature corresponding to the symptom and the second to-be-optimized fusion feature corresponding to the symptom to obtain the to-be-optimized comprehensive fusion feature corresponding to the symptom;

[0075] Input the fusion features to be optimized corresponding to each symptom into the second fully connected layer in the classification model, so as to output the symptom features to be optimized corresponding to each symptom through the second fully connected layer, which specifically includes:

[0076] Input the comprehensive fusion features to be optimized corresponding to each symptom into the second fully connected layer in the classification model, so as to output the symptom features to be optimized corresponding to each symptom through the second fully connected layer.

[0077] Optionally, input the sample global image and the sample local image into the classification model to be trained, so as to output the feature data to be optimized corresponding to the sample global image through the first fully connected layer in the classification model as each first feature data to be optimized, and output the feature data to be optimized corresponding to the sample local image as each second feature data to be optimized, which specifically includes:

[0078] Input the sample global image and the sample local image into the classification model to be trained, so as to extract features from the sample global image through the first encoding layer in the classification model to obtain the first image feature to be optimized; extract features from the sample local image through the second encoding layer in the classification model to obtain the second image feature to be optimized;

[0079] Input the first image feature to be optimized into the pooling layer in the classification model, so as to output the first pooled feature to be optimized through the pooling layer; input the second image feature to be optimized into the pooling layer in the classification model, so as to output the second pooled feature to be optimized through the pooling layer;

[0080] Input the first pooled feature to be optimized into the first sub fully connected layer in the classification model, so as to output the feature data to be optimized corresponding to the sample global image through the first sub fully connected layer as each first feature data to be optimized; input the second pooled feature to be optimized into the second sub fully connected layer in the classification model, so as to output the feature data to be optimized corresponding to the sample local image through the second sub fully connected layer as each second feature data to be optimized.

[0081] Optionally, splice the symptom features to be optimized corresponding to each symptom to obtain the comprehensive symptom features to be optimized, which specifically includes:

[0082] Splice the symptom features to be optimized corresponding to each symptom to obtain the spliced features to be optimized;

[0083] Splice the spliced features to be optimized with the first pooled feature to be optimized to obtain the comprehensive symptom features to be optimized.

[0084] Optionally, the symptom classification layer is a gating network layer for different diseases. For each symptom, the gating network layer of this symptom is composed of a fully connected layer of this symptom and a normalization layer.

[0085] An information recommendation device provided in this specification includes:

[0086] An acquisition module, configured to acquire an image collected for a lesion site of a user;

[0087] A first determination module, configured to input the image into a pre-trained classification model, so as to output respective feature data through a first fully connected layer in the classification model;

[0088] A second determination module, configured to input the respective feature data into a symptom classification layer in the classification model, so as to determine, for each symptom, the probability that each feature data belongs to this symptom through the symptom classification layer;

[0089] A fusion module, configured to input the respective feature data and the probability that each feature data belongs to each symptom into a fusion layer in the classification model, so as to determine, for each symptom, a fusion feature corresponding to this symptom based on the probability that each feature data belongs to this symptom and the respective feature data through the fusion layer;

[0090] A third determination module, configured to input the fusion feature corresponding to each symptom into a second fully connected layer in the classification model, so as to output a symptom feature corresponding to each symptom through the second fully connected layer;

[0091] A splicing module, configured to input the symptom feature corresponding to each symptom into a splicing layer in the classification model, so as to splice the symptom features corresponding to each symptom through the splicing layer to obtain a comprehensive symptom feature;

[0092] A disease recognition module, configured to input the comprehensive symptom feature into a disease classification layer in the classification model, so as to recognize the disease suffered by the user based on the comprehensive symptom feature through the disease classification layer;

[0093] An information recommendation module, configured to perform information recommendation based on the recognized disease of the user.

[0094] A computer-readable storage medium provided in this specification, where the storage medium stores a computer program, and when the computer program is executed by a processor, the above information recommendation method is implemented.

[0095] An electronic device provided in this specification includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above information recommendation method is implemented.

[0096] The above at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0097] After obtaining the image collected for the lesion site of the user in the embodiments of this specification, the image is input into the classification model. The classification model determines each feature data and determines the probability that each feature data belongs to each symptom. Based on the probability that each feature data belongs to each symptom and each feature data, the symptom features corresponding to each symptom are determined. The symptom features corresponding to each symptom are spliced to obtain the comprehensive symptom features, and the disease suffered by the user is identified based on the comprehensive symptom features. Finally, information recommendation is performed based on the identified disease. In this method, the symptoms at the lesion site can be decoupled to obtain the symptom features of each symptom, and then the disease suffered by the patient can be determined based on the symptom features of each symptom, which can effectively improve the accuracy of classifying diseases with similar symptoms. Description of the Drawings

[0098] The drawings described herein are used to provide a further understanding of this specification, and constitute a part of this specification. The schematic embodiments of this specification and their descriptions are used to explain this specification and do not constitute an improper limitation to this specification. In the drawings:

[0099] Figure 1 It is a schematic flowchart of the information recommendation method provided by the embodiments of this specification;

[0100] Figure 2 It is a schematic diagram of the use of the classification model for disease classification using one image provided by the embodiments of this specification;

[0101] Figure 3 It is a schematic diagram of the training of the classification model for disease classification using one image provided by the embodiments of this specification;

[0102] Figure 4 It is a schematic diagram of the training of the classification model for disease classification using two images provided by the embodiments of this specification;

[0103] Figure 5 It is a schematic diagram of the use of the classification model for disease classification using two images provided by the embodiments of this specification;

[0104] Figure 6 It is a schematic diagram of the structure of the information recommendation device provided by the embodiments of this specification;

[0105] Figure 7 It is a schematic diagram of the structure of the electronic device provided by the embodiments of this specification. Detailed Embodiments

[0106] The information recommendation method provided in this specification aims to classify the symptoms of the images collected for the lesion site through a classification model, identify the types of diseases suffered by the patient based on the symptom characteristics of each symptom, and perform information recommendation based on the identified disease types.

[0107] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0108] The following will detail the technical solutions provided in each embodiment of this specification in conjunction with the drawings.

[0109] Figure 1 It is a schematic flowchart of the information recommendation method provided in an embodiment of this specification, including:

[0110] S100: Obtain the image collected for the lesion site of the user.

[0111] In the embodiment of this specification, the image of the patient's lesion site can be collected first through a medical imaging device, and the image collected for the lesion site of the user (i.e., the patient) can be obtained. Among them, the image can be a color image. In addition, the image can refer to the global image collected for the diseased site including the lesion site and / or the local image collected only for the lesion site.

[0112] For example: If the lesion part of the patient is the eyeball, the global image can be an image of the entire eye, and the local image can be an image containing only the eyeball.

[0113] S102: Input the image into a pre-trained classification model to output each feature data through the first fully connected layer in the classification model.

[0114] In the embodiments of this specification, after acquiring the images collected for the lesion site of a user, the images can be input into a pre-trained classification model to identify or classify the type of disease suffered by the user through the classification model. Among them, the model structure of the classification model at least includes: an encoding layer for extracting image features, a pooling layer for dimensionality reduction, a first fully connected layer for linear mapping, a symptom classification layer for symptom classification, a fusion layer for fusing the features of each symptom, a symptom feature classification layer for symptom feature classification, a splicing layer for splicing symptom features, a disease classification layer for disease classification, etc. The symptom feature classification layer can be composed of a second fully connected layer and a normalization layer for disease symptom feature classification. The symptom classification layer can be composed of a third fully connected layer corresponding to each symptom and a normalization layer corresponding to each symptom. The fusion layer can be composed of multiple fourth fully connected layers for linear mapping. The splicing layer can be composed of a fifth fully connected layer. The disease classification layer can be composed of multiple sixth fully connected layers for disease type mapping and a normalization layer for disease classification.

[0115] In the embodiments of this specification, the images can be first input into the encoding layer of the pre-trained classification model to extract the features of the images through the encoding layer and obtain the image features of the images. Among them, the encoding layer can refer to a residual convolutional network. Then, the image features are input into the pooling layer of the classification model to perform average pooling on the image features through the pooling layer and obtain the pooled features. After that, the pooled features are input into the first fully connected layer of the classification model to output the respective feature data represented by the lesion site through the first fully connected layer. Among them, there can be multiple first fully connected layers, and the respective feature data can be the features obtained after the linear mapping of the pooled features through multiple first fully connected layers.

[0116] For example: The image features are: feat = Encoder(image), feat ∈ R c*h*w , where c represents the number of image convolution channels, h represents the height of the image, and w represents the width of the image. The pooled features are: F 池 = Pool avg (feat), F 池 ∈ R c . c represents the feature dimension. The set of respective feature data is: F 集 = {F ′ 0, F ′ 1,..., F ′ n}, for any one of the feature data F ′ n , F ′ n = FC1(ReLU(FC0(F 池))),FC0 and FC1 represent the first fully connected layer, and ReLU represents the activation function.

[0117] S104: Input the respective feature data into the symptom classification layer in the classification model, so as to, through the symptom classification layer, for each symptom, determine the probability that each feature data belongs to this symptom.

[0118] In the embodiments of this specification, after obtaining the respective feature data, the feature data related to each symptom can be screened out from the respective feature data through the symptom classification layer in the classification model. Among them, the symptom classification layer can be a gated network layer for different symptoms. For each symptom, the gated network layer of this symptom is composed of the third fully connected layer and the normalization layer of this symptom.

[0119] Specifically, input the respective feature data into the symptom classification layer in the classification model, so as to, through the symptom classification layer, for each symptom, determine the probability that each feature data belongs to this symptom. The greater the probability that a certain feature data belongs to this symptom, the more relevant this feature data is to this symptom. That is, the more this feature data can characterize this symptom.

[0120] Taking keratitis as an example, keratitis can be divided into four disease categories such as Acanthamoeba keratitis, bacterial keratitis, fungal keratitis, and viral keratitis. And the symptoms shown by different keratitis diseases can be divided into: granular sensation, density, distribution uniformity, edge shape, surface dryness and humidity, etc. If there are 10 feature data, perhaps the probability of feature data 1 and feature data 4 in characterizing the granular sensation is relatively large, the probability of feature data 2 and feature data 3 in characterizing the density is relatively large, the probability of feature data 5 and feature data 7 in characterizing the distribution uniformity is relatively large, the probability of feature data 6 and feature data 10 in characterizing the edge shape is relatively large, and the probability of feature data 8 and feature data 9 in characterizing the surface dryness and humidity is relatively large.

[0121] S106: Input the respective feature data and the probability that each feature data belongs to each symptom into the fusion layer in the classification model, so as to, through the fusion layer, for each symptom, based on the probability that each feature data belongs to this symptom and the respective feature data, determine the fusion feature corresponding to this symptom.

[0122] In the embodiments of this specification, through the fusion layer in the classification model, for each symptom, the feature data that can characterize this symptom can be selected from the respective feature data, and the selected feature data can be fused to determine the total feature that can characterize this symptom.

[0123] Specifically, input each piece of feature data and the probability of each piece of feature data belonging to each type of symptom into the fusion layer in the classification model, so that through the fusion layer, for each type of symptom, based on the probability of each piece of feature data belonging to this type of symptom and each piece of feature data, determine the fusion feature corresponding to this type of symptom, that is, the total feature.

[0124] Further, for each piece of feature data under each type of symptom, use the probability of this piece of feature data belonging to this type of symptom as a weight, multiply the probability of this piece of feature data belonging to this type of symptom by this piece of feature data to obtain the local feature corresponding to this piece of feature data. Then, accumulate the local features corresponding to each piece of feature data to obtain the fusion feature corresponding to this type of symptom.

[0125] For example: F e represents the fusion feature of the e-th type of symptom, n represents n pieces of feature data, g i represents the probability that the i-th piece of feature data belongs to the e-th type of symptom, F[[ID=I3]] i ′ represents the i-th piece of feature data.

[0126] S108: Input the fusion feature corresponding to each type of symptom into the second fully connected layer in the classification model, so as to output the symptom feature corresponding to each type of symptom through the second fully connected layer.

[0127] In the embodiments of this specification, after determining the fusion feature corresponding to each type of symptom, the fusion feature of each type of symptom can be classified to determine the symptom feature corresponding to each type of symptom. Among them, the symptom feature corresponding to each type of symptom can represent the attribute of each type of symptom. For example: whether the granularity is obvious or not, whether the density is large or small, whether the inflammation is evenly distributed or not, whether the edge shape is circular or arc-shaped, whether the surface is dry or wet, etc.

[0128] Specifically, input the fusion feature corresponding to each type of symptom into the symptom feature classification layer, so as to output the symptom feature corresponding to each type of symptom through the second fully connected layer in the symptom feature classification layer.

[0129] S110: Input the symptom feature corresponding to each type of symptom into the splicing layer in the classification model, so as to splice the symptom features corresponding to each type of symptom through the splicing layer to obtain the comprehensive symptom feature.

[0130] S112: Input the comprehensive symptom feature into the disease classification layer in the classification model, so as to identify the disease suffered by the user based on the comprehensive symptom feature through the disease classification layer.

[0131] S114: Perform information recommendation based on the identified disease suffered by the user.

[0132] In the embodiments of this specification, after obtaining the symptom features corresponding to each symptom, the symptom features of each symptom can be comprehensively considered to distinguish the disease type represented by the input lesion site.

[0133] Specifically, the symptom features corresponding to each symptom are input into the splicing layer in the classification model, so that through the splicing layer, the symptom features corresponding to each symptom are spliced to obtain comprehensive symptom features. Then, the comprehensive symptom features are input into the disease classification layer in the classification model, so that through the disease classification layer, based on the comprehensive symptom features, the disease type suffered by the user (i.e., the patient) is identified. That is, the disease type represented by the lesion site is identified.

[0134] Taking keratitis as an example, if the granular sensation is not obvious, the density is not high, the distribution is uneven, the edge shape is semi-circular, and the surface humidity is high, it may be identified as Acanthamoeba keratitis; if the granular sensation is not obvious, the density is high, the distribution is uneven, the edge shape is toothpaste-like, and the surface is dry, it may be identified as fungal keratitis.

[0135] After identifying the disease type suffered by the user through the classification model, information recommendation can be performed for the identified disease type.

[0136] Specifically, for the identified disease type, treatment information matching the identified disease type can be recommended to the user. Among them, the treatment information may include at least one of treatment equipment, treatment drugs, and treatment doctor information.

[0137] Based on the disease type identification process described in the above steps S100 to S112, this specification provides a schematic diagram of the use of a classification model for disease classification using one image, as Figure 2 shown.

[0138] It should be noted that all actions of obtaining signals, information, or image data in this application are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where the location is located and obtaining the authorization given by the owner of the corresponding device.

[0139] Through the above Figure 1As can be seen from the method shown, after obtaining the image collected for the lesion site of the user, the image is input into the classification model. The classification model determines each feature data and determines the probability that each feature data belongs to each symptom. Based on the probability that each feature data belongs to each symptom and each feature data, the symptom feature corresponding to each symptom is determined. The symptom features corresponding to each symptom are concatenated to obtain the comprehensive symptom feature, and the disease suffered by the user is identified based on the comprehensive symptom feature. Finally, information recommendation is performed based on the identified disease. In this method, the symptoms at the lesion site can be decoupled to obtain the symptom features of each symptom, and then the disease suffered by the patient can be determined based on the symptom features of each symptom, which can effectively improve the accuracy of classifying diseases with similar symptoms.

[0140] Further, before using the classification model, the classification model for classifying diseases using a single image is trained, as Figure 3 shown.

[0141] In Figure 3 , the sample image collected for the lesion site of the patient can be obtained first, and then the sample image is input into the classification model to be trained, so that each to-be-optimized feature data corresponding to the sample image is output through the first fully connected layer in the classification model.

[0142] Specifically, the sample image is input into the classification model to be trained, and the encoding layer in the classification model is used to extract features from the sample image to obtain the to-be-optimized image features. Then, the to-be-optimized image features are input into the pooling layer in the classification model to output the to-be-optimized pooled features through the pooling layer. Finally, the to-be-optimized pooled features are input into the first fully connected layer in the classification model to output each to-be-optimized feature data corresponding to the sample image through the first fully connected layer.

[0143] After determining each to-be-optimized feature data, each to-be-optimized feature data is input into the symptom classification layer in the classification model, so that, for each symptom, the probability that each to-be-optimized feature data belongs to this symptom is determined through the symptom classification layer. Then, each to-be-optimized feature data and the probability that each to-be-optimized feature data belongs to each symptom are input into the fusion layer in the classification model, so that, for each symptom, based on the probability that each to-be-optimized feature data belongs to this symptom and each to-be-optimized feature data, the to-be-optimized fusion feature corresponding to this symptom is determined through the fusion layer.

[0144] Specifically, for each to-be-optimized feature data under each symptom, the probability that the to-be-optimized feature data belongs to this symptom is multiplied by the to-be-optimized feature data to obtain the local feature corresponding to the to-be-optimized feature data. Then, the local features corresponding to each to-be-optimized feature data are accumulated to obtain the to-be-optimized fusion feature corresponding to this symptom.

[0145] Input the fusion features to be optimized corresponding to each symptom into the second fully connected layer of the symptom feature classification layer in the classification model, so as to output the symptom features to be optimized corresponding to each symptom through the second fully connected layer.

[0146] After determining the symptom features to be optimized corresponding to each symptom, the symptom features to be optimized corresponding to each symptom can be input into the concatenation layer in the classification model, so as to concatenate the symptom features to be optimized corresponding to each symptom through the concatenation layer to obtain the comprehensive symptom features to be optimized.

[0147] Input the comprehensive symptom features to be optimized into the disease classification layer in the classification model, so as to identify the disease type suffered by the patient based on the comprehensive symptom features to be optimized through the disease classification layer. Take the minimum difference between the identified disease type and the true disease type corresponding to the sample image as the optimization goal to train the classification model.

[0148] When training the classification model, it is also necessary to take the minimum difference between the attribute category identified by the symptom features of each symptom and the true attribute category as the optimization goal to train the classification model to ensure the accurate feature expression of the symptom features of each symptom.

[0149] Specifically, after outputting the symptom features to be optimized corresponding to each symptom through the second fully connected layer of the symptom feature classification layer, the symptom features to be optimized corresponding to each symptom can be input into the normalization layer of the symptom feature classification layer, so as to perform attribute classification on the symptom features to be optimized corresponding to each symptom through the normalization layer for symptom feature classification to obtain the optimized attribute category corresponding to each symptom. Then, for each symptom, determine the difference between the optimized attribute category corresponding to this symptom and the true attribute category of this symptom as the difference corresponding to this symptom. Take the minimum difference corresponding to each symptom and the minimum difference between the identified disease type and the true disease type corresponding to the sample image as the optimization goal to train the classification model.

[0150] Furthermore, the first loss can be determined according to the difference between the optimized attribute category corresponding to each symptom and its respective true attribute category. At the same time, the second loss can be determined according to the difference between the identified disease type and the true disease type corresponding to the sample image. Finally, the comprehensive loss is determined according to the first loss and the second loss. Take the minimum comprehensive loss as the optimization goal to train the classification model. Among them, the first loss and the second loss can refer to the cross-entropy loss.

[0151] When determining the first loss, for each symptom, the loss corresponding to the symptom can be determined according to the difference between the attribute category to be optimized corresponding to the symptom and the true attribute category of the symptom. Then, the losses corresponding to each symptom are accumulated to obtain the first loss.

[0152] When determining the comprehensive loss, the first loss and the second loss can be weighted and summed based on the first weight corresponding to the first loss and the second weight of the second loss to obtain the comprehensive loss.

[0153] Among them, the formula for the first loss is: represents the nth attribute category of the ith symptom, and t represents the tth symptom. For the second loss: n represents the nth disease type. The formula for the comprehensive loss is: Loss 综 = α * Loss1 + β * Loss2, where α represents the first weight and β represents the second weight.

[0154] In addition, in order to enable the classification model to learn or express more accurate features, feature extraction, pooling, and symptom classification can be performed on two images collected for the lesion site. When performing the above processing on the two images, independent encoding layers, fully connected layers, and symptom classification layers can be used.

[0155] Next, first train the classification model for disease classification using two images, as Figure 4 shown.

[0156] First, a sample global image collected for the lesion part can be obtained, and then, based on the sample global image, a sample local image of the image area that only involves the lesion site can be determined. Among them, the sample global image can refer to an image collected including the lesion site, and the sample local image can refer to an image collected only for the lesion part.

[0157] Specifically, the sample global image can be input into a pre-trained attention mechanism model to determine, through the attention mechanism model, the set of pixels located in the lesion part of the sample global image as the first pixel set. Then, the sample global image can be input into a pre-trained segmentation model to determine, through the segmentation model, the set of pixels located in the lesion part of the sample global image as the second pixel set. After that, the common pixel points in the first pixel set and the second pixel set are determined, and the image area of the minimum circumscribed rectangle containing the common pixel points is determined from the sample global image as the sample local image. Among them, the attention mechanism model and the segmentation model are trained using different training methods. The attention mechanism model can be trained based on the label method, and the segmentation model can be trained based on the mask reconstruction method.

[0158] In Figure 4 it, after obtaining the sample global image and the sample local image, the sample global image and the sample local image are input into the classification model to be trained, so as to output, through the first fully-connected layer in the classification model, the respective to-be-optimized feature data corresponding to the sample global image as the respective first to-be-optimized feature data, and output the respective to-be-optimized feature data corresponding to the sample local image as the respective second to-be-optimized feature data. Among them, the first fully-connected layer may include: a first sub-fully-connected layer and a second sub-fully-connected layer.

[0159] Specifically, the sample global image and the sample local image are input into the classification model to be trained, so as to extract features from the sample global image through the first encoding layer in the classification model to obtain the first to-be-optimized image features. At the same time, features are extracted from the sample local image through the second encoding layer in the classification model to obtain the second to-be-optimized image features. The first to-be-optimized image features are input into the pooling layer in the classification model to output, through the pooling layer, the first to-be-optimized pooled features. The second to-be-optimized image features are input into the pooling layer in the classification model to output, through the pooling layer, the second to-be-optimized pooled features. The first to-be-optimized pooled features are input into the first sub-fully-connected layer in the classification model to output, through the first sub-fully-connected layer, the respective to-be-optimized feature data corresponding to the sample global image as the respective first to-be-optimized feature data. At the same time, the second to-be-optimized pooled features are input into the second sub-fully-connected layer in the classification model to output, through the second sub-fully-connected layer, the respective to-be-optimized feature data corresponding to the sample local image as the respective second to-be-optimized feature data.

[0160] The respective first to-be-optimized feature data are input into the first symptom classification layer in the classification model to determine, through the first symptom classification layer, for each symptom, the probability that each first to-be-optimized feature data belongs to the symptom. At the same time, the respective second to-be-optimized feature data are input into the second symptom classification layer in the classification model to determine, through the second symptom classification layer, for each symptom, the probability that each second to-be-optimized feature data belongs to the symptom.

[0161] The respective first to-be-optimized feature data and the probability that each first to-be-optimized feature data belongs to each symptom are input into the first fusion layer in the classification model to determine, through the first fusion layer, for each symptom, the first to-be-optimized fusion feature corresponding to the symptom based on the probability that each first to-be-optimized feature data belongs to the symptom and the respective first to-be-optimized feature data.

[0162] Specifically, for each first feature data to be optimized under each symptom, multiply the probability that the first feature data to be optimized belongs to this symptom by the first feature data to be optimized to obtain the local feature corresponding to the first feature data to be optimized. Then, accumulate the local features corresponding to each first feature data to be optimized to obtain the first optimized fusion feature corresponding to this symptom.

[0163] Meanwhile, input each second feature data to be optimized and the probability that each second feature data to be optimized belongs to each symptom into the second fusion layer in the classification model, so that through the second fusion layer, for each symptom, based on the probability that each second feature data to be optimized belongs to this symptom and each second feature data to be optimized, determine the second optimized fusion feature corresponding to this symptom.

[0164] Specifically, for each second feature data to be optimized under each symptom, multiply the probability that the second feature data to be optimized belongs to this symptom by the second feature data to be optimized to obtain the local feature corresponding to the second feature data to be optimized. Then, accumulate the local features corresponding to each second feature data to be optimized to obtain the second optimized fusion feature corresponding to this symptom.

[0165] After obtaining the first optimized fusion feature corresponding to each symptom and the second optimized fusion feature corresponding to each symptom, for each symptom, the first optimized fusion feature corresponding to this symptom and the second optimized fusion feature corresponding to this symptom can be input into the third fusion layer in the classification model, so that through the third fusion layer, fuse the first optimized fusion feature corresponding to this symptom and the second optimized fusion feature corresponding to this symptom to obtain the optimized comprehensive fusion feature corresponding to this symptom.

[0166] After obtaining the optimized comprehensive fusion feature corresponding to each symptom, the optimized comprehensive fusion feature corresponding to each symptom can be input into the second fully connected layer in the classification model, so that through the second fully connected layer, output the optimized symptom feature corresponding to each symptom.

[0167] Input the optimized symptom feature corresponding to each symptom into the splicing layer in the classification model, so that through the splicing layer, splice the optimized symptom features corresponding to each symptom to obtain the optimized comprehensive symptom feature.

[0168] Specifically, the optimized symptom features corresponding to each symptom can be directly spliced to obtain the post-splicing feature to be optimized, which is used as the optimized comprehensive symptom feature.

[0169] Input the optimized comprehensive symptom feature into the disease classification layer in the classification model, so that through the disease classification layer, based on the optimized comprehensive symptom feature, identify the disease type suffered by the patient.

[0170] The classification model is trained with the optimization objective of minimizing the difference between the identified disease type and the true disease type corresponding to the sample image.

[0171] When training the classification model, after the second fully connected layer of the symptom feature classification layer outputs the symptom features to be optimized corresponding to each symptom, the symptom features to be optimized corresponding to each symptom can be input into the normalization layer of the symptom feature classification layer, so as to perform attribute classification on the symptom features to be optimized corresponding to each symptom through the normalization layer for symptom feature classification, and obtain the attribute categories to be optimized corresponding to each symptom. Then, for each symptom, the difference between the attribute category to be optimized corresponding to this symptom and the true attribute category of this symptom is determined as the difference corresponding to this symptom. The classification model is trained with the optimization objective of minimizing the difference corresponding to each symptom and minimizing the difference between the identified disease type and the true disease type corresponding to the sample image.

[0172] In addition, in order to improve the accuracy of the classification model in identifying disease types, the symptom features to be optimized corresponding to each symptom can be concatenated through a concatenation layer to obtain the concatenated features to be optimized. Then, the concatenated features to be optimized are concatenated with the first pooled features to be optimized to obtain the comprehensive symptom features to be optimized.

[0173] After the training of the classification model including two independent encoding layers, fully connected layers, and disease classification layers is completed, the classification model can be used to classify diseases with two images, as Figure 5 shown.

[0174] First, the global image collected for the lesion site can be obtained. Then, the global image is input into a pre-trained attention mechanism model to determine, through the attention mechanism model, the pixel set located in the lesion part of the global image as the first set. The global image is input into a pre-trained segmentation model to determine, through the segmentation model, the pixel set located in the lesion part of the global image as the second set. The common pixel points in the first set and the second set are determined, and the image region of the minimum circumscribed rectangle containing the common pixel points is determined from the global image as the local image.

[0175] In Figure 5 , after obtaining the global image and the local image, the global image and the local image can be input into a pre-trained classification model, so that each feature data corresponding to the global image is output through the first fully connected layer in the classification model as each first feature data, and each feature data corresponding to the local image is output as each second feature data.

[0176] Specifically, the global image and the local image are input into a pre-trained classification model. Through the first encoding layer in the classification model, feature extraction is performed on the global image to obtain the first image feature. Through the second encoding layer in the classification model, feature extraction is performed on the local image to obtain the second image feature. The first image feature is input into the pooling layer in the classification model to output the first pooled feature through the pooling layer. The second image feature is input into the pooling layer in the classification model to output the second pooled feature through the pooling layer. The first pooled feature is input into the first sub fully-connected layer in the classification model to output the respective feature data corresponding to the global image through the first sub fully-connected layer, as the respective first feature data. The second pooled feature is input into the second sub fully-connected layer in the classification model to output the respective feature data corresponding to the local image through the second sub fully-connected layer, as the respective second feature data.

[0177] The respective first feature data are input into the first symptom classification layer in the classification model. Through the first symptom classification layer, for each symptom, the probability that each first feature data belongs to that symptom is determined. The respective second feature data are input into the second symptom classification layer in the classification model. Through the second symptom classification layer, for each symptom, the probability that each second feature data belongs to that symptom is determined.

[0178] The respective first feature data and the probability that each first feature data belongs to each symptom are input into the first fusion layer in the classification model. Through the first fusion layer, for each symptom, based on the probability that each first feature data belongs to that symptom and the respective first feature data, the first fusion feature corresponding to that symptom is determined.

[0179] The respective second feature data and the probability that each second feature data belongs to each symptom are input into the second fusion layer in the classification model. Through the second fusion layer, for each symptom, based on the probability that each second feature data belongs to that symptom and the respective second feature data, the second fusion feature corresponding to that symptom is determined.

[0180] Specifically, under each symptom, for each first feature data, the probability that the first feature data belongs to that symptom is multiplied by the first feature data to obtain the local feature corresponding to the first feature data. The local features corresponding to each first feature data are accumulated to obtain the first fusion feature corresponding to that symptom. Similarly, for each second feature data, the probability that the second feature data belongs to that symptom is multiplied by the second feature data to obtain the local feature corresponding to the second feature data. The local features corresponding to each second feature data are accumulated to obtain the second fusion feature corresponding to that symptom.

[0181] For each symptom, input the first fusion feature corresponding to the symptom and the second fusion feature corresponding to the symptom into the third fusion layer in the classification model, so as to fuse the first fusion feature corresponding to the symptom and the second fusion feature corresponding to the symptom through the third fusion layer, and obtain the comprehensive fusion feature corresponding to the symptom.

[0182] After obtaining the comprehensive fusion feature corresponding to each symptom, the comprehensive fusion feature corresponding to each symptom can be input into the second fully connected layer in the classification model, so as to output the symptom feature corresponding to each symptom through the second fully connected layer.

[0183] Input the symptom feature corresponding to each symptom into the splicing layer in the classification model, so as to splice the symptom features corresponding to each symptom through the splicing layer and obtain the comprehensive symptom feature.

[0184] Specifically, through the splicing layer, splice the symptom features corresponding to each symptom to obtain the spliced feature, which is used as the comprehensive symptom feature.

[0185] Input the comprehensive symptom feature into the disease classification layer in the classification model, so as to identify the disease suffered by the user based on the comprehensive symptom feature through the disease classification layer.

[0186] In addition, through the splicing layer, the symptom features corresponding to each symptom can be spliced to obtain the spliced feature, and then, the spliced feature and the first pooled feature can be spliced to obtain the comprehensive symptom feature.

[0187] The above is the information recommendation method provided in the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media, and electronic devices.

[0188] Figure 6 The following is a schematic structural diagram of an information recommendation device provided in the embodiments of this specification. The device includes:

[0189] An acquisition module 601, configured to acquire an image collected for the lesion site of a user;

[0190] A first determination module 602, configured to input the image into a pre-trained classification model, so as to output each feature data through the first fully connected layer in the classification model;

[0191] A second determination module 603, configured to input each feature data into the symptom classification layer in the classification model, so as to determine the probability that each feature data belongs to each symptom through the symptom classification layer;

[0192] A fusion module 604, configured to input the respective feature data and the probability of each feature data belonging to each symptom into a fusion layer in the classification model, so that, through the fusion layer, for each symptom, based on the probability of each feature data belonging to this symptom and the respective feature data, determine a fusion feature corresponding to this symptom;

[0193] A third determination module 605, configured to input the fusion feature corresponding to each symptom into a second fully-connected layer in the classification model, so as to output, through the second fully-connected layer, a symptom feature corresponding to each symptom;

[0194] A splicing module 606, configured to input the symptom feature corresponding to each symptom into a splicing layer in the classification model, so that, through the splicing layer, splice the symptom features corresponding to each symptom to obtain a comprehensive symptom feature;

[0195] A disease recognition module 607, configured to input the comprehensive symptom feature into a disease classification layer in the classification model, so as to, through the disease classification layer, based on the comprehensive symptom feature, recognize the disease suffered by the user;

[0196] An information recommendation module 608, configured to perform information recommendation based on the recognized disease suffered by the user.

[0197] Optionally, the acquisition module 601 is specifically configured to acquire a global image collected for a lesion site of a user and acquire a local image collected for the lesion site of the user.

[0198] Optionally, the acquisition module 601 is specifically configured to input the global image into a pre-trained attention mechanism model, so as to determine, through the attention mechanism model, a pixel set located in the lesion part in the global image as a first set; input the global image into a pre-trained segmentation model, so as to determine, through the segmentation model, a pixel set located in the lesion part in the global image as a second set; wherein, the attention mechanism model and the segmentation model are trained by different training methods; determine the common pixel points in the first set and the second set; determine, from the global image, an image region of the smallest circumscribed rectangle containing the common pixel points as a local image.

[0199] Optionally, the first determination module 602 is specifically configured to input the image into a pre-trained classification model, so as to perform feature extraction on the image through an encoding layer in the classification model to obtain an image feature; input the image feature into a pooling layer in the classification model, so as to output, through the pooling layer, a pooled feature; input the pooled feature into a first fully-connected layer in the classification model, so as to output, through the first fully-connected layer, the respective feature data represented by the lesion site.

[0200] Optionally, the first determination module 602 is specifically configured to input the global image and the local image into a pre-trained classification model, so as to output, through a first fully connected layer in the classification model, each feature data corresponding to the global image as each first feature data, and output each feature data corresponding to the local image as each second feature data.

[0201] Optionally, the first fully connected layer includes: a first sub-fully connected layer and a second sub-fully connected layer.

[0202] Optionally, the first determination module 602 is specifically configured to input the global image and the local image into a pre-trained classification model, so as to extract features of the global image through a first encoding layer in the classification model to obtain a first image feature; extract features of the local image through a second encoding layer in the classification model to obtain a second image feature; input the first image feature into a pooling layer in the classification model to output a first pooled feature through the pooling layer; input the second image feature into a pooling layer in the classification model to output a second pooled feature through the pooling layer; input the first pooled feature into the first sub-fully connected layer in the classification model to output, through the first sub-fully connected layer, each feature data corresponding to the global image as each first feature data; input the second pooled feature into the second sub-fully connected layer in the classification model to output, through the second sub-fully connected layer, each feature data corresponding to the local image as each second feature data.

[0203] Optionally, the second determination module 603 is specifically configured to input each first feature data into a first symptom classification layer in the classification model, so as to determine, through the first symptom classification layer, for each symptom, the probability that each first feature data belongs to the symptom; input each second feature data into a second symptom classification layer in the classification model, so as to determine, through the second symptom classification layer, for each symptom, the probability that each second feature data belongs to the symptom.

[0204] Optionally, the fusion module 604 is specifically configured to multiply, for each feature data, the probability that the feature data belongs to the symptom by the feature data to obtain a local feature corresponding to the feature data; accumulate the local features corresponding to each feature data to obtain a fusion feature corresponding to the symptom.

[0205] Optionally, the fusion module 604 is specifically configured to input each first feature data and the probability of each first feature data belonging to each symptom into the first fusion layer in the classification model, so that through the first fusion layer, for each symptom, based on the probability of each first feature data belonging to this symptom and each first feature data, determine the first fusion feature corresponding to this symptom; input each second feature data and the probability of each second feature data belonging to each symptom into the second fusion layer in the classification model, so that through the second fusion layer, for each symptom, based on the probability of each second feature data belonging to this symptom and each second feature data, determine the second fusion feature corresponding to this symptom; input the first fusion feature corresponding to this symptom and the second fusion feature corresponding to this symptom into the third fusion layer in the classification model, so that through the third fusion layer, fuse the first fusion feature corresponding to this symptom and the second fusion feature corresponding to this symptom to obtain the comprehensive fusion feature corresponding to this symptom.

[0206] Optionally, the third determination module 605 is specifically configured to input the comprehensive fusion feature corresponding to each symptom into the second fully connected layer in the classification model, so as to output the symptom feature corresponding to each symptom through the second fully connected layer.

[0207] Optionally, the splicing module 606 is specifically configured to splice the symptom features corresponding to each symptom to obtain the spliced feature; splice the spliced feature and the first pooled feature to obtain the comprehensive symptom feature.

[0208] Optionally, the apparatus further includes: a training module 609;

[0209] A training module 609, configured to obtain a sample image collected for a lesion site of a patient; input the sample image into a classification model to be trained, so as to output respective feature data to be optimized through a first fully connected layer in the classification model; input the respective feature data to be optimized into a symptom classification layer in the classification model, so as to, through the symptom classification layer, for each symptom, determine the probability that each feature data to be optimized belongs to this symptom; input the respective feature data to be optimized and the probability that each feature data to be optimized belongs to each symptom into a fusion layer in the classification model, so as to, through the fusion layer, for each symptom, based on the probability that each feature data to be optimized belongs to this symptom and the respective feature data to be optimized, determine a fusion feature to be optimized corresponding to this symptom; input the fusion feature to be optimized corresponding to each symptom into a second fully connected layer in the classification model, so as to output a symptom feature to be optimized corresponding to each symptom through the second fully connected layer; input the symptom feature to be optimized corresponding to each symptom into a splicing layer in the classification model, so as to, through the splicing layer, splice the symptom features to be optimized corresponding to each symptom to obtain a comprehensive symptom feature to be optimized; input the comprehensive symptom feature to be optimized into a disease classification layer in the classification model, so as to, through the disease classification layer, based on the comprehensive symptom feature to be optimized, identify the disease type suffered by the patient; and train the classification model with the objective of minimizing the difference between the disease type and the true disease type corresponding to the sample image.

[0210] Optionally, the training module 609 is specifically configured to, for each feature data to be optimized, multiply the probability that the feature data to be optimized belongs to this symptom by the feature data to be optimized to obtain a local feature corresponding to the feature data to be optimized; and accumulate the local features corresponding to each feature data to be optimized to obtain a fusion feature to be optimized corresponding to this symptom.

[0211] Optionally, the training module 609 is specifically configured to input the symptom feature to be optimized corresponding to each symptom into a normalization layer for symptom feature classification, so as to, through the normalization layer, perform attribute classification on the symptom feature to be optimized corresponding to each symptom to obtain an attribute category to be optimized corresponding to each symptom; determine a first loss according to the difference between the attribute category to be optimized corresponding to each symptom and its respective true attribute category; determine a second loss according to the difference between the disease type and the true disease type corresponding to the sample image; determine a comprehensive loss according to the first loss and the second loss; and train the classification model with the objective of minimizing the comprehensive loss.

[0212] Optionally, the training module 609 is specifically configured to obtain a sample global image and a sample local image collected for the lesion site of a patient; input the sample global image and the sample local image into a classification model to be trained, so as to output, through a first fully connected layer in the classification model, each to-be-optimized feature data corresponding to the sample global image as each first to-be-optimized feature data, and output each to-be-optimized feature data corresponding to the sample local image as each second to-be-optimized feature data; input the first to-be-optimized feature data into a first symptom classification layer in the classification model, so as to determine, through the first symptom classification layer, for each symptom, the probability that each first to-be-optimized feature data belongs to the symptom; input the second to-be-optimized feature data into a second symptom classification layer in the classification model, so as to determine, through the second symptom classification layer, for each symptom, the probability that each second to-be-optimized feature data belongs to the symptom; input the first to-be-optimized feature data and the probability that each first to-be-optimized feature data belongs to each symptom into a first fusion layer in the classification model, so as to determine, through the first fusion layer, for each symptom, a first to-be-optimized fusion feature corresponding to the symptom based on the probability that each first to-be-optimized feature data belongs to the symptom and the first to-be-optimized feature data; input the second to-be-optimized feature data and the probability that each second to-be-optimized feature data belongs to each symptom into a second fusion layer in the classification model, so as to determine, through the second fusion layer, for each symptom, a second to-be-optimized fusion feature corresponding to the symptom based on the probability that each second to-be-optimized feature data belongs to the symptom and the second to-be-optimized feature data; input the first to-be-optimized fusion feature corresponding to the symptom and the second to-be-optimized fusion feature corresponding to the symptom into a third fusion layer in the classification model, so as to fuse, through the third fusion layer, the first to-be-optimized fusion feature corresponding to the symptom with the second to-be-optimized fusion feature corresponding to the symptom to obtain a to-be-optimized comprehensive fusion feature corresponding to the symptom; input the to-be-optimized comprehensive fusion feature corresponding to each symptom into a second fully connected layer in the classification model, so as to output, through the second fully connected layer, a to-be-optimized symptom feature corresponding to each symptom.

[0213] Optionally, the training module 609 is specifically configured to input the sample global image and the sample local image into a classification model to be trained, so as to extract features of the sample global image through a first encoding layer in the classification model to obtain first image features to be optimized; extract features of the sample local image through a second encoding layer in the classification model to obtain second image features to be optimized; input the first image features to be optimized into a pooling layer in the classification model to output first pooled features to be optimized through the pooling layer; input the second image features to be optimized into the pooling layer in the classification model to output second pooled features to be optimized through the pooling layer; input the first pooled features to be optimized into a first sub fully-connected layer in the classification model to output respective feature data to be optimized corresponding to the sample global image as respective first feature data to be optimized; and input the second pooled features to be optimized into a second sub fully-connected layer in the classification model to output respective feature data to be optimized corresponding to the sample local image as respective second feature data to be optimized.

[0214] Optionally, the training module 609 is specifically configured to splice the to-be-optimized symptom features corresponding to each symptom to obtain spliced to-be-optimized features; and splice the spliced to-be-optimized features and the first pooled to-be-optimized features to obtain comprehensive to-be-optimized symptom features.

[0215] Optionally, the symptom classification layer is a gated network layer for different diseases. For each symptom, the gated network layer of this symptom is composed of a fully-connected layer of this symptom and a normalization layer.

[0216] This specification also provides a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, it can be used to execute the above Figure 1 provided information recommendation method.

[0217] Based on Figure 1 the information recommendation method shown, an embodiment of this specification also provides Figure 7 a schematic structural diagram of an electronic device shown. As Figure 7 , at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, there may also be other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 described information recommendation method.

[0218] Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0219] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logical function is determined by the user programming the device. Designers can program themselves to "integrate" a digital system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL), and there is not just one type of HDL, but many types, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into an integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0220] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, ASICs, programmable logic controllers, embedded microcontrollers, etc. to achieve the same functions. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0221] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0222] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0223] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0224] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and combinations of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0225] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0227] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0228] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0229] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0230] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0231] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0232] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0233] The various embodiments in this specification are described in a progressive manner. For the parts that are the same or similar among the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments.

[0234] The above are only the embodiments of this specification and are not intended to limit this specification. For those skilled in the art, various modifications and changes can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. An information recommendation method, characterized in that, The method includes: Obtaining an image collected for the lesion site of a user; Inputting the image into a pre-trained classification model to output respective feature data through a first fully-connected layer in the classification model; Inputting the respective feature data into a symptom classification layer in the classification model to determine, for each symptom, the probability that each feature data belongs to that symptom through the symptom classification layer; Inputting the respective feature data and the probability that each feature data belongs to each symptom into a fusion layer in the classification model to determine, for each symptom, a fusion feature corresponding to that symptom based on the probability that each feature data belongs to that symptom and the respective feature data through the fusion layer; Inputting the fusion feature corresponding to each symptom into a second fully-connected layer in the classification model to output a symptom feature corresponding to each symptom through the second fully-connected layer; Inputting the symptom feature corresponding to each symptom into a splicing layer in the classification model to splice the symptom features corresponding to each symptom through the splicing layer to obtain a comprehensive symptom feature; Inputting the comprehensive symptom feature into a disease classification layer in the classification model to identify the disease suffered by the user based on the comprehensive symptom feature through the disease classification layer; Performing information recommendation based on the identified disease suffered by the user; Among them, determining the fusion feature corresponding to each symptom based on the probability that each feature data belongs to that symptom and the respective feature data specifically includes: For each feature data, multiplying the probability that the feature data belongs to that symptom by the feature data to obtain a local feature corresponding to the feature data; Accumulating the local features corresponding to each feature data to obtain the fusion feature corresponding to that symptom.

2. The method according to claim 1, characterized in that Inputting the image into a pre-trained classification model to output respective feature data through a first fully-connected layer in the classification model specifically includes: Inputting the image into a pre-trained classification model to extract features of the image through an encoding layer in the classification model to obtain image features; Inputting the image features into a pooling layer in the classification model to output pooled features through the pooling layer; Inputting the pooled features into a first fully-connected layer in the classification model to output the respective feature data represented by the lesion site through the first fully-connected layer.

3. The method according to claim 1, characterized in that Obtaining an image collected for the lesion site of a user specifically includes: Obtaining a global image collected for the lesion site of a user and obtaining a local image collected for the lesion site of a user; Inputting the image into a pre-trained classification model to output respective feature data through a first fully-connected layer in the classification model specifically includes: Inputting the global image and the local image into a pre-trained classification model to output the respective feature data corresponding to the global image as respective first feature data and output the respective feature data corresponding to the local image as respective second feature data; Input the respective feature data into the symptom classification layer in the classification model, so as to, through the symptom classification layer, determine the probability that each feature data belongs to each symptom for each symptom. Specifically, it includes: Input the respective first feature data into the first symptom classification layer in the classification model, so as to, through the first symptom classification layer, determine the probability that each first feature data belongs to each symptom for each symptom; input the respective second feature data into the second symptom classification layer in the classification model, so as to, through the second symptom classification layer, determine the probability that each second feature data belongs to each symptom for each symptom; Input the respective feature data and the probability that each feature data belongs to each symptom into the fusion layer in the classification model, so as to, through the fusion layer, for each symptom, based on the probability that each feature data belongs to this symptom and the respective feature data, determine the fusion feature corresponding to this symptom. Specifically, it includes: Input the respective first feature data and the probability that each first feature data belongs to each symptom into the first fusion layer in the classification model, so as to, through the first fusion layer, for each symptom, based on the probability that each first feature data belongs to this symptom and the respective first feature data, determine the first fusion feature corresponding to this symptom; Input the respective second feature data and the probability that each second feature data belongs to each symptom into the second fusion layer in the classification model, so as to, through the second fusion layer, for each symptom, based on the probability that each second feature data belongs to this symptom and the respective second feature data, determine the second fusion feature corresponding to this symptom; Input the first fusion feature corresponding to this symptom and the second fusion feature corresponding to this symptom into the third fusion layer in the classification model, so as to, through the third fusion layer, fuse the first fusion feature corresponding to this symptom and the second fusion feature corresponding to this symptom to obtain the comprehensive fusion feature corresponding to this symptom; Input the fusion feature corresponding to each symptom into the second fully connected layer in the classification model, so as to, through the second fully connected layer, output the symptom feature corresponding to each symptom. Specifically, it includes: Input the comprehensive fusion feature corresponding to each symptom into the second fully connected layer in the classification model, so as to, through the second fully connected layer, output the symptom feature corresponding to each symptom.

4. The method according to claim 3, wherein Obtain the local image collected for the lesion site of the user. Specifically, it includes: Input the global image into the pre-trained attention mechanism model, so as to, through the attention mechanism model, determine the pixel set located in the lesion part in the global image as the first set; input the global image into the pre-trained segmentation model, so as to, through the segmentation model, determine the pixel set located in the lesion part in the global image as the second set; wherein, the attention mechanism model and the segmentation model are trained by different training methods; Determine the common pixel points in the first set and the second set; Determine an image region of the minimum circumscribed rectangle containing the common pixel points from the global image as the local image.

5. The method according to claim 3, characterized in that, The first fully connected layer includes: a first sub-fully connected layer and a second sub-fully connected layer; Input the global image and the local image into a pre-trained classification model, so as to output respective feature data corresponding to the global image as respective first feature data and output respective feature data corresponding to the local image as respective second feature data through the first fully connected layer in the classification model. Specifically, it includes: Input the global image and the local image into a pre-trained classification model, so as to extract features of the global image through the first encoding layer in the classification model to obtain a first image feature; and extract features of the local image through the second encoding layer in the classification model to obtain a second image feature; Input the first image feature into the pooling layer in the classification model, so as to output a first pooled feature through the pooling layer; input the second image feature into the pooling layer in the classification model, so as to output a second pooled feature through the pooling layer; Input the first pooled feature into the first sub-fully connected layer in the classification model, so as to output respective feature data corresponding to the global image as respective first feature data through the first sub-fully connected layer; input the second pooled feature into the second sub-fully connected layer in the classification model, so as to output respective feature data corresponding to the local image as respective second feature data through the second sub-fully connected layer.

6. The method according to claim 5, wherein Concatenate the symptom features corresponding to each symptom to obtain a comprehensive symptom feature. Specifically, it includes: Concatenate the symptom features corresponding to each symptom to obtain a concatenated feature; Concatenate the concatenated feature with the first pooled feature to obtain a comprehensive symptom feature.

7. The method according to claim 1, characterized in that, Before acquiring the image collected for the lesion site of the user, the method further includes: Acquire a sample image collected for the lesion site of the patient; Input the sample image into the classification model to be trained, so as to output respective to-be-optimized feature data through the first fully connected layer in the classification model; Input the respective to-be-optimized feature data into the symptom classification layer in the classification model, so as to determine, for each symptom, the probability that each to-be-optimized feature data belongs to this symptom through the symptom classification layer; Input the respective to-be-optimized feature data and the probability that each to-be-optimized feature data belongs to each symptom into the fusion layer in the classification model, so as to determine, for each symptom, a to-be-optimized fusion feature corresponding to this symptom based on the probability that each to-be-optimized feature data belongs to this symptom and the respective to-be-optimized feature data through the fusion layer; Input the to-be-optimized fusion feature corresponding to each symptom into the second fully connected layer in the classification model, so as to output a to-be-optimized symptom feature corresponding to each symptom through the second fully connected layer; Input the to-be-optimized symptom feature corresponding to each symptom into the concatenation layer in the classification model, so as to concatenate the to-be-optimized symptom features corresponding to each symptom through the concatenation layer to obtain a to-be-optimized comprehensive symptom feature; Input the to-be-optimized comprehensive symptom features into the disease classification layer in the classification model, so as to identify the disease type suffered by the patient based on the to-be-optimized comprehensive symptom features through the disease classification layer; Train the classification model with the goal of minimizing the difference between the disease type and the true disease type corresponding to the sample image.

8. The method according to claim 7, characterized in that, Based on the probability that each to-be-optimized feature data belongs to this symptom and the to-be-optimized feature data, determine the to-be-optimized fusion feature corresponding to this symptom, which specifically includes: For each to-be-optimized feature data, multiply the probability that the to-be-optimized feature data belongs to this symptom by the to-be-optimized feature data to obtain the local feature corresponding to the to-be-optimized feature data; Accumulate the local features corresponding to each to-be-optimized feature data to obtain the to-be-optimized fusion feature corresponding to this symptom.

9. The method according to claim 7, wherein Train the classification model with the goal of minimizing the difference between the disease type and the true disease type corresponding to the sample image, which specifically includes: Input the to-be-optimized symptom features corresponding to each symptom into the normalization layer for symptom feature classification, so as to classify the attributes of the to-be-optimized symptom features corresponding to each symptom through the normalization layer and obtain the to-be-optimized attribute category corresponding to each symptom; Determine the first loss according to the difference between the to-be-optimized attribute category corresponding to each symptom and its true attribute category; Determine the second loss according to the difference between the disease type and the true disease type corresponding to the sample image; Determine the comprehensive loss according to the first loss and the second loss; 10. The method according to claim 7, wherein Train the classification model with the goal of minimizing the comprehensive loss. Obtain the sample image collected for the lesion site of the patient, which specifically includes: Obtain the sample global image and the sample local image collected for the lesion site of the patient; Input the sample image into the classification model to be trained, so as to output each to-be-optimized feature data through the first fully connected layer in the classification model, which specifically includes: Input the sample global image and the sample local image into the classification model to be trained, so as to output each to-be-optimized feature data corresponding to the sample global image through the first fully connected layer in the classification model as each first to-be-optimized feature data, and output each to-be-optimized feature data corresponding to the sample local image as each second to-be-optimized feature data; Input the to-be-optimized feature data into the symptom classification layer in the classification model, so as to determine the probability that each to-be-optimized feature data belongs to this symptom for each symptom through the symptom classification layer, which specifically includes: Input the first to-be-optimized feature data into the first symptom classification layer in the classification model, so as to determine the probability that each first to-be-optimized feature data belongs to this symptom for each symptom through the first symptom classification layer; input the second to-be-optimized feature data into the second symptom classification layer in the classification model, so as to determine the probability that each second to-be-optimized feature data belongs to this symptom for each symptom through the second symptom classification layer; Input the to-be-optimized feature data and the probability of each to-be-optimized feature data belonging to each symptom into the fusion layer in the classification model, so that through the fusion layer, for each symptom, based on the probability of each to-be-optimized feature data belonging to this symptom and the to-be-optimized feature data, determine the to-be-optimized fusion feature corresponding to this symptom, specifically including: Input the first to-be-optimized feature data and the probability of each first to-be-optimized feature data belonging to each symptom into the first fusion layer in the classification model, so that through the first fusion layer, for each symptom, based on the probability of each first to-be-optimized feature data belonging to this symptom and the first to-be-optimized feature data, determine the first to-be-optimized fusion feature corresponding to this symptom; Input the second to-be-optimized feature data and the probability of each second to-be-optimized feature data belonging to each symptom into the second fusion layer in the classification model, so that through the second fusion layer, for each symptom, based on the probability of each second to-be-optimized feature data belonging to this symptom and the second to-be-optimized feature data, determine the second to-be-optimized fusion feature corresponding to this symptom; Input the first to-be-optimized fusion feature corresponding to this symptom and the second to-be-optimized fusion feature corresponding to this symptom into the third fusion layer in the classification model, so that through the third fusion layer, fuse the first to-be-optimized fusion feature corresponding to this symptom and the second to-be-optimized fusion feature corresponding to this symptom to obtain the to-be-optimized comprehensive fusion feature corresponding to this symptom; Input the to-be-optimized fusion feature corresponding to each symptom into the second fully connected layer in the classification model, so that through the second fully connected layer, output the to-be-optimized symptom feature corresponding to each symptom, specifically including: Input the to-be-optimized comprehensive fusion feature corresponding to each symptom into the second fully connected layer in the classification model, so that through the second fully connected layer, output the to-be-optimized symptom feature corresponding to each symptom.

11. The method according to claim 10, wherein, Input the sample global image and the sample local image into the classification model to be trained, so that through the first fully connected layer in the classification model, output the to-be-optimized feature data corresponding to the sample global image as the first to-be-optimized feature data, and output the to-be-optimized feature data corresponding to the sample local image as the second to-be-optimized feature data, specifically including: Input the sample global image and the sample local image into the classification model to be trained, so that through the first encoding layer in the classification model, extract features from the sample global image to obtain the first to-be-optimized image feature; through the second encoding layer in the classification model, extract features from the sample local image to obtain the second to-be-optimized image feature; Input the first to-be-optimized image feature into the pooling layer in the classification model, so that through the pooling layer, output the first to-be-optimized pooled feature; input the second to-be-optimized image feature into the pooling layer in the classification model, so that through the pooling layer, output the second to-be-optimized pooled feature; Input the first pooled feature to be optimized into the first sub - fully connected layer in the classification model, so as to output each feature data to be optimized corresponding to the sample global image through the first sub - fully connected layer as each first feature data to be optimized; input the second pooled feature to be optimized into the second sub - fully connected layer in the classification model, so as to output each feature data to be optimized corresponding to the sample local image through the second sub - fully connected layer as each second feature data to be optimized.

12. The method according to claim 11, wherein Concatenate the features to be optimized corresponding to each symptom to obtain the comprehensive symptom feature to be optimized, specifically including: Concatenate the features to be optimized corresponding to each symptom to obtain the concatenated feature to be optimized; Concatenate the concatenated feature to be optimized and the first pooled feature to be optimized to obtain the comprehensive symptom feature to be optimized.

13. The method according to claim 1, characterized in that, The symptom classification layer is a gated network layer for different diseases. For each symptom, the gated network layer of this symptom is composed of the fully connected layer and the normalization layer of this symptom.

14. An information recommendation device, characterized in that, Including: An acquisition module for acquiring an image collected for the lesion site of a user; A first determination module for inputting the image into a pre - trained classification model to output each feature data through the first fully connected layer in the classification model; A second determination module for inputting each feature data into the symptom classification layer in the classification model to determine the probability that each feature data belongs to this symptom for each symptom through the symptom classification layer; A fusion module for inputting each feature data and the probability that each feature data belongs to each symptom into the fusion layer in the classification model to determine the fusion feature corresponding to this symptom for each symptom based on the probability that each feature data belongs to this symptom and each feature data through the fusion layer; A third determination module for inputting the fusion feature corresponding to each symptom into the second fully connected layer in the classification model to output the symptom feature corresponding to each symptom through the second fully connected layer; A concatenation module for inputting the symptom feature corresponding to each symptom into the concatenation layer in the classification model to concatenate the symptom features corresponding to each symptom through the concatenation layer to obtain the comprehensive symptom feature; A disease recognition module for inputting the comprehensive symptom feature into the disease classification layer in the classification model to recognize the disease suffered by the user based on the comprehensive symptom feature through the disease classification layer; Among them, determining the fusion feature corresponding to this symptom based on the probability that each feature data belongs to this symptom and each feature data specifically includes: For each feature data, multiply the probability that this feature data belongs to this symptom by this feature data to obtain the local feature corresponding to this feature data; Accumulate the local features corresponding to each feature data to obtain the fusion feature corresponding to this symptom; An information recommendation module for making information recommendations based on the recognized disease of the user.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 - 13 above.

16. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1-13 above.

Images