Image classification model training method, device, electronic device and storage medium

Abstract

The present disclosure provides a training method, device, electronic device and storage medium for an image classification model, including: obtaining visual features corresponding to a cross-section image and a longitudinal section image corresponding to a target object; mapping the visual features to target nodes included in a first attribute graph based on an inter-graph transformer included in the image classification model, and determining a prediction probability of an attribute feature of each target node; performing feature fusion on each classification node in the classification graph based on an intra-graph transformer included in the image classification model, and determining a classification prediction probability corresponding to each classification node; confirming a first sub-loss based on the prediction probability of the attribute feature of each target node and a label of the attribute feature of each target node; confirming a second sub-loss based on the classification prediction probability corresponding to each classification node and a classification label of the target object; and adjusting parameters of the image classification model based on the first sub-loss and the second sub-loss.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a training method, device, electronic device and storage medium for an image classification model. Background Art

[0002] The task of classifying lesions is usually complicated. It not only requires a comprehensive scan of the lesion site, but also requires judgment based on the attribute characteristics of different cross-sections of the lesion. If the lesions are classified only by relying on ultrasound images, the classification results may not be accurate. Summary of the invention

[0003] The present disclosure provides a training method, device, electronic device and storage medium for an image classification model to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a method for training an image classification model is provided, comprising:

[0005] Obtaining visual features corresponding to the cross-section image and the longitudinal section image corresponding to the target object;

[0006] Based on the inter-graph transformer included in the image classification model, the visual features are mapped to each target node included in the first attribute graph, and the predicted probability of the attribute feature of each target node is determined; wherein the number of target nodes included in the first attribute graph is the same as the number of attribute features included in the target object;

[0007] Mapping each target node included in the first attribute graph to each classification node included in the classification graph based on the inter-graph transformer;

[0008] Based on the intra-graph transformer included in the image classification model, feature fusion is performed on each classification node in the classification graph to determine the classification prediction probability corresponding to each classification node;

[0009] Confirming a first sub-loss based on the predicted probability of the attribute features of each target node and the label of the attribute features of each target node; confirming a second sub-loss based on the classification prediction probability corresponding to each classification node and the classification label of the target object;

[0010] Parameters of the image classification model are adjusted based on the first sub-loss and the second sub-loss.

[0011] In the above solution, the step of obtaining visual features corresponding to the transverse image and the longitudinal image of the target object includes:

[0012] The transverse image and the longitudinal image corresponding to the target object are input into the encoder of the image classification model, and the output of the encoder is confirmed to be the visual feature of the target object.

[0013] In the above scheme, the inter-graph transformer included in the image classification model maps the visual features to each target node included in the first attribute graph, and determines the predicted probability of the attribute features of each target node, including performing the following operations on each target node in the first attribute graph:

[0014] Based on the edge weights and the first projection matrix between each source node in the visual feature and the first target node in the first attribute graph, updating the feature of the first target node;

[0015] Based on the feature of the first target node, determining the predicted probability of the attribute feature of the first target node;

[0016] The source node included in the visual feature includes pixels in the visual feature.

[0017] In the above solution, the step of confirming the predicted probability of the attribute feature of the first target node based on the feature of the first target node includes:

[0018] Based on the feature of the first target node and the second projection matrix, convert the feature of the first target node into a one-dimensional probability;

[0019] Each element in the one-dimensional probability corresponds to the predicted probability of an attribute feature.

[0020] In the above solution, the edge weight between each source node in the visual feature and the first target node in the first attribute graph is determined by:

[0021] Projecting the first target node and each source node in the visual feature into a common feature space respectively, and confirming the first target node projection and the source node projections corresponding to each source node;

[0022] Determine the attention weight between the first target node projection and each source node projection based on the attention mechanism;

[0023] The attention weights are normalized, and the normalized results are confirmed to be edge weights between the first target node and each source node.

[0024] In the above solution, the first attribute graph is confirmed based on a set of target nodes and a set of edges between any two target nodes among the target nodes.

[0025] In the above scheme, mapping each target node included in the first attribute graph to each classification node included in the classification graph based on the inter-graph converter includes performing the following operations on each classification node in the classification graph:

[0026] The inter-graph transformer updates the features of the first classification node in the classification graph based on the edge weights between each target node included in the first attribute graph and the first classification node in the classification graph and the third projection matrix.

[0027] In the above solution, the edge weights between each target node included in the first attribute graph and the first classification node in the classification graph are determined based on the following method:

[0028] Projecting the updated first classification node and each updated classification node in the first attribute graph into a common feature space respectively, and confirming the first classification node projection and the target node projection corresponding to each target node;

[0029] Determine the attention weight between the first classification node projection and each target node projection based on the attention mechanism;

[0030] The attention weights are normalized, and the normalized results are confirmed to be the edge weights between the first classification node and each target node.

[0031] In the above scheme, the in-graph transformer included in the image classification model performs feature fusion on each classification node in the classification graph to determine the classification prediction probability corresponding to each classification node, including performing the following operations on each classification node in the classification graph:

[0032] Based on the edge weights between the first classification node and other classification nodes except the first classification node, and the fourth projection matrix, the classification nodes in the classification graph are merged, and the first classification node is updated based on the fusion result;

[0033] The classification prediction probability of the first classification node is confirmed based on the updated first classification node.

[0034] In the above scheme, the first sub-loss is confirmed based on the predicted probability of the attribute features of each target node and the label of the attribute features of each target node; the second sub-loss is confirmed based on the classification prediction probability corresponding to each classification node and the classification label of the target object, including:

[0035] Confirming the predicted probability of the attribute feature of each target node and the cross entropy loss of the label of the attribute feature of each target node as the first sub-loss;

[0036] Confirming the classification prediction probability corresponding to each classification node and the cross entropy loss of the classification label of the target object as the second sub-loss.

[0037] According to a second aspect of the present disclosure, there is provided an image classification method, which is implemented based on an image classification model trained by the above method, and the method comprises:

[0038] Obtaining a transverse image and a longitudinal image corresponding to the target object to be classified;

[0039] Inputting the transverse image and the longitudinal image into an image converter included in the image classification model, and outputting a second attribute map corresponding to the target object to be classified;

[0040] Inputting the second attribute graph into the graph converter, and outputting a classification graph corresponding to the target object to be classified;

[0041] The classification graph is input into an intra-graph transformer included in the image classification model, and it is confirmed that the output of the intra-graph transformer is a classification result of the target object to be classified.

[0042] According to a third aspect of the present disclosure, a training device for an image classification model is provided, the device comprising:

[0043] A first acquisition unit is used to acquire visual features corresponding to the transverse image and the longitudinal image corresponding to the target object;

[0044] A first mapping unit is used to map the visual features to each target node included in the first attribute graph based on the inter-graph transformer included in the image classification model, and determine the prediction probability of the attribute feature of each target node; wherein the number of target nodes included in the first attribute graph is the same as the number of attribute features included in the target object;

[0045] A second mapping unit, configured to map each target node included in the first attribute graph to each classification node included in the classification graph based on the inter-graph converter;

[0046] A fusion unit, configured to perform feature fusion on each classification node in the classification graph based on an intra-graph transformer included in the image classification model, and determine a classification prediction probability corresponding to each classification node;

[0047] A confirmation unit, configured to confirm a first sub-loss based on the predicted probability of the attribute feature of each target node and the label of the attribute feature of each target node; and confirm a second sub-loss based on the classification prediction probability corresponding to each classification node and the classification label of the target object;

[0048] An adjustment unit, configured to adjust parameters of the image classification model based on the first sub-loss and the second sub-loss.

[0049] According to a fourth aspect of the present disclosure, an image classification device is provided, characterized in that the image classification model trained based on the above method is implemented, and the method includes:

[0050] A second acquisition unit is used to acquire a transverse image and a longitudinal image corresponding to the target object to be classified;

[0051] A first input unit, used for inputting the transverse image and the longitudinal image into an image converter included in the image classification model, and outputting a second attribute map corresponding to the target object to be classified;

[0052] A second input unit, used for inputting the second attribute graph into the graph converter, and outputting a classification graph corresponding to the target object to be classified;

[0053] The classification unit is used to input the classification map into the intra-map transformer included in the image classification model, and confirm that the output of the intra-map transformer is the classification result of the target object to be classified.

[0054] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0055] at least one processor; and

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0058] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present disclosure.

[0059] The training method of the image classification model disclosed in the present invention obtains the visual features corresponding to the cross-section image and the longitudinal section image corresponding to the target object; based on the inter-graph transformer included in the image classification model, maps the visual features to each target node included in the first attribute graph, and determines the prediction probability of the attribute features of each target node; wherein the number of target nodes included in the first attribute graph is the same as the number of attribute features included in the target object; based on the inter-graph transformer, maps each target node included in the first attribute graph to each classification node included in the classification graph; based on the intra-graph transformer included in the image classification model, performs feature fusion on each classification node in the classification graph, and determines the classification prediction probability corresponding to each classification node; based on the prediction probability of the attribute features of each target node and the label of the attribute features of each target node, confirms the first sub-loss; based on the classification prediction probability corresponding to each classification node and the classification label of the target object, confirms the second sub-loss; adjusts the parameters of the image classification model based on the first sub-loss and the second sub-loss; and in combination with different cross-sections of the target object, makes full use of the attribute features closely related to the classification, so that the model can not only output the classification result, but also predict the corresponding attribute features.

[0060] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0062] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0063] Figure 1 An optional flow chart of a training method for an image classification model provided by an embodiment of the present disclosure is shown;

[0064] Figure 2 Another optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown;

[0065] Figure 3 The schematic structure of the image classification model provided by the embodiment of the present disclosure is shown;

[0066] Figure 4 An optional flow chart of the image classification method provided by the embodiment of the present disclosure is shown;

[0067] Figure 5 An optional structural schematic diagram of a training device for an image classification model provided by an embodiment of the present disclosure is shown;

[0068] Figure 6 An optional structural schematic diagram of an image classification device provided by an embodiment of the present disclosure is shown;

[0069] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0070] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0071] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0072] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0074] It should be understood that in the various embodiments of the present disclosure, the size of the serial number of each implementation process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0075] Taking the classification of breast ultrasound images as an example, the BIRADS grade is generally used to classify benign and malignant breast lesions in clinical practice, which is divided into six categories: 2, 3, 4A, 4B, 4C, and 5. Among them, BIRADS 2 is usually a benign lesion that is cystic, solid, regular in shape, and has clear boundaries, and there is no risk of deterioration; BIRADS 3 is a typical benign change in clinical practice, and the risk of deterioration is less than or equal to 2%; BIRADS 4A, 4B, and 4C are usually lesions with one, two, three, or four malignant signs on ultrasound, and the risk of deterioration is 3% to 95%, respectively; and BIRADS 5 has a 95% probability of being malignant. Therefore, it is very important to correctly determine the BIRADS grade for the diagnosis of lesions.

[0076] Usually, the diagnosis of BIRADS of lesions relies on the comprehensive judgment of multiple attribute characteristic indicators of ultrasound images, such as echo type, boundary, shape, calcification, etc.

[0077] In the field of computer vision, most existing methods for BIRADS classification of breast lesions are based on ultrasound images and directly use the CNN convolutional neural network model to extract features for BIRADS classification. It does not focus on the lesion features that are closely related to BIRADS. Secondly, compared with the Transformer structure, the amount of calculation is large and the model calculation efficiency is not high.

[0078] Considering that the classification task of breast ultrasound lesions is relatively complex, clinical judgment of the BIRADS grade of breast lesions requires not only a comprehensive scan of the lesion site, but also a combination of the attribute characteristics of different cross-sections of the lesion. Based on the above analysis, if the lesions are classified only based on ultrasound images, it is not comprehensive and the results obtained are likely to be suboptimal.

[0079] In the related technology, the lesion attribute features that are closely related to the BIRADS classification are not fully utilized; and the CNN model has a larger amount of calculation than the Transformer structure, and the model calculation efficiency is not high.

[0080] In view of the defects existing in the image classification methods in the related art, the present disclosure provides a training method for an image classification model, which encodes the lesion attributes and BIRADS as attribute graphs respectively, so that the features can be dynamically converted between and within graphs between multi-graph structure data, thereby solving at least some or all of the above technical problems. It should be understood by those skilled in the art that the above description of breast ultrasound images is only an example and is not intended to limit the training method of the image classification model and the application scenario of the image classification method provided in the embodiments of the present disclosure.

[0081] Before further describing the embodiments of the present disclosure in detail, the principles of the inter-graph converter and the intra-graph converter involved in the embodiments of the present application are described.

[0082] 1) Image Transformer (Tr1)

[0083] The inter-image converter is applicable to feature transfer between different images. For the sake of convenience, the images are defined as the first image and the second image, and the inter-image converter is applicable to inter-image transfer from the first image to the second image. Optionally, when the first image is a image corresponding to the visual feature, the second image is a first attribute image; when the target object to be classified is the first attribute image, the second image is a classification image.

[0084] The inter-graph transformer can transform the features of the first graph, and pass its corresponding edge weights (edge ​​weights between nodes in the first graph and nodes in the second graph) to the nodes in the second graph, and update the nodes in the second graph. The update formula can be:

[0085]

[0086] Among them, v j is the feature of node j in the second graph, W s is the projection matrix, Represents the edge weight between node i in the first graph and node j in the second graph. After projecting each node in the first graph to the space corresponding to the second graph, the edge weights between the projected point and each node in the second graph are weighted summed, and after nonlinear activation, they are added to the original features of the nodes in the second graph to update the features of the nodes in the second graph.

[0087] Optionally, the attention weight between the first image and the second image can be automatically learned through the attention mechanism information:

[0088]

[0089] represents the attention weight from node i in the first graph to node j in the second graph, W s A Represents the mapping matrix that projects the node features in the first graph into the common feature space, W t A The representation projects the node features in the second graph into a mapping matrix in the common feature space. The Attention operation converts the two projected features (node ​​features in the first graph and node features in the second graph) into probabilities as the association weights of the edges between the two nodes. Attention uses a scaled dot product operation with multi-head attention.

[0090] Then each node in the second graph normalizes the attention weights between all nodes in the first graph, indicating the relative importance of each node in the first graph to any node in the second graph:

[0091]

[0092] Wherein, N is the number of nodes in the second graph (for example, taking the sample image as a breast ultrasound image, in the present disclosure, the number of nodes in the first attribute graph is 29, and the number of nodes in the classification graph is 6). After the normalized attention weight is obtained, the feature of the target node can be updated by formula (1).

[0093] 2) In-graph Transformer (Tr2)

[0094] The purpose of intra-graph feature transfer is to model the correlation between nodes in the same graph and perform feature fusion based on the closeness between nodes. Specifically, it is updated by combining the characteristics of adjacent nodes and itself. The node edge update formula is as follows:

[0095]

[0096] Among them, W tis the projection matrix, which projects the node features into the output space; e n,m Represents the edge weight between the nth node and the mth node in the image. The edge weight can be learned through the attention mechanism, and the process is the same as above. The nodes in the same graph can be updated through formula (4).

[0097] Figure 1 An optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown, and will be explained according to each step.

[0098] Step S101, obtaining visual features corresponding to the transverse image and the longitudinal image of the target object.

[0099] In some embodiments, the target object may be a lesion sample (lesion), and the transverse image and longitudinal image corresponding to the target object may be the transverse image and longitudinal image of the lesion corresponding to the lesion sample.

[0100] In some embodiments, the training device of the image classification model can input the cross-sectional image and the longitudinal section image corresponding to the target object into the encoder of the image classification model, and confirm that the output of the encoder is the visual feature of the target object; wherein the encoder can be resent50.

[0101] Step S102, based on the inter-graph transformer included in the image classification model, the visual features are mapped to the target nodes included in the first attribute graph, and the predicted probability of the attribute features of each target node is determined.

[0102] In some embodiments, the number of target nodes included in the first attribute graph is the same as the number of attribute features potentially contained in the target object; the device can update the features of the first target node based on the edge weights between each source node in the visual feature and the first target node in the first attribute graph and the first projection matrix; based on the features of the first target node, confirm the predicted probability of the attribute features of the first target node; wherein the source nodes included in the visual feature include pixels in the visual feature; the first projection matrix is ​​used to project each original node into the feature space corresponding to the first attribute graph; the number of attribute features potentially contained in the target object can be the total number of attribute features that may exist in the target object. If the possible boundary of the target object is one of clear, still clear, unclear and unclear, then the number of attribute features potentially contained in the target object is 4.

[0103] In a specific implementation, the device may map the visual features to each target node included in the first attribute graph based on the above formula (1), and update the features of the first target node.

[0104] At this time, the target object to be classified is the image corresponding to the visual feature, the second image is the first attribute image, and v in formula (1) i is the feature of the i-th source node in the visual feature, N is the total number of source nodes in the visual feature; W s is the matrix that maps the source node to the feature space corresponding to the first attribute graph (i.e., the first projection matrix), represents the edge weight between the source node i and the target node j in the first attribute graph, v j is the feature of target node j.

[0105] Furthermore, in formula (2), W s Represents the projection matrix that projects the source node into the common space, W t Represents the matrix that projects the target node into the common space.

[0106] In some embodiments, after confirming the characteristics of the first target node, the device may also confirm the predicted probability of the attribute characteristics of the first target node based on the characteristics of the first target node.

[0107] In specific implementation, the device converts the features of the first target node into a one-dimensional probability based on the features of the first target node and the second projection matrix; each element in the one-dimensional probability corresponds to the predicted probability of an attribute feature. Furthermore, the number of elements included in the one-dimensional probability is the same as the number of target nodes included in the first attribute graph; the number of target nodes included in the first attribute graph is the same as the number of attribute features potentially contained in the target object; for example, as shown in Table 1, the number of attribute features potentially contained in the target object is 29, and the number of target nodes included in the first attribute graph is 29. It should be noted that the attribute features potentially contained in the target object are not necessarily the attribute features that the target object will definitely contain. Taking the attribute as the boundary as an example, the corresponding attribute features are clear, clear, unclear and unclear. The real boundary of the target object is one of clear, clear, unclear and unclear, and cannot be fully possessed.

[0108] In some embodiments, the first property graph is confirmed based on a set of target nodes and a set of edges between any two of the target nodes.

[0109] Specifically, the first attribute graph G can be represented as G=(V, E), where: v a ∈R d represents the set of target nodes in the first attribute graph (or the set of features of the target nodes in the first attribute graph), and the feature dimension of each target node is d-dimensional; E = {e ab} a，b＝[1，M]是The set of edges between any two target nodes in the first attribute graph, where M is the total number of target nodes included in the first attribute graph.

[0110] Step S103: Map each target node included in the first property graph to each classification node included in the classification graph based on the inter-graph converter.

[0111] In some embodiments, the device updates the features of the first classification node based on the edge weights between each target node included in the first property graph and the first classification node in the classification graph and the third projection matrix; it should be noted that the first property graph in step S103 is the target property graph formed after each target node is updated.

[0112] In specific implementation, the device projects the updated first classification node and each updated classification node in the first attribute graph into the common feature space respectively, confirms the first classification node projection and the target node projection corresponding to each target node; confirms the attention weight between the first classification node projection and each target node projection based on the attention mechanism; normalizes the attention weight, and confirms that the normalization result is the edge weight between the first classification node and each target node.

[0113] Specifically, the device can update the features of the first classification node based on formula (1) to formula (3). At this time, the first graph is the first attribute graph, and the second graph is the classification graph; N is the number of possible classification results of the target object (2, 3, 4A, 4B, 4C, 5 six categories, that is, the number of possible classification results is 6); v j is the feature of classification node j in the classification graph, represents the edge weight between the target node i in the first attribute graph and the classification node j in the classification graph, W s is the third projection matrix, v i is the feature of target node i in the first attribute graph.

[0114] Step S104, based on the intra-graph transformer included in the image classification model, feature fusion is performed on each classification node in the classification graph to determine the classification prediction probability corresponding to each classification node.

[0115] In some embodiments, the device fuses the classification nodes in the classification graph based on the edge weights between the first classification node and other classification nodes except the first classification node, and the fourth projection matrix, updates the first classification node based on the fusion result; and confirms the classification prediction probability of the first classification node based on the updated first classification node.

[0116] Specifically, the device can determine the edge weights between the first classification node and other classification nodes except the first classification node based on formula (2) to formula (3), fuse the classification nodes in the classification graph based on formula (4), and update the first classification node based on the fusion result.

[0117] At this time, the mapping matrix W in formula (2) s A and W t A The purpose is to map the features of each classification node in the classification graph to the common feature space, v m and v n Respectively represent the characteristics of classification node m and classification node n in the classification graph; e n，m is the edge weight between classification node m and classification node n, W t is the fourth projection matrix; N is the number of possible classification results of the target object.

[0118] Step S105: confirm the first sub-loss based on the predicted probability of the attribute features of each target node and the label of the attribute features of each target node; confirm the second sub-loss based on the classification prediction probability corresponding to each classification node and the classification label of the target object.

[0119] In some embodiments, the device confirms the predicted probability of the attribute features of each target node and the cross entropy loss of the label of the attribute features of each target node as a first sub-loss; confirms the classification prediction probability corresponding to each classification node and the cross entropy loss of the classification label of the target object as a second sub-loss.

[0120] Step S106: Adjust parameters of the image classification model based on the first sub-loss and the second sub-loss.

[0121] In some embodiments, the apparatus performs a weighted sum Loss=λLoss1+(1-λ)Loss2 on the first sub-loss Loss1 and the second sub-loss Loss2, and then back-propagates to optimize the parameters of the image classification model. When the overall loss Loss tends to be stable, it is determined that the image classification model training is completed.

[0122] In this way, the training method of the target detection model provided by the embodiment of the present disclosure fully utilizes the attribute features closely related to image classification. The image classification model can not only output the final image classification result, but also predict the corresponding attribute features; in addition, the image classification model provided by the embodiment of the present disclosure has a small amount of calculation compared with the traditional CNN model. Through the two feature transfer methods of inter-graph Transformer and intra-graph Transformer, the image classification model can better interact between features, thereby improving the prediction accuracy.

[0123] Next, we will take the target object as breast ultrasound image as an example to explain: Figure 2 Another optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown. Figure 3 The schematic structure of the image classification model provided by the embodiment of the present disclosure is shown. Figure 2 and Figure 3 Provide explanation.

[0124] Breast lesions include six attributes: margin, shape, calcifications_in_mass, posterior echo, echo pattern, and vascularity. Each attribute has its corresponding attribute characteristics, as shown in Table 1. During training, ultrasound transverse images and ultrasound longitudinal images of the same lesion are used as image pairs for training; the attribute characteristics of the lesion and BIRADS need to be annotated in advance.

[0125] Table 1. Breast lesion attributes and included features

[0126]

[0127] Referring to Table 1, there are 29 attribute features of breast lesions and 6 classifications. When training the image classification model, the visual features of the input cross-section and longitudinal images of the lesion are extracted, and then the visual features are converted into attribute graphs using the inter-graph Transformer. The attribute graph features are further converted into a higher-level BIRADS image (classification graph) using the inter-graph Transformer for further classification prediction. The node features of the same image (classification graph) are fused using the intra-graph Transformer. The training of the image classification model is optimized with simultaneous supervision of attributes and BIRADS.

[0128] Step S201, obtaining visual features corresponding to the transverse image and the longitudinal image.

[0129] In some embodiments, the cross-section image and the longitudinal section image are an image pair of the same lesion (the same target object), and the training device of the image classification model uses resent50 as an encoder to extract the visual features X of the lesion for the input cross-section and longitudinal section images of the lesion. Next, the visual features are converted into the first attribute map using the inter-image Transformer.

[0130] Step S202: Map the visual features to a first attribute map and confirm the first sub-loss.

[0131] In some embodiments, before model training, the first attribute graph is initialized. Let G = (V, E) represent the first attribute graph. v a ∈R d represents the set of target nodes in the first attribute graph (or the set of features of the target nodes in the first attribute graph), and the feature dimension of each target node is d-dimensional; E = {e ab} a，b＝[1，M] It is the set of edges between any two target nodes in the first attribute graph, M is the total number of target nodes included in the first attribute graph, and when training with breast ultrasound images, M is the total number of attribute features, that is, 29. Each target node in the first attribute graph corresponds to a feature in Table 1.

[0132] The visual feature (Visualfeature) X obtained in step S201 is passed to the first attribute graph (refer to step S102) using the inter-graph Transformer method. It should be noted that when the Visualfeature is input, each pixel in it is regarded as a node in the graph structure:

[0133] f n =Tr1(X)

[0134] p n =sigmoid(W n f n )

[0135] f n is the feature (potential feature) of the nth target node, n∈[1,M], W n is the linear projection matrix (second projection matrix), which converts the features of the target node into one-dimensional probability. The one-dimensional probability of the nth target node can be expressed as p n =(p n1 , p n2 , ..., p nN )∈(0,1),p n Each element in represents the predicted probability of a certain attribute feature corresponding to the nth target node, and the binary label y of the nth target node n ∈{0, 1} to calculate the cross entropy loss Loss1.

[0136] Step S203: Map the first attribute graph to the classification graph.

[0137] Attribute features are very important reference information for determining the classification of lesions. Therefore, the features of each target node in the first attribute graph are mapped to the classification graph using an inter-graph transformer (see step S103). The number of classification nodes included in the classification graph is 6, corresponding to 2, 3, 4A, 4B, 4C, and 5 of BIRADS.

[0138] f c =Tr1(f n )

[0139] f c is the feature of the c-th classification node in the classification graph, c∈[1,6].

[0140] Step S204, fusing the features of each classification node in the classification graph to determine the second sub-loss.

[0141] In some embodiments, the graph converter updates the features of the first classification node based on the edge weights between each target node and the first classification node in the classification graph and the third projection matrix included in the first attribute graph; specifically, the updated first classification node and each updated classification node in the first attribute graph are projected into a common feature space, and the first classification node projection and the target node projection corresponding to each target node are confirmed; the attention weight between the first classification node projection and each target node projection is confirmed based on the attention mechanism; the attention weight is normalized, and the normalization result is confirmed to be the edge weight between the first classification node and each target node. Based on the edge weights between the first classification node and other classification nodes except the first classification node, and the fourth projection matrix, each classification node in the classification graph is fused, and the first classification node is updated based on the fusion result; the classification prediction probability of the first classification node is confirmed based on the updated first classification node.

[0142] In some embodiments, the device performs feature fusion on the correlation of each classification node in the classification graph based on the in-graph Transformer, and then performs classification by linear projection to obtain the predicted probability of each classification node:

[0143] f c =Tr2(f c )

[0144] p = sigmoid(W c f c )

[0145] Similarly, W c is a linear projection (the third projection matrix), p represents the predicted probability of each classification node, and the cross entropy loss Loss2 is calculated with the classification label of the target object.

[0146] Step S205: Adjust parameters of the image classification model based on the first sub-loss and the second sub-loss.

[0147] In some embodiments, the apparatus performs a weighted sum Loss=λLoss1+(1-λ)Loss2 on the first sub-loss Loss1 and the second sub-loss Loss2, and then back-propagates to optimize the parameters of the image classification model. When the overall loss Loss tends to be stable, it is determined that the image classification model training is completed.

[0148] In this way, the training method of the target detection model provided by the embodiment of the present disclosure fully utilizes the attribute features closely related to image classification. The image classification model can not only output the final image classification result, but also predict the corresponding attribute features; in addition, the image classification model provided by the embodiment of the present disclosure has a small amount of calculation compared with the traditional CNN model. Through the two feature transfer methods of inter-graph Transformer and intra-graph Transformer, the image classification model can better interact between features, thereby improving the prediction accuracy.

[0149] Figure 4 An optional flow chart of the image classification method provided by the embodiment of the present disclosure is shown, and will be explained according to each step.

[0150] Step S301, obtaining a transverse image and a longitudinal image corresponding to the target object to be classified.

[0151] The transverse image and the longitudinal image are the transverse image and the longitudinal image corresponding to the same lesion (target object to be classified).

[0152] Step S302: input the transverse image and the longitudinal image into the image converter included in the image classification model, and output a second attribute map corresponding to the target object to be classified.

[0153] In some embodiments, the image classification device inputs the cross-section image and the longitudinal section image into an encoder included in the image classification model to obtain visual features corresponding to the cross-section image and the longitudinal section image; inputs the visual features into the image converter, and outputs the second attribute graph. The number of target nodes included in the second attribute graph is the same as the total number of possible (potential) attribute features of the target object to be classified.

[0154] Among them, the possible attribute features of the target object to be classified include 29 attribute features in Table 1.

[0155] Step S303: input the second attribute graph into the graph converter, and output a classification graph corresponding to the target object to be classified.

[0156] In some embodiments, the image classification device inputs the second attribute graph into the inter-graph transformer, and the inter-graph transformer maps each target node in the second attribute graph to a classification graph, and updates the features of each classification node included in the classification graph.

[0157] Step S304: input the classification graph into an intra-graph transformer included in the image classification model, and confirm that the output of the intra-graph transformer is a classification result of the target object to be classified.

[0158] In some embodiments, the features of each classification node are fused based on the intra-graph transformer, and the output of the intra-graph transformer is confirmed to be the classification result of the target object to be classified.

[0159] In this way, the image classification method provided by the embodiment of the present disclosure fully utilizes the attribute features closely related to image classification, so that both the final image classification result can be output and the corresponding attribute features can be predicted.

[0160] Figure 5 An optional structural schematic diagram of a training device for an image classification model provided by an embodiment of the present disclosure is shown and will be described according to each part.

[0161] In some embodiments, the training device 500 for the image classification model includes a first acquisition unit 501 , a first mapping unit 502 , a second mapping unit 503 , a fusion unit 504 , a confirmation unit 505 and an adjustment unit 506 .

[0162] The first acquisition unit 501 is used to acquire visual features corresponding to the transverse image and the longitudinal image corresponding to the target object;

[0163] The first mapping unit 502 is used to map the visual features to each target node included in the first attribute graph based on the inter-graph transformer included in the image classification model, and determine the prediction probability of the attribute feature of each target node; wherein the number of target nodes included in the first attribute graph is the same as the number of attribute features included in the target object;

[0164] The second mapping unit 503 is used to map each target node included in the first property graph to each classification node included in the classification graph based on the inter-graph converter;

[0165] The fusion unit 504 is used to perform feature fusion on each classification node in the classification graph based on the intra-graph transformer included in the image classification model, and determine the classification prediction probability corresponding to each classification node;

[0166] The confirmation unit 505 is used to confirm the first sub-loss based on the predicted probability of the attribute feature of each target node and the label of the attribute feature of each target node; confirm the second sub-loss based on the classification prediction probability corresponding to each classification node and the classification label of the target object;

[0167] The adjustment unit 506 is used to adjust the parameters of the image classification model based on the first sub-loss and the second sub-loss.

[0168] The first acquisition unit 501 is specifically used to input the transverse image and the longitudinal image corresponding to the target object into the encoder of the image classification model, and confirm that the output of the encoder is the visual feature of the target object.

[0169] The first mapping unit 502 is specifically configured to perform the following operations on each target node in the first property graph:

[0170] Based on the edge weights and the first projection matrix between each source node in the visual feature and the first target node in the first attribute graph, updating the feature of the first target node;

[0171] Based on the feature of the first target node, determining the predicted probability of the attribute feature of the first target node;

[0172] The source node included in the visual feature includes pixels in the visual feature.

[0173] The first mapping unit 502 is specifically configured to convert the feature of the first target node into a one-dimensional probability based on the feature of the first target node and the second projection matrix;

[0174] Each element in the one-dimensional probability corresponds to the predicted probability of an attribute feature.

[0175] The first mapping unit 502 is specifically used to project the first target node and each source node in the visual feature into a common feature space respectively, and confirm the first target node projection and the source node projection corresponding to each source node;

[0176] Determine the attention weight between the first target node projection and each source node projection based on the attention mechanism;

[0177] The attention weights are normalized, and the normalized results are confirmed to be edge weights between the first target node and each source node.

[0178] In some embodiments, the first property graph is confirmed based on a set of target nodes and a set of edges between any two of the target nodes.

[0179] The second mapping unit 503 is specifically configured to perform the following operations on each classification node in the classification graph:

[0180] The inter-graph transformer updates the features of the first classification node in the classification graph based on the edge weights between each target node included in the first attribute graph and the first classification node in the classification graph and the third projection matrix.

[0181] The second mapping unit 503 is specifically used to project the updated first classification node and each updated classification node in the first attribute graph into a common feature space, and confirm the first classification node projection and the target node projection corresponding to each target node;

[0182] Determine the attention weight between the first classification node projection and each target node projection based on the attention mechanism;

[0183] The attention weights are normalized, and the normalized results are confirmed to be the edge weights between the first classification node and each target node.

[0184] The fusion unit 504 is specifically configured to perform the following operations on each classification node in the classification graph:

[0185] Based on the edge weights between the first classification node and other classification nodes except the first classification node, and the fourth projection matrix, the classification nodes in the classification graph are merged, and the first classification node is updated based on the fusion result;

[0186] The classification prediction probability of the first classification node is confirmed based on the updated first classification node.

[0187] The confirmation unit 505 is specifically used to confirm that the predicted probability of the attribute feature of each target node and the cross entropy loss of the label of the attribute feature of each target node are the first sub-loss;

[0188] Confirming the classification prediction probability corresponding to each classification node and the cross entropy loss of the classification label of the target object as the second sub-loss.

[0189] Figure 6 An optional structural schematic diagram of an image classification device provided in an embodiment of the present disclosure is shown, and will be described according to each part.

[0190] In some embodiments, the image classification device 600 includes a second acquisition unit 601 , a first input unit 602 , a second input unit 603 , and a classification unit 604 .

[0191] The second acquisition unit 601 is used to acquire a transverse image and a longitudinal image corresponding to the target object to be classified;

[0192] The first input unit 602 is used to input the transverse image and the longitudinal image into the image converter included in the image classification model, and output a second attribute map corresponding to the target object to be classified;

[0193] The second input unit 603 is used to input the second attribute graph into the graph converter, and output the classification graph corresponding to the target object to be classified;

[0194] The classification unit 604 is used to input the classification map into the intra-map transformer included in the image classification model, and confirm that the output of the intra-map transformer is the classification result of the target object to be classified.

[0195] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0196] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0197] like Figure 7 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0198] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0199] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the training method of the image classification model and / or the image classification method. For example, in some embodiments, the training method of the image classification model and / or the image classification method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training method of the image classification model and / or the image classification method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the image classification model training method and / or the image classification method in any other appropriate manner (for example, by means of firmware).

[0200] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0202] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0204] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0205] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0206] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0207] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0208] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A training method for an image classification model, characterized in that: The method comprises: Obtaining visual features corresponding to the cross-section image and the longitudinal section image corresponding to the target object; Based on the inter-graph transformer included in the image classification model, the visual features are mapped to each target node included in the first attribute graph, and the predicted probability of the attribute feature of each target node is determined; wherein the number of target nodes included in the first attribute graph is the same as the number of attribute features included in the target object; Mapping each target node included in the first attribute graph to each classification node included in the classification graph based on the inter-graph converter; Based on the intra-graph transformer included in the image classification model, feature fusion is performed on each classification node in the classification graph to determine the classification prediction probability corresponding to each classification node; Confirming a first sub-loss based on the predicted probability of the attribute feature of each target node and the label of the attribute feature of each target node; confirming a second sub-loss based on the classification prediction probability corresponding to each classification node and the classification label of the target object; Parameters of the image classification model are adjusted based on the first sub-loss and the second sub-loss.

2. The method according to claim 1, characterized in that The step of obtaining visual features corresponding to the transverse image and the longitudinal image of the target object includes: The transverse image and the longitudinal image corresponding to the target object are input into the encoder of the image classification model, and it is confirmed that the output of the encoder is the visual feature corresponding to the target object.

3. The method according to claim 1, characterized in that The inter-graph transformer included in the image classification model maps the visual features to each target node included in the first attribute graph, and determines the predicted probability of the attribute features of each target node, including performing the following operations on each target node in the first attribute graph: Based on the edge weights and the first projection matrix between each source node in the visual feature and the first target node in the first attribute graph, updating the feature of the first target node; Based on the feature of the first target node, determining the predicted probability of the attribute feature of the first target node; The source node included in the visual feature includes pixels in the visual feature.

4. The method according to claim 3, characterized in that The step of confirming the predicted probability of the attribute feature of the first target node based on the feature of the first target node includes: Based on the feature of the first target node and the second projection matrix, convert the feature of the first target node into a one-dimensional probability; Each element in the one-dimensional probability corresponds to the predicted probability of an attribute feature.

5. The method according to claim 3, characterized in that: The edge weights between each source node in the visual feature and the first target node in the first attribute graph are determined in the following manner: Projecting the first target node and each source node in the visual feature into a common feature space respectively, and confirming the first target node projection and the source node projections corresponding to each source node; Determine the attention weight between the first target node projection and each source node projection based on the attention mechanism; The attention weights are normalized, and the normalized results are confirmed to be edge weights between the first target node and each source node.

6. The method according to claim 1, characterized in that The first attribute graph is confirmed based on a set of target nodes and a set of edges between any two of the target nodes.

7. The method according to claim 1, characterized in that The mapping of each target node included in the first attribute graph to each classification node included in the classification graph based on the inter-graph converter includes performing the following operations on each classification node in the classification graph: The inter-graph transformer updates the features of the first classification node in the classification graph based on the edge weights between each target node included in the first attribute graph and the first classification node in the classification graph and the third projection matrix.

8. The method according to claim 7, characterized in that The edge weights between each target node included in the first attribute graph and the first classification node in the classification graph are determined based on the following method: Projecting the updated first classification node and each updated classification node in the first attribute graph into a common feature space respectively, and confirming the first classification node projection and the target node projection corresponding to each target node; Determine the attention weight between the first classification node projection and each target node projection based on the attention mechanism; The attention weights are normalized, and the normalized results are confirmed to be the edge weights between the first classification node and each target node.

9. The method according to claim 1 or 7, characterized in that: The image classification model includes an in-graph transformer, which performs feature fusion on each classification node in the classification graph to determine the classification prediction probability corresponding to each classification node, including performing the following operations on each classification node in the classification graph: Based on the edge weights between the first classification node and other classification nodes except the first classification node, and the fourth projection matrix, the classification nodes in the classification graph are merged, and the first classification node is updated based on the fusion result; The classification prediction probability of the first classification node is confirmed based on the updated first classification node.

10. The method according to claim 1, characterized in that The first sub-loss is confirmed based on the predicted probability of the attribute features of each target node and the label of the attribute features of each target node; the second sub-loss is confirmed based on the classification prediction probability corresponding to each classification node and the classification label of the target object, including: Confirming the predicted probability of the attribute feature of each target node and the cross entropy loss of the label of the attribute feature of each target node as the first sub-loss; Confirming the classification prediction probability corresponding to each classification node and the cross entropy loss of the classification label of the target object as the second sub-loss.

11. An image classification method, characterized in that: The image classification model trained by the method according to any one of claims 1 to 10 is implemented, and the method comprises: Obtaining a transverse image and a longitudinal image corresponding to the target object to be classified; Inputting the transverse image and the longitudinal image into an image converter included in the image classification model, and outputting a second attribute map corresponding to the target object to be classified; Inputting the second attribute graph into the graph converter, and outputting a classification graph corresponding to the target object to be classified; The classification graph is input into an intra-graph transformer included in the image classification model, and it is confirmed that the output of the intra-graph transformer is a classification result of the target object to be classified.

12. A training device for an image classification model, characterized in that: The device comprises: A first acquisition unit is used to acquire visual features corresponding to the transverse image and the longitudinal image corresponding to the target object; A first mapping unit is used to map the visual features to each target node included in the first attribute graph based on the inter-graph transformer included in the image classification model, and determine the prediction probability of the attribute feature of each target node; wherein the number of target nodes included in the first attribute graph is the same as the number of attribute features included in the target object; A second mapping unit, configured to map each target node included in the first attribute graph to each classification node included in the classification graph based on the inter-graph converter; A fusion unit, configured to perform feature fusion on each classification node in the classification graph based on an intra-graph transformer included in the image classification model, and determine a classification prediction probability corresponding to each classification node; A confirmation unit, configured to confirm a first sub-loss based on the predicted probability of the attribute feature of each target node and the label of the attribute feature of each target node; and confirm a second sub-loss based on the classification prediction probability corresponding to each classification node and the classification label of the target object; An adjustment unit, configured to adjust parameters of the image classification model based on the first sub-loss and the second sub-loss.

13. An image classification device, characterized in that: The image classification model trained by the method according to any one of claims 1 to 10 is implemented, and the method comprises: A second acquisition unit is used to acquire a transverse image and a longitudinal image corresponding to the target object to be classified; A first input unit, used for inputting the transverse image and the longitudinal image into an image converter included in the image classification model, and outputting a second attribute map corresponding to the target object to be classified; A second input unit, used for inputting the second attribute graph into the graph converter, and outputting a classification graph corresponding to the target object to be classified; The classification unit is used to input the classification map into the intra-map transformer included in the image classification model, and confirm that the output of the intra-map transformer is the classification result of the target object to be classified.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 10; Alternatively, the method of claim 11 can be performed.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10; Alternatively, the method of claim 11 can be performed.

Images