Image classification network model training method, image classification method and related equipment

Abstract

The present application provides an image classification network model training method, an image classification method and related equipment. The image classification network model training method includes: obtaining a training image and an external knowledge base, the external knowledge base includes the true category label of the training image; encoding the external knowledge base to obtain a category distance matrix; inputting the training image and its true category label and the category distance matrix into the image classification network model to obtain the predicted category probability distribution of the training image, wherein the predicted category probability distribution includes the probability of the gap between the predicted category label output by the image classification network model and the true category label; calculating the target loss function using the depth distance between the true category label and the predicted category label in the category distance matrix and the predicted category probability distribution; training the network model based on the target loss function. The present application is used to obtain an image classification network model that takes into account both improving image classification accuracy and enhancing the interpretability of prediction results.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for an image classification network model, an image classification method, and related equipment. Background Art

[0002] Image classification is one of the most fundamental problems in image processing technology. Existing techniques primarily employ deep neural network image classification methods. Specifically, the image to be classified and its class label are input into a deep neural network model to train the model. However, the predicted class labels for the images to be classified, as output by the deep neural network model using this method, are subject to error and lack interpretability. Summary of the Invention

[0003] The present application provides a training method for an image classification network model, an image classification method, and related equipment.

[0004] To solve the above technical problems, the present application provides a method for training an image classification network model, the method comprising:

[0005] Obtaining a training image and an external knowledge base, wherein the external knowledge base includes true category labels of the training image;

[0006] Performing encoding processing on the external knowledge base to obtain a category distance matrix;

[0007] Inputting the training image, its true category label, and the category distance matrix into the image classification network model to obtain a predicted category probability distribution of the training image, wherein the predicted category probability distribution includes a difference probability between the predicted category label output by the image classification network model and the true category label;

[0008] Calculating a target loss function using the depth distance between the true class label and the predicted class label in the class distance matrix and the predicted class probability distribution;

[0009] The image classification network model is trained based on the target loss function.

[0010] To solve the above technical problems, the present application provides an image classification method, which includes:

[0011] Get the image to be classified;

[0012] Inputting the image to be classified into an image classification network model to obtain a category label of the image to be classified, wherein the image classification network model is an image classification network model trained using the above method;

[0013] The category labels of the images to be classified are evaluated to obtain an interpretability score.

[0014] To solve the above technical problems, the present application provides a terminal device, the device comprising a memory and a processor coupled to the memory;

[0015] The memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned image classification network model training method and / or the above-mentioned image classification method.

[0016] To solve the above technical problems, the present application also provides a computer storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the training method of the image classification network model and / or the above-mentioned image classification method.

[0017] The beneficial effects of the present application are: obtaining training images and an external knowledge base, the external knowledge base including the true category labels of the training images; encoding the external knowledge base to obtain a category distance matrix; inputting the training images and their true category labels and the category distance matrix into the image classification network model to obtain the predicted category probability distribution of the training images, wherein the predicted category probability distribution includes the probability of the gap between the predicted category labels output by the image classification network model and the true category labels; calculating the target loss function using the depth distance between the true category labels and the predicted category labels in the category distance matrix and the predicted category probability distribution; training the network model based on the target loss function. The present application references an external knowledge base to constrain the predicted category probability distribution output by the image classification network model, thereby improving the accuracy of image classification and enhancing the interpretability of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0019] Figure 1 This is a flow chart of an embodiment of a training method for an image classification network model provided by the present application;

[0020] Figure 2 This is a simplified schematic diagram of the external knowledge base in the training method of the image classification network model provided in this application;

[0021] Figure 3 yes Figure 1Schematic diagram of the flow of S102 in an embodiment of the training method of the image classification network model shown;

[0022] Figure 4 yes Figure 1 Schematic diagram of the flow chart of S104 in an embodiment of the training method of the image classification network model shown;

[0023] Figure 5 This is a flowchart of an embodiment of the image classification method provided by the present application;

[0024] Figure 6 This is a schematic structural diagram of an embodiment of a terminal device provided by this application;

[0025] Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided by this application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] This application proposes a training method for an image classification network model. For details, please refer to Figure 1 , Figure 1 The figure is a flow chart of an embodiment of a training method for an image classification network model provided by the present application. The training method for an image classification network model in this embodiment can be applied to an image classification device. The image classification device of the present application can be a server, a mobile device, or a system in which a server and a mobile device cooperate with each other. Accordingly, the various parts of the mobile device, such as the various units, subunits, modules, and submodules, can all be set in the server, all be set in the mobile device, or be set separately in the server and the mobile device.

[0028] Furthermore, the server described above may be either hardware or software. When the server is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it may be implemented as multiple software programs or software modules, such as software or software modules for providing a distributed server, or as a single software program or software module, without further limitation.

[0029] The training method of the image classification network model of this embodiment specifically includes the following steps:

[0030] S101: Acquire training images and an external knowledge base, where the external knowledge base includes true category labels of the training images.

[0031] In the embodiments of the present disclosure, considering that the prior art method of training an image classification network model solely using training images and their true category labels may result in errors in the predicted category labels output by the image classification network model and may be unexplainable, the image classification device of the present application references an external knowledge base to constrain the prediction results of the image classification network model.

[0032] See Figure 2 , Figure 2 This is a simplified schematic diagram of the external knowledge base in the training method of the image classification network model provided by this application. As can be seen from the figure, the external knowledge base is a tree structure composed of multiple category labels. Each node in the tree structure represents a category label, and the category labels between nodes that are closer in position in the tree structure are more similar. In order to use the external knowledge base to constrain the predicted category probability distribution output by the image classification network model, the external knowledge base of this embodiment should include all category labels that the image classification network model can distinguish. Furthermore, in the process of training the image classification network model, the external knowledge base at least includes the true category labels of the training images.

[0033] Furthermore, considering the multi-category nature of training images, a single external knowledge base may not include the true category labels of all training images. To this end, the training method of the image classification network model of this embodiment can supplement the missing category labels in the single external knowledge base by manually extracting category labels from an additional knowledge base.

[0034] Considering the impact of the number of training images on the prediction results output by the image classification network model, the number of training images required in this embodiment should be as large as possible. In a specific embodiment, the number of training images is at least 1000.

[0035] It should be noted that before the image classification device of this embodiment uses the training image to train the image classification network model, the pixel size of the training image should be unified, for example, uniformly scaled to 256x256, so as to facilitate the use of training images of the same pixel size to train the image classification network model.

[0036] S102: Encode the external knowledge base to obtain a category distance matrix.

[0037] You can continue to read Figure 2In order to use the deep distance between category labels in the external knowledge base to constrain the predicted category probability distribution output by the network model, this embodiment requires encoding the external knowledge base to obtain a category distance matrix. The category distance matrix includes the deep distance between any two category labels in the external knowledge base, that is, the semantic distance.

[0038] Optionally, this embodiment may adopt Figure 3 The embodiment implements S102, specifically including S201 to S203:

[0039] S201: Obtain any two category labels in the external knowledge base.

[0040] To facilitate obtaining the depth distance between the true class label and the predicted class label from the class distance matrix, the image classification device of this embodiment can pre-acquire the class distance matrix including the depth distance between the true class label and the predicted class label. Specifically, to obtain the class distance matrix, the image classification device first needs to obtain any two class labels from the external knowledge base.

[0041] S202: Obtain a common category label between any two category labels.

[0042] Furthermore, the image classification device obtains the common category label between any two category labels in the external knowledge base, that is, the most common ancestor, wherein the most common ancestor is the ancestor of one category label and the other category label of any two category labels, and the ancestor depth is as large as possible.

[0043] S203: Calculate the depth distance between any two category labels based on the common category label to obtain a category distance matrix including the depth distance between any two category labels.

[0044] The image classification apparatus of this embodiment uses the common category labels to calculate the depth distance between any two category labels to obtain a category distance matrix including the depth distance between any two category labels.

[0045] Specifically, the image classification device obtains the public category label, the depth of one category label of the two category labels, and the depth of the other category label respectively; calculates the sum of the depth of one category label and the depth of the other category label; and uses the ratio of the depth of the public category label to the sum of the depth of the one category label and the depth of the other category label to calculate the depth distance between any two category labels.

[0046] Among them, the Wup (Wu-Palmer) semantic similarity is used to calculate the depth distance between any two category labels in the external knowledge base. The specific calculation formula of the depth distance is as follows:

[0047]

[0048] Among them, c1 and c2 are two category labels in the external knowledge base, depth(c1) is the depth of category label c1, depth(c2) is the depth of category label c2, lcs(c1, c2) is the common category label of category label c1 and category label c2, depth(lcs(c1, c2)) is the depth of the common category label, and d(c1, c2) is the depth distance between category label c1 and category label c2.

[0049] Furthermore, the image classification device of this embodiment locates the label position of the public category label in the external knowledge base, uses the label position in the external knowledge base to obtain the number of layers of the public category label, and thus determines the depth of the public category label. In this specific embodiment, the depth of category labels c1 and c2 is obtained by referring to the depth of public category labels, and will not be repeated here.

[0050] S103: Input the training image and its true category label and category distance matrix into the image classification network model to obtain the predicted category probability distribution of the training image.

[0051] The image classification apparatus of this embodiment inputs a training image, its true class label, and a class distance matrix into an image classification network model to obtain a predicted class probability distribution for the training image. The predicted class probability distribution includes the probability of the difference between the predicted class label output by the image classification network model and the true class label.

[0052] S104: Calculate the target loss function using the depth distance between the true category label and the predicted category label in the category distance matrix and the predicted category probability distribution.

[0053] Because the loss function used in the existing image classification network model training method has a loss function value when the predicted category label of the training image is consistent with the true category label. When the predicted category label of the training image is inconsistent with the true category label, the loss function value is 0. Therefore, the existing loss function ignores the impact of the inconsistency between the predicted category label of the training image and the true category label on the image classification network model training, resulting in the predicted category probability distribution output by the image classification network model being inconsistent with common sense. To solve the above problem, the image classification network model training method of this embodiment takes into account the impact of the inconsistency between the predicted category label of the training image and the true category label on the image classification network model by expanding the loss function. Specifically, the image classification device of this embodiment uses the depth distance between the true category label and the predicted category label in the category distance matrix and the predicted category probability distribution to calculate the target loss function.

[0054] Optionally, this embodiment may adopt Figure 4The embodiment implements S104, specifically including S301 to S304:

[0055] S301: Obtain the depth distance between the true category label and the predicted category label in the category distance matrix.

[0056] Since this embodiment expands the loss function in the existing image classification network model training method, the target loss function in the image classification network model of this embodiment includes a first loss function and a second loss function. The first loss function and the second loss function respectively characterize different aspects of the network model. Specifically, the first loss function characterizes the loss between the predicted category probability distribution output by the image classification network model and the preset category probability distribution when there is consistency between the predicted category and the true category of the training image. The second loss function characterizes the loss between the predicted category probability distribution output by the image classification network model and the depth distance, i.e., the semantic distance, between the predicted category and the true category of the training image when there is inconsistency between the predicted category and the true category of the training image.

[0057] S302: Calculate a first loss function using the predicted category probability distribution and the depth distance.

[0058] The image classification device calculates a first loss function using the predicted category probability distribution and the depth distance.

[0059] Specifically, the first loss function satisfies the following formula:

[0060]

[0061] Among them, L CE (k) is the first loss function, k is the training image, l is the predicted class label of the training image, and p(k, l) is the predicted class probability output by the image classification network model. I(k, l) is the indicator function, which is 1 when l matches the true class label of k and 0 when l does not match the true class label of k.

[0062] It should be noted that, in a specific embodiment, the first loss function may be a cross entropy loss function.

[0063] S303: Calculate a second loss function using the predicted category probability distribution and the true category label.

[0064] Based on the first loss function in S302, when the true class labels of l and k are inconsistent, the indicator function is 0, causing the first loss function to be 0. The prediction results output by the image classification network model ignore the inconsistency between the true class labels and the predicted class labels of the training images. To address this issue, the image classification device of this embodiment expands the first loss function to constrain the prediction results for class labels other than the true class labels. Specifically, the image classification device calculates the second loss function using the predicted class probability distribution and the true class labels.

[0065] Specifically, the second loss function satisfies the following formula:

[0066]

[0067] Among them, L Sem (k) is the second loss function, t k is the true category label corresponding to the training image, d(t k , l) is the category distance matrix including the depth distance between the predicted category label and the true category label, and p(k, l) is the predicted category probability output by the image classification network model.

[0068] S304: Calculate a target loss function based on the first loss function and the second loss function.

[0069] The image classification device calculates the target loss function using the first loss function and the second loss function.

[0070] Specifically, the objective loss function satisfies the following formula:

[0071]

[0072] Where L(k) is the target loss function and α is the weight coefficient, which is used to balance the first loss function and the second loss function to optimize the training of the image classification network model.

[0073] In a specific embodiment, the image classification device may use a grid search method to determine the weight coefficient α.

[0074] S105: Train the image classification network model based on the target loss function.

[0075] The image classification device of this embodiment trains the image classification network model with a target loss function. Specifically, the image classification device of this embodiment can train the target loss function using a gradient descent technique.

[0076] In the above scheme, the image classification device references an external knowledge base to constrain the predicted category probability distribution output by the image classification network model, thereby improving the accuracy of image classification and enhancing the interpretability of the prediction results; the target loss function is calculated by using the predicted category probability distribution and the depth distance, which expands the existing loss function and avoids the situation where the existing loss function ignores the inconsistency between the predicted category label and the true category label of the training image, resulting in the predicted category probability distribution output by the image classification network model being inconsistent with common sense.

[0077] See Figure 5 , Figure 5 This is a flow chart of an embodiment of the image classification method provided by the present application. The image classification method of this embodiment can be applied to the image classification network model trained in the above-mentioned image classification network model training method, thereby improving both the accuracy of image classification and the interpretability of prediction results. The image classification method provided by the present application is described below using a server used in the image classification method as an example. The image classification method of this embodiment specifically includes the following steps:

[0078] S401: Obtain an image to be classified.

[0079] The acquisition of the image to be classified in this embodiment is similar to the acquisition of the training image in S101 of the above embodiment, and will not be repeated here.

[0080] S402: Input the image to be classified into the image classification network model to obtain the category label of the image to be classified.

[0081] The image classification device of this embodiment inputs the image to be classified into the image classification network model to obtain the category label of the image to be classified.

[0082] S403: Evaluate the category labels of the images to be classified to obtain an interpretability score.

[0083] To balance improving image classification accuracy and enhancing the interpretability of prediction results, this embodiment evaluates the category labels of the image to be classified, as output by the image classification network model, to obtain an interpretability score. Specifically, this embodiment inputs the image to be classified into the image classification network model, obtains the category label ranking value and category probability distribution, and calculates the interpretability score using the category label ranking value, which includes the first category label ranking value and the second category label ranking value.

[0084] The first category label ranking value is the probability ranking value of the difference between the category label of the image to be classified in the category probability distribution and the true category label of the image to be classified. The second category label ranking value is the depth distance ranking value between the category label of the image to be classified in the category distance matrix and the true category label of the image to be classified.

[0085] Among them, the interpretability evaluation satisfies the following formula:

[0086]

[0087] Among them, r k , l It is the category probability ranking value of the category probability that the image k to be classified belongs to category label l in the category probability distribution. The image k to be classified belongs to the category label The category probability of the category probability ranking value in the category probability distribution. k , l is the category label l and the real category label t of the image to be classified k The ranking value of the depth distance between the two in the category distance matrix, that is, s k , l for The corresponding sort value. For category labels and the true category label t of the image to be classified k The depth distance between the ranking values ​​in the category distance matrix, that is, for The corresponding sort value.

[0088] In this embodiment, an image to be classified is obtained, the image to be classified is input into an image classification network model, a category label of the image to be classified is obtained, the category label of the image to be classified is evaluated, and an interpretability score is obtained, thereby achieving both improving the accuracy of image classification and enhancing the interpretability of prediction results.

[0089] In order to implement the image classification network model training method and / or image classification method of the above embodiment, this application proposes a terminal device, please refer to Figure 6 , Figure 6 It is a structural diagram of an embodiment of a terminal device provided by this application.

[0090] The terminal device 600 includes a memory 61 and a processor 62 , wherein the memory 61 and the processor 62 are coupled.

[0091] The memory 61 is used to store program data, and the processor 62 is used to execute the program data to implement the image classification network model training method and / or image classification method of the above embodiment.

[0092] In this embodiment, the processor 62 may also be referred to as a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip having signal processing capabilities. The processor 62 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor 62 may be any conventional processor.

[0093] This application also provides a computer storage medium 700, such as Figure 7 As shown, the computer storage medium 700 is used to store program data 71. When the program data 71 is executed by the processor, it is used to implement the image classification network model training method and / or image classification method as described in the method embodiment of the present application.

[0094] The methods involved in the image classification network model training method and / or image classification method embodiments of the present application exist in the form of software functional units when implemented and are sold or used as independent products, and can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0095] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A training method for an image classification network model, characterized in that: The method comprises: Obtaining a training image and an external knowledge base, wherein the external knowledge base includes true category labels of the training image, and the external knowledge base is a tree structure composed of multiple category labels, wherein each node in the tree structure represents a category label; Encoding the external knowledge base to obtain a category distance matrix, wherein the category distance matrix represents the semantic distance between any two category labels in the external knowledge base; Inputting the training image, its true category label, and the category distance matrix into the image classification network model to obtain a predicted category probability distribution of the training image, wherein the predicted category probability distribution includes a difference probability between the predicted category label output by the image classification network model and the true category label; Calculating a target loss function using the depth distance between the true class label and the predicted class label in the class distance matrix and the predicted class probability distribution; The image classification network model is trained based on the target loss function.

2. The training method according to claim 1, characterized in that The step of encoding the external knowledge base to obtain a category distance matrix includes: Obtain any two category labels in the external knowledge base; Obtaining a common category label between any two category labels; The depth distance between the arbitrary two category labels is calculated based on the common category label to obtain a category distance matrix including the depth distance between the arbitrary two category labels.

3. The training method according to claim 2, characterized in that The step of calculating the depth distance between any two category labels based on the common category label includes: respectively obtaining the depth of the common category label, one category label of the arbitrary two category labels, and the other category label; Calculating the sum of the depth of the one category label and the depth of the other category label; The depth distance between the arbitrary two category labels is calculated using the ratio of the depth of the common category label to the sum.

4. The training method according to claim 3, characterized in that The external knowledge base is a tree structure, and the step of obtaining the depth of the public category label includes: Positioning the public category label at a label position in the tree structure; Acquire the number of layers between the node corresponding to the label position and the root node in the tree structure based on the label position; The depth of the common category label is determined using the number of layers.

5. The training method according to claim 1, wherein: The target loss function includes a first loss function and a second loss function, and the step of calculating the target loss function using the depth distance between the true category label and the predicted category label in the category distance matrix and the predicted category probability distribution includes: Obtaining the depth distance between the true category label and the predicted category label in the category distance matrix; Calculating the first loss function using the predicted category probability distribution and the depth distance; Calculating a second loss function using the predicted category probability distribution and the true category label; The target loss function is calculated based on the first loss function and the second loss function.

6. The training method according to claim 1, characterized in that: The step of training the image classification network model based on the target loss function includes: The objective loss function is trained using gradient descent technology.

7. An image classification method, characterized in that: The image classification method comprises: Get the image to be classified; Inputting the image to be classified into an image classification network model to obtain a category label of the image to be classified, wherein the image classification network model is an image classification network model trained using the method described in any one of claims 1 to 6 above; The category labels of the images to be classified are evaluated to obtain an interpretability score.

8. The method according to claim 7, characterized in that The step of evaluating the category label of the image to be classified to obtain an interpretability score includes: Inputting the image to be classified into the image classification network model to obtain the category label ranking value and category probability distribution; The category label ranking value includes a first category label ranking value and a second category label ranking value, the first category label ranking value is a probability ranking value of the gap between the category label of the image to be classified in the category probability distribution and the true category label of the image to be classified, and the second category label ranking value is a depth distance ranking value between the category label of the image to be classified and the true category label of the image to be classified in the category distance matrix; An explainability score is calculated using the first ranking value and the second ranking value.

9. A terminal device, characterized in that: The device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the training method of the image classification network model according to any one of claims 1 to 6 and / or the image classification method according to any one of claims 7 to 8.

10. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by the processor, it is used to implement the training method of the image classification network model according to any one of claims 1 to 6 and / or the image classification method according to any one of claims 7 to 8.

Images