Image multi-label classification method and device, computer device and storage medium

Abstract

Embodiments of the present application relate to the technical field of image processing, and disclose a kind of image multi-label classification method, device, computer equipment and storage medium, the method comprises: obtaining n initial sample images and m initial labels;According to n the initial sample image and m the initial label, construct sample association network, label association network and sample label network;Random walk is carried out to the sample association network, the label association network and the sample label network, and the confidence between each initial sample image and each initial label is obtained;According to the confidence between each initial sample image and each initial label, determine the multi-label classification result of each initial sample image. By implementing the embodiments of the present application, the problems of ignoring the dependency between labels in existing image multi-label classification, reducing the precision and accuracy of label classification, etc. can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image multi-label classification method, apparatus, computer device, and storage medium. Background Technology

[0002] With the continuous generation of massive amounts of media data due to technological advancements, including images, videos, titles, and text, the ability to perform object detection and tracking within this data using scientific tools has become a crucial problem in computer vision and pattern recognition. Solving these problems often involves image classification to improve the accuracy and speed of subsequent object detection and tracking. Therefore, effective image classification is of paramount importance.

[0003] Currently, traditional image classification is often divided into single-label classification and multi-label classification based on the number of categories. In real life, an image often contains rich semantic information, such as target scenes, behaviors, attributes, and objects (e.g., landscapes and animals). A single category or label is insufficient to fully describe the semantic information contained in the image, thus single-label classification is prone to causing information loss in image processing. Multi-label classification, on the other hand, can assign multiple labels to an image to fully express its content, thereby avoiding information loss to some extent.

[0004] In existing technologies, image multi-label classification typically involves converting the multi-label dataset into a single-label dataset and then using a single-label classification method to classify the converted dataset. However, this approach ignores the dependencies between the labels, thus reducing the accuracy and precision of the label classification. Summary of the Invention

[0005] This application discloses an image multi-label classification method, apparatus, computer device, and storage medium, which can solve the problems of ignoring the dependencies between labels and reducing the accuracy and precision of label classification in existing image multi-label classification.

[0006] The first aspect of this application discloses an image multi-label classification method, the method comprising:

[0007] Obtain n initial sample images and m initial labels, where m and n are both positive integers;

[0008] Based on n initial sample images and m initial labels, a sample association network, a label association network, and a sample label network are constructed. The sample association network is used to reflect the similarity between the initial sample images, the label association network is used to reflect the association between the initial labels, and the sample label network is used to reflect the association between the initial sample images and the initial labels.

[0009] A random walk is performed on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label;

[0010] The multi-label classification result of each initial sample image is determined based on the confidence level between each initial sample image and each initial label.

[0011] In some embodiments, constructing a sample association network, a label association network, and a sample label network based on n initial sample images and m initial labels includes:

[0012] Calculate the sample similarity between any two initial sample images from the n initial sample images;

[0013] The sample association network is constructed based on the sample similarity between any two initial sample images;

[0014] Based on the m initial labels and the known labels corresponding to each initial sample image, construct the label association network and the sample label network, where the known labels are the labels among the n initial labels.

[0015] In some embodiments, calculating the sample similarity between any two of the n initial sample images includes:

[0016] A comprehensive similarity calculation is performed on any two of the initial sample images to obtain the comprehensive similarity between the two initial sample images;

[0017] Gaussian kernel similarity is calculated for any two initial sample images to obtain the Gaussian kernel similarity between the two initial sample images;

[0018] The sample similarity between any two initial sample images is calculated based on the comprehensive similarity and the Gaussian kernel similarity between them.

[0019] In some embodiments, the step of calculating the comprehensive similarity between any two initial sample images to obtain the comprehensive similarity between the two initial sample images includes:

[0020] The cosine similarity between any two initial sample images is calculated.

[0021] The cosine similarity between any two initial sample images is processed by a preset logic to obtain the processed similarity between any two initial sample images.

[0022] Clustering is performed on the processing similarity between any two initial sample images to obtain the comprehensive similarity between the two initial sample images.

[0023] In some embodiments, performing clustering processing on the processing similarity between any two initial sample images to obtain the comprehensive similarity between the two initial sample images includes:

[0024] Determine the clustering coefficient, which reflects the clustering coefficient of the same image cluster when any two initial sample images are located in the same image cluster;

[0025] The processing similarity between any two initial sample images is corrected based on the clustering coefficient to obtain the comprehensive similarity between any two initial sample images.

[0026] In some embodiments, calculating the Gaussian kernel similarity between any two initial sample images to obtain the Gaussian kernel similarity between the two initial sample images includes:

[0027] The Gaussian kernel bandwidth of each initial sample image is determined based on the association between each initial sample image and each initial label;

[0028] The similarity between any two initial sample images is calculated based on the Gaussian kernel bandwidth to obtain the Gaussian kernel similarity between the two initial sample images.

[0029] In some embodiments, constructing the label association network based on the m initial labels and the known labels corresponding to each initial sample image includes:

[0030] Based on the known labels corresponding to each initial sample image, determine the number of sample associations between any two initial labels among the m initial labels, where the number of sample associations is the number of times any two initial labels are simultaneously associated with the same sample image;

[0031] The weight of the newly created edge between any two initial labels is determined based on the number of sample associations between any two initial labels.

[0032] The label association network is constructed based on the weights of the newly created edges between any two initial labels. The label association network includes the weights of the newly created edges between any two initial labels.

[0033] In some embodiments, before performing a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label, the method further includes:

[0034] A similarity coefficient is calculated between each initial sample image and each initial label to obtain the similarity coefficient between each initial sample image and each initial label;

[0035] Based on the similarity coefficient between each initial sample image and each initial label, calculate the walk step size for each initial sample image and the walk step size for each initial label;

[0036] The step of performing a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label includes:

[0037] According to the walk step size of each initial sample image, a random walk is performed on the label association network and the sample label network to obtain a first prediction value between each initial sample image and each initial label;

[0038] According to the walk step size of each initial label, a random walk is performed on the sample association network and the sample label network to obtain a second predicted value between each initial sample image and each initial label;

[0039] The confidence level between each initial sample image and each initial label is calculated based on the first predicted value and the second predicted value between each initial sample image and each initial label.

[0040] The second aspect of this application discloses an image multi-label classification device, comprising:

[0041] The acquisition module is used to acquire n initial sample images and m initial labels, where m and n are both positive integers;

[0042] A construction module is used to construct a sample association network, a label association network, and a sample label network based on n initial sample images and m initial labels. The sample association network is used to reflect the similarity between the initial sample images, the label association network is used to reflect the association between the initial labels, and the sample label network is used to reflect the association between the initial sample images and the initial labels.

[0043] The processing module is used to perform random walks on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label;

[0044] The determination module is used to determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label.

[0045] For any content not described in the embodiments of this application, please refer to the relevant descriptions in the aforementioned first aspect of the disclosed embodiments, which will not be repeated here.

[0046] A third aspect of this application discloses a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect or any optional embodiment of the first aspect.

[0047] For any content not described in the embodiments of this application, please refer to the relevant descriptions in the aforementioned first aspect of the disclosed embodiments, which will not be repeated here.

[0048] The fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect or any optional embodiment of the first aspect.

[0049] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0050] In this embodiment, a computer device acquires n initial sample images and m initial labels. Based on the n initial sample images and m initial labels, a sample association network, a label association network, and a sample label network are constructed. The sample association network reflects the similarity between the initial sample images, the label association network reflects the association between the initial labels, and the sample label network reflects the association between the initial sample images and the initial labels. A random walk is performed on the sample association network, the label association network, and the sample label network to obtain the confidence score between each initial sample image and each initial label. Based on the confidence score between each initial sample image and each initial label, the multi-label classification result of each initial sample image is determined. Therefore, implementing this embodiment comprehensively considers the association between sample images, the association between labels, and the association between sample images and labels to achieve multi-label classification of sample images. This solves the problems of existing multi-label image classification methods that ignore the dependencies between labels themselves and reduce the accuracy and precision of label classification, thus improving the accuracy and precision of multi-label image classification. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating an image multi-label classification method in one embodiment of this application;

[0053] Figure 2 This is a flowchart illustrating an image multi-label classification method in another embodiment of this application;

[0054] Figure 3 This is a flowchart illustrating an image multi-label classification method in another embodiment of this application;

[0055] Figure 4 This is a flowchart illustrating the comprehensive similarity calculation steps in one embodiment of this application;

[0056] Figure 5 This is a flowchart illustrating the tag association network construction steps in one embodiment of this application;

[0057] Figure 6 This is a flowchart illustrating the confidence calculation steps in one embodiment of this application;

[0058] Figure 7This is a schematic diagram of the structure of an image multi-label classification device in one embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of a computer device in one embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0062] First, let me introduce some of the technical terms or concepts involved in this application.

[0063] Image classification refers to grouping images with similar features together based on the similarity of their features.

[0064] Multi-label classification refers to a sample image having more than one label, meaning that a sample image corresponds to multiple labels.

[0065] A random walk is the process of randomly selecting a neighboring node as the next hop node. Specifically, starting from a given vertex, a graph is traversed. At any given vertex, the walk iterates to a neighboring node with a probability of 1-α, and then randomly jumps to another vertex in the graph with a probability of α. After each walk, a probability distribution is obtained, which describes the probability of each vertex being visited. This probability distribution is used as the input for the next walk, and the process is iterated repeatedly. Under certain conditions, this probability distribution will converge in a region. After convergence, a stationary probability distribution is obtained.

[0066] Restarted random walk is an improvement on regular random walk. Specifically, starting from a vertex in the graph, each step presents two choices: move to a neighboring node with a probability of 1-α, or return to the starting node with a restart probability of α. After iteration until convergence, the resulting probability distribution can be seen as a probability distribution influenced by the starting node. Restarted random walk can capture multifaceted relationships between two nodes and capture the overall structural information of the graph.

[0067] Secondly, relevant embodiments of the image multi-label classification method involved in this application are introduced. Please refer to... Figure 1 This is a schematic flowchart of a possible image multi-label classification method disclosed in an embodiment of this application. Figure 1 The method shown is applied to computer devices, including but not limited to servers, personal computers, laptops, tablets, smartphones, or other devices with communication capabilities. The method includes the following implementation steps:

[0068] S101. Obtain n initial sample images and m initial labels, where m and n are both positive integers.

[0069] This application can obtain a set of sample images and a set of labels. The set of sample images includes n initial sample images, which can be represented as P = {p1, p2, ..., p...}. n The tag set includes m initial tags, which can be represented as L = {l1, l2, ..., l}. m}. Where p i ∈P represents the i-th initial sample image, where i is a positive integer and 1≤i≤n. p i Specifically, it can be represented as a d-dimensional initial feature vector (p1, p2, ..., p...). d ).in Represents the set of real numbers. The i-th initial sample image p i There is at least one known label, which is any one of the n initial labels. The i-th initial sample image p i The set of tags to which it belongs is represented as L i L i ∈L.

[0070] S102. Based on the n initial sample images and m initial labels, construct a sample association network, a label association network, and a sample label network. The sample association network is used to reflect the similarity between the initial sample images, the label association network is used to reflect the association between the initial labels, and the sample label network is used to reflect the association between the initial sample images and the initial labels.

[0071] This application can construct a sample association network, a label association network, and a sample label network based on n initial sample images, m initial labels, and the known labels of each of the n initial sample images.

[0072] S103. Perform a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label.

[0073] This application can use a restarted random walk algorithm to perform random walks in the sample association network, the label association network, and the sample label network, thereby obtaining the confidence level between each initial sample image and each initial label, wherein the confidence level is used to reflect the probability that the initial sample image is classified as the initial label.

[0074] S104. Determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label.

[0075] This application can determine the multi-label classification result of each initial sample image based on the confidence between each initial sample image and each initial label. Specifically, for example, when the confidence between any initial sample image and any initial label is greater than a preset threshold, it can be determined that the label classification of any initial sample image includes any initial label, that is, the multi-label classification result of any initial sample label includes any initial label.

[0076] By implementing the embodiments of this application, a computer device acquires n initial sample images and m initial labels; based on the n initial sample images and m initial labels, a sample association network, a label association network, and a sample label network are constructed. The sample association network reflects the similarity between the initial sample images, the label association network reflects the association between the initial labels, and the sample label network reflects the association between the initial sample images and the initial labels; a random walk is performed on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label; based on the confidence level between each initial sample image and each initial label, the multi-label classification result of each initial sample image is determined. It is evident that implementing the embodiments of this application comprehensively considers the association between sample images, the association between labels, and the association between sample images and labels to achieve multi-label classification of sample images. This solves the problems of existing multi-label image classification methods that ignore the dependencies between labels themselves and reduce the accuracy and precision of label classification, thereby improving the accuracy and precision of multi-label image classification.

[0077] Please see Figure 2 This is a flowchart illustrating another image multi-label classification method disclosed in an embodiment of this application. Figure 2 The method shown includes the following implementation steps:

[0078] S201. Obtain n initial sample images and m initial labels, where m and n are both positive integers.

[0079] S202. Calculate the sample similarity of the n initial sample images to obtain the sample similarity between any two initial sample images among the n initial sample images.

[0080] This application can perform similarity calculations on any two initial sample images from n initial sample images to obtain the sample similarity between the two initial sample images. The specific implementation method of the similarity calculation is not limited in this application; for example, cosine similarity calculation, Euclidean distance calculation, or other similarity calculation methods may be used. The sample similarity involved in this application refers to the similarity between two corresponding initial sample images.

[0081] S203. Construct the sample association network based on the sample similarity between any two initial sample images.

[0082] The sample association network described in this application is constructed based on the sample similarity between any two initial sample images from the n initial sample images. The sample association network includes several nodes, each representing an initial sample image; these nodes can be called sample nodes. The sample association network may further include connections between nodes that have an association relationship, where the weight of the connection can be the sample similarity between the nodes. In other words, the sample association network includes the sample similarity between any two initial sample images.

[0083] S204. Based on the m initial labels and the known labels corresponding to each initial sample image, construct the label association network and the sample label network, wherein the known labels are the labels among the n initial labels.

[0084] The label association network described in this application includes several nodes, each representing an initial label, which can be referred to as a label node. The label association network may further include: connections between nodes with association relationships and weights on these connections. The weight on each connection may be the number of sample images simultaneously associated with the nodes at both ends of the connection on the same sample image, also referred to as the sample association quantity. In other words, the label association network includes the sample association quantity between any two initial labels.

[0085] The sample label network includes several nodes and connections between nodes that are related. The nodes in the network can represent initial sample images or initial labels. The connection refers to the connection between an initial sample image and an initial label. The weights on the connection can be used to reflect / indicate the relationship between the initial sample image and the initial label on the connection.

[0086] S205. Perform a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label.

[0087] S206. Determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label.

[0088] For any content not described in the embodiments of this application, please refer to the foregoing. Figure 1 The relevant descriptions in the embodiments will not be repeated here.

[0089] By implementing the embodiments of this application, a computer device acquires n initial sample images and m initial labels; calculates sample similarity among the n initial sample images to obtain the sample similarity between any two initial sample images; constructs the sample association network based on the sample similarity between the two initial sample images; constructs the label association network and the sample label network based on the m initial labels and the known labels corresponding to each initial sample image; performs a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label; and determines the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label. As can be seen, by implementing the embodiments of this application, a sample association network can be created based on the sample similarity between initial sample images. Then, a label association network and a sample label network can be created by combining the association relationship between initial labels and the association relationship between initial sample images and initial labels. This allows for comprehensive consideration of the association relationships between samples, the association relationships between labels, and the association relationships between samples and labels to achieve multi-label classification. This solves the problems in existing image multi-label classification, such as ignoring the dependencies between labels themselves and reducing the accuracy and precision of label classification, thereby improving the accuracy and precision of image multi-label classification.

[0090] Please see Figure 3 This is a flowchart illustrating another image multi-label classification method disclosed in an embodiment of this application. Figure 3 The method shown includes the following implementation steps:

[0091] S301. Obtain n initial sample images and m initial labels, where m and n are both positive integers.

[0092] S302. Perform a comprehensive similarity calculation on any two of the n initial sample images to obtain the comprehensive similarity between the two initial sample images.

[0093] The specific implementation method of the comprehensive similarity calculation described in this application is not limited, and may include, for example, cosine similarity calculation, Hammanton distance calculation, standardized Euclidean distance calculation, etc. For example, please refer to... Figure 4 This is a schematic diagram of a comprehensive similarity calculation process disclosed in an embodiment of this application. For example... Figure 4 The process shown includes the following implementation steps:

[0094] S401. Calculate the cosine similarity between any two initial sample images to obtain the cosine similarity between the two initial sample images.

[0095] This application can utilize cosine similarity to measure the similarity between any two initial sample images, i.e., sample-to-sample similarity. Taking the arbitrary two initial sample images, including the first sample image p... i Second sample image p j For example, in practical applications, each sample image can be represented by a feature vector, i.e., p. i Specifically, p can represent the initial feature vector of the first sample image. j Specifically, it can represent the initial feature vector of the second sample image. The first sample image p i and the second sample image p j The cosine similarity between them can be calculated using the following formula (1):

[0096]

[0097] Where, p i p represents the initial feature vector of the first sample image. j This represents the initial feature vector of the second sample image. p i transpose, ||p i || represents vector p i The length of ||p j || represents vector p j The length.

[0098] S402. Perform preset logic processing on the cosine similarity between any two initial sample images to obtain the processed similarity between any two initial sample images.

[0099] The preset logic processing described in this application can be customized by the system, such as processing invalid data to 0 and amplifying valid data. Understandably, since a very low similarity provides limited information for image classification prediction, this application can convert the similarity to 0 for more accurate and efficient computation. Specifically, this application can convert the cosine similarity... The dataset is divided into multiple level ranges, for example, 10 level ranges ((0,0.1],(0.1,0.2],...,(0.9,1]), and the average similarity between initial sample images with the same label within each sub-range (i.e., level range) is calculated. Specifically, in the actual calculation process... It can be an n×n matrix, and this application can handle the matrix. The average similarity of data within each sub-range is calculated to obtain the initial sub-range. The average similarity.

[0100] Furthermore, this application may be randomly shuffled. The data in the middle, and the data after shuffling. The process of reclassifying the grading ranges and calculating the average similarity is the same as before the data was shuffled, thus obtaining the average similarity of each sub-range after shuffling. When the average similarity after random shuffling within the same sub-range is less than the average similarity before shuffling (i.e., the initial sub-range), the average similarity is determined. If the average similarity is calculated, then the cosine similarity within that subrange can be determined as invalid data; otherwise, it can be determined as valid data.

[0101] Furthermore, this application may be applicable to Pre-defined logic processing is performed to reduce invalid similarity data to near zero and amplify valid similarity data. Specifically, this is also done using any two initial sample images, including the first sample image p. i Second sample image p j For example, this application can use the following formula (2) to... Perform preset logic processing to obtain p i and p j Processing similarity between

[0102]

[0103] Where c and d are both adjustments The penalty parameter is c, which is the upper limit of the first valid subrange, and d = log(999).

[0104] S403. Perform clustering processing on the processing similarity between any two initial sample images to obtain the comprehensive similarity between any two initial sample images.

[0105] This application first determines the clustering coefficient, which reflects the clustering coefficient of the same image cluster when any two initial sample images are located in the same image cluster. Then, the processing similarity between any two initial sample images is corrected based on the clustering coefficient to obtain the comprehensive similarity between the two initial sample images.

[0106] In practical implementation, when two initial sample images belong to the same cluster, they will be more similar. Based on this assumption, this application can construct a weighted sample image sharing network. This network uses n initial sample images as nodes, and the weights on the edges reflect the number of common labels among the initial sample images. This application can apply clustering algorithms (such as ClusterONE clustering) to this network to identify potential sample image clusters. When two initial sample images belong to the same cluster, similarity can be processed... Add a clustering coefficient, which can be expressed as shown in the following formula (3):

[0107]

[0108] Among them, W in (C) represents the total weight of the edges in the cluster, W bound (C) represents the total weight of the edges connecting a node in the cluster to other nodes, P(C) represents the penalty term, and C represents the cluster (specifically, all nodes in the cluster).

[0109] When this application determines that any two initial sample images are located in the same cluster, the comprehensive similarity of the two initial sample images is calculated by including the first sample image p. i Second sample image p j For example, p i and p j The overall similarity between them can be expressed by the following formula (4):

[0110]

[0111] in, p represents the first sample image i Second sample image p j The overall similarity between them p represents the first sample image i Second sample image pj The processing similarity between them, f(C) represents the clustering coefficient.

[0112] S303. Perform Gaussian kernel similarity calculation on any two initial sample images to obtain the Gaussian kernel similarity between any two initial sample images.

[0113] This application can determine the Gaussian kernel bandwidth of each initial sample image based on the association between each initial sample image and each initial label. Then, it calculates the similarity between any two initial sample images based on the Gaussian kernel bandwidth to obtain the Gaussian kernel similarity between the two initial sample images.

[0114] In its implementation, this application assumes that similar initial sample images will typically be associated with or rejected by the same initial label. Based on this assumption, this application can introduce Gaussian kernel similarity to further enhance the calculation of sample similarity. Specifically, taking any two initial sample images including the first sample image p... i Second sample image p j For example, the first sample image p i The Gaussian kernel bandwidth can be expressed by the following formula (5):

[0115]

[0116] Where, γ p p represents the initial sample image i Gaussian kernel bandwidth, IP(p) represents the average number of known labels for each initial sample image. i || represents the initial sample image p i A binary vector that has or does not have a known association with the m initial labels.

[0117] The first sample image p i and the second sample image p j The Gaussian kernel similarity between them can be expressed by the following formula (6):

[0118]

[0119] in, p i and p j Gaussian similarity between them, γ p p represents the initial sample image i Gaussian kernel bandwidth.

[0120] S304. Calculate the sample similarity between any two initial sample images based on the comprehensive similarity and the Gaussian kernel similarity between any two initial sample images.

[0121] This application can calculate the sample similarity between any two initial sample images by performing preset rules such as averaging on the comprehensive similarity between any two initial sample images and the Gaussian similarity between any two initial sample images.

[0122] The first sample image p is included in any two of the initial sample images. i Second sample image p j For example, p i and p j Sample similarity S between p It can be represented by the following formula (6):

[0123]

[0124] S305. Construct the label association network based on the m initial labels and the known labels corresponding to each initial sample image.

[0125] The method of constructing the tag association network described in this application is not limited; for example, please combine it with other methods. Figure 5 This is a schematic diagram illustrating the process of constructing a tag association network as disclosed in an embodiment of this application. Figure 5 The method shown includes the following implementation steps:

[0126] S501. Based on the known labels corresponding to each initial sample image, determine the number of sample associations between any two initial labels among the m initial labels, wherein the number of sample associations is the number of times any two initial labels are simultaneously associated with the same sample image.

[0127] S502. Determine the weight of the newly created edge between any two initial labels based on the number of sample associations between any two initial labels.

[0128] S503. Construct the label association network according to the weight of the newly created edge between any two initial labels, wherein the label association network includes the weight of the newly created edge between any two initial labels.

[0129] This application requires constructing the label association network G = (V, E), where V represents the set of nodes consisting of all initial labels, and E represents the set of edges, specifically the connection between any two initial labels (nodes) that have an association relationship. i and tag l jSimultaneously associated with the same sample image p k At that time, in these two initial tags (such as tag l) i and tag l j Add a new edge between () and () with weight w. ij , which is defined as the number of sample images simultaneously associated with both initial labels, or simply the sample association number. The w ij As shown in formula (7) below:

[0130]

[0131] Among them, |p k | indicates the label i and tag l j The number of samples simultaneously associated with the same initial sample label, i.e., the number of sample associations. L k p represents the initial sample image k The corresponding set of known labels.

[0132] The adjacency matrix S of the label association network l Given an m×m symmetric matrix, the data in the i-th row and j-th column of this adjacency matrix can be represented by the following formula (8):

[0133]

[0134] Among them, w ij This represents the weight of the newly created edge between label i and label j. This represents the total weight of the j-th column.

[0135] S306. Construct the sample label network based on the known labels corresponding to each initial sample image.

[0136] This application can construct a sample label network based on the known labels corresponding to each initial sample image. Specifically, since the initial sample image (e.g., initial sample image p) i ) and known labels (e.g., initial label l) j If a known association exists, then the sample label network W pl (p i ,l j ) = 1. Otherwise, when the initial sample image (e.g., the initial sample image p) is 1. i ) and initial label (e.g., initial label l) j If no known association exists, then the sample label network W pl (p i ,l j ) = 0. In other words, the sample label network includes the association between each initial sample image and each initial label, such as W pl (pi ,l j ).

[0137] S307. Perform a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label.

[0138] This application employs a restarted random walk algorithm to perform random walks on the three networks—the sample association network, the label association network, and the sample label network—to obtain the confidence level between each initial sample image and each initial label. Specifically, this application can first construct a heterogeneous association network based on the sample association network, the label association network, and the sample label network. For example, the sample association network, the label association network, and the sample label network can be merged into a single heterogeneous association network, which can simultaneously include the three networks and their corresponding information. Then, a restarted random walk algorithm is used to perform a double random walk on the heterogeneous association network (which can also be understood as the three networks) to obtain the confidence level between each initial sample image and each initial label.

[0139] Please see Figure 6 This is a schematic diagram of a confidence level calculation process disclosed in an embodiment of this application. Figure 6 The method shown includes the following implementation steps:

[0140] S601. Calculate the similarity coefficient between each initial sample image and each initial label to obtain the similarity coefficient between each initial sample image and each initial label.

[0141] Since the walk step size of a node typically depends on its influence within the network, this application proposes a similarity coefficient (Jaccard coefficient) to quantify the personalized random walk step size of a node. This application can calculate the similarity coefficient between each initial sample image and each initial label, thus obtaining the similarity coefficient between each initial sample image and each initial label. Taking the initial sample image p... i and initial label l j For example, the initial sample image p i and the initial label l j The similarity coefficient between them can be expressed by the following formula (9):

[0142]

[0143] Where, N p (p i ) represents the initial sample image p i The neighboring nodes, N l (lj ) indicates label l j The neighboring nodes.

[0144] In practical applications, due to the sample label association network W pl In, N p (p i )∩N l (l j Since the initial sample image p is an empty set, this application requires improvements to the Jaccard coefficients described above. i and the initial label l j The similarity coefficient between them can be expressed by the following formula (10):

[0145]

[0146] in, Indicates the relationship with the initial label l j The set of sample images associated with neighboring nodes. JI(p i ,l j ) represents the initial sample image p i and the initial label l j The similarity coefficient between them can also be called the degree of influence.

[0147] S602. Calculate the walking step size of each initial sample image and the walking step size of each initial label based on the similarity coefficient between each initial sample image and each initial label.

[0148] This application assumes that nodes with high influence after quantization are more likely to interact with other nodes during random walks and should therefore have larger walk steps. Based on this assumption, this application can quantify the personalized walk step size of each node (specifically, the initial sample image or the initial label) according to the similarity coefficient between the initial sample image and the initial label. Using the initial sample image p... i and initial label l j Taking these two nodes as examples, the initial sample image p i and the initial label l j The respective walking stride lengths are shown in formula (11):

[0149]

[0150] Among them, O p (p i ) represents the initial sample image p i walking stride length, O l (l j ) indicates the initial label l j The stride length.

[0151] S603. Perform a random walk on the label association network and the sample label network according to the walk step size of each initial sample image to obtain a first prediction value between each initial sample image and each initial label.

[0152] S604. Perform a random walk on the sample association network and the sample label network according to the walk step size of each initial label to obtain a second predicted value between each initial sample image and each initial label.

[0153] This application can construct a heterogeneous association network based on the sample association network, the label association network, and the sample label network, and then perform a restart random walk on the heterogeneous association network according to the walk step size of each node (specifically, the initial label and the initial sample image).

[0154] Specifically, this application uses an initial sample image as the starting node and the initial label associated with that starting node as the intermediate node, obtaining the potential label relationship between the initial sample image and the initial label through a traversal walk. To simulate the above process, this application can perform a random walk starting from the initial sample image, and then traverse the label nodes according to the walk step size of the initial sample image and the heterogeneous association network. Using the initial sample image p... i For example, regarding the initial sample image p i The random walk process with the starting node can be represented by the following formula (12):

[0155]

[0156] in, In the t-th iteration, p i and l j The first predicted value is the one that indicates a potential label association, where t is a positive integer set by a user. α represents the restart probability of the random walk, and α > 0. It is for S l The result of Laplace normalization, Among them, D l It is a diagonal matrix. Indicates O p (p i Round down. When t > 0 p (p i When this happens, this application will stop redirecting from p. i The start of a random walk is the end of the current random walk.

[0157] In other words, with the initial sample image pi As the starting node, this application can proceed according to the initial sample image p. i The walk step size, for the label association network (such as S) l ) and the sample label network W pl A random walk is performed to obtain the initial sample image p. i The first predicted value between each of the initial labels, such as wait.

[0158] Similarly, this application can also use an initial label as the starting node and traverse the initial sample image (node) based on a heterogeneous association network. This traversal process can obtain the probability of a potential association between the initial sample image and the initial label. To simulate this process, this application can perform a random walk starting from the initial label, traversing the sample image nodes according to the random step size of the initial label and the heterogeneous association network. Using the initial label l... j For example, regarding the initial label l j The random walk process with the starting node can be represented by the following formula (13):

[0159]

[0160] in, In the t-th iteration, p i and l j There is a second predicted value with potential label association, where t is a positive integer set by a user, and α represents the restart probability of the random walk, and α > 0. It is for S p The result of Laplace normalization, Among them, D p It is a diagonal matrix. Indicates O l (l j Round down. When t > 0 l (l j When this happens, this application will stop redirecting from l j The start of a random walk is the end of the current random walk.

[0161] In other words, with the initial label l j As the starting node, this application can be based on this initial tag l j The walk step size, for the sample association network S p and the sample label network W pl Perform a random walk to obtain the initial label l j The second predicted value between each of the initial sample images, such as wait.

[0162] S605. Calculate the confidence level between each initial sample image and each initial label based on the first predicted value and the second predicted value between each initial sample image and each initial label.

[0163] After obtaining the first and second predicted values ​​between each initial sample image and each initial label, this application can perform calculations on them according to predefined rules, such as averaging, thereby obtaining the confidence level between each initial sample image and each initial label. Referring to the example shown in formulas (12) and (13) above, the initial sample image p i and initial label l j The confidence level between them can be expressed by the following formula (14):

[0164]

[0165] Among them, F(p) i ,l j ) represents p i and l j The confidence level between F(p) i ,l j The larger the value of ), the stronger the initial sample image p. i Marked as the initial label l j The higher the probability, the greater the probability. In other words, F(p) i ,l j ) can be used to reflect p i Tagged / categorized as l j The probability size.

[0166] S308. Determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label.

[0167] After obtaining the confidence level between each initial sample image and each initial label, this application can determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label. Specifically, taking any initial sample image as an example, this application will determine whether the confidence level between any initial sample image and each initial label exceeds a preset threshold, and classify the initial labels corresponding to those exceeding the preset threshold into the multi-label classification result of the any initial sample image.

[0168] By implementing the embodiments of this application, a computer device acquires n initial sample images and m initial labels; based on the n initial sample images and m initial labels, a sample association network, a label association network, and a sample label network are constructed. The sample association network reflects the similarity between the initial sample images, the label association network reflects the association between the initial labels, and the sample label network reflects the association between the initial sample images and the initial labels; a random walk is performed on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label; based on the confidence level between each initial sample image and each initial label, the multi-label classification result of each initial sample image is determined. It is evident that implementing the embodiments of this application comprehensively considers the association between sample images, the association between labels, and the association between sample images and labels to achieve multi-label classification of sample images. This solves the problems of existing multi-label image classification methods that ignore the dependencies between labels themselves and reduce the accuracy and precision of label classification, thereby improving the accuracy and precision of multi-label image classification.

[0169] It should be understood that, although Figures 1-6 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-6 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0170] Based on the above embodiments, embodiments of the apparatus and device involved in this application are described below. Please refer to... Figure 7 This is a schematic diagram of the structure of an image multi-label classification device disclosed in an embodiment of this application. Figure 7 The device shown includes: an acquisition module 701, a construction module 702, a processing module 703, and a determination module 704, wherein:

[0171] The acquisition module 701 is used to acquire n initial sample images and m initial labels, where m and n are both positive integers;

[0172] The construction module 702 is used to construct a sample association network, a label association network, and a sample label network based on n initial sample images and m initial labels. The sample association network is used to reflect the similarity between the initial sample images, the label association network is used to reflect the association between the initial labels, and the sample label network is used to reflect the association between the initial sample images and the initial labels.

[0173] The processing module 703 is used to perform a random walk on the sample association network, the label association network and the sample label network to obtain the confidence between each initial sample image and each initial label;

[0174] The determining module 704 is used to determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label.

[0175] In some implementations, the construction module 702 is specifically used for:

[0176] Calculate the sample similarity between any two initial sample images from the n initial sample images;

[0177] The sample association network is constructed based on the sample similarity between any two initial sample images;

[0178] Based on the m initial labels and the known labels corresponding to each initial sample image, construct the label association network and the sample label network, where the known labels are the labels among the n initial labels.

[0179] In some implementations, the construction module 702 is specifically used for:

[0180] A comprehensive similarity calculation is performed on any two of the initial sample images to obtain the comprehensive similarity between the two initial sample images;

[0181] Gaussian kernel similarity is calculated for any two initial sample images to obtain the Gaussian kernel similarity between the two initial sample images;

[0182] The sample similarity between any two initial sample images is calculated based on the comprehensive similarity and the Gaussian kernel similarity between them.

[0183] In some implementations, the construction module 702 is specifically used for:

[0184] The cosine similarity between any two initial sample images is calculated.

[0185] The cosine similarity between any two initial sample images is processed by a preset logic to obtain the processed similarity between any two initial sample images.

[0186] Clustering is performed on the processing similarity between any two initial sample images to obtain the comprehensive similarity between the two initial sample images.

[0187] In some implementations, the construction module 702 is specifically used for:

[0188] Determine the clustering coefficient, which reflects the clustering coefficient of the same image cluster when any two initial sample images are located in the same image cluster;

[0189] The processing similarity between any two initial sample images is corrected based on the clustering coefficient to obtain the comprehensive similarity between any two initial sample images.

[0190] In some implementations, the construction module 702 is specifically used for:

[0191] The Gaussian kernel bandwidth of each initial sample image is determined based on the association between each initial sample image and each initial label;

[0192] The similarity between any two initial sample images is calculated based on the Gaussian kernel bandwidth to obtain the Gaussian kernel similarity between the two initial sample images.

[0193] In some embodiments, before performing a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label, the processing module 703 is further configured to:

[0194] A similarity coefficient is calculated between each initial sample image and each initial label to obtain the similarity coefficient between each initial sample image and each initial label;

[0195] Based on the similarity coefficient between each initial sample image and each initial label, calculate the walk step size for each initial sample image and the walk step size for each initial label;

[0196] Accordingly, the processing module 703 is specifically used for:

[0197] According to the walk step size of each initial sample image, a random walk is performed on the label association network and the sample label network to obtain a first prediction value between each initial sample image and each initial label;

[0198] According to the walk step size of each initial label, a random walk is performed on the sample association network and the sample label network to obtain a second predicted value between each initial sample image and each initial label;

[0199] The confidence level between each initial sample image and each initial label is calculated based on the first predicted value and the second predicted value between each initial sample image and each initial label.

[0200] By implementing the embodiments of this application, n initial sample images and m initial labels are obtained; based on the n initial sample images and m initial labels, a sample association network, a label association network, and a sample label network are constructed. The sample association network reflects the similarity between the initial sample images, the label association network reflects the association between the initial labels, and the sample label network reflects the association between the initial sample images and the initial labels; a random walk is performed on the sample association network, the label association network, and the sample label network to obtain the confidence score between each initial sample image and each initial label; based on the confidence score between each initial sample image and each initial label, the multi-label classification result of each initial sample image is determined. It is evident that implementing the embodiments of this application comprehensively considers the association between sample images, the association between labels, and the association between sample images and labels to achieve multi-label classification of sample images. This solves the problems of existing multi-label image classification methods that ignore the dependencies between labels themselves and reduce the accuracy and precision of label classification, thereby improving the accuracy and precision of multi-label image classification.

[0201] Specific limitations regarding the image multi-label classification device can be found in the limitations of the image multi-label classification method above, and will not be repeated here. Each module in the aforementioned image multi-label classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0202] Please see Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device shown can be a server, and its internal structure diagram can be as follows. Figure 8As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image data and label data, such as initial sample images and initial labels. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an image multi-label classification method.

[0203] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0204] In some embodiments, the image multi-label classification apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 8 The computer device shown operates on this device. The computer device's memory can store the various program modules that make up the image multi-label classification device, for example, Figure 7 The diagram shows an acquisition module 701, a construction module 702, a processing module 703, and a determination module 704. The computer program comprised of these modules causes the processor to execute the steps in the image multi-label classification methods of the various embodiments of this application described in this specification.

[0205] For example, Figure 8 The computer device shown can be used as follows Figure 7 The image multi-label classification device shown in the diagram executes the step of acquiring n initial sample images and m initial labels. The computer device can execute the step of constructing a sample association network, a label association network, and a sample label network based on the n initial sample images and m initial labels through the construction module. The computer device can execute the step of performing random walks on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label through the processing module. The computer device can execute the step of determining the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label through the determination module.

[0206] This application further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0207] This application further discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0208] The above provides a detailed description of an image multi-label classification method and related products disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of the application.

Claims

1. An image multi-label classification method, characterized in that, The method includes: Obtain n initial sample images and m initial labels, where m and n are both positive integers; Based on n initial sample images and m initial labels, a sample association network, a label association network, and a sample label network are constructed. The sample association network is used to reflect the similarity between the initial sample images, the label association network is used to reflect the association between the initial labels, and the sample label network is used to reflect the association between the initial sample images and the initial labels. A random walk is performed on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label; Based on the confidence level between each initial sample image and each initial label, determine the multi-label classification result for each initial sample image; Before performing a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label, the method further includes: A similarity coefficient is calculated between each initial sample image and each initial label to obtain the similarity coefficient between each initial sample image and each initial label; Based on the similarity coefficient between each initial sample image and each initial label, calculate the walk step size for each initial sample image and the walk step size for each initial label; The step of performing a random walk on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label includes: According to the walk step size of each initial sample image, a random walk is performed on the label association network and the sample label network to obtain a first prediction value between each initial sample image and each initial label; According to the walk step size of each initial label, a random walk is performed on the sample association network and the sample label network to obtain a second predicted value between each initial sample image and each initial label; The confidence level between each initial sample image and each initial label is calculated based on the first predicted value and the second predicted value between each initial sample image and each initial label.

2. The method of claim 1, wherein, The construction of a sample association network, a label association network, and a sample label network based on n initial sample images and m initial labels includes: Calculate the sample similarity between any two initial sample images from the n initial sample images; The sample association network is constructed based on the sample similarity between any two initial sample images; Based on the m initial labels and the known labels corresponding to each initial sample image, construct the label association network and the sample label network, where the known labels are the labels among the m initial labels.

3. The method of claim 2, wherein, The step of calculating the sample similarity between any two initial sample images from the n initial sample images includes: A comprehensive similarity calculation is performed on any two initial sample images to obtain the comprehensive similarity between the two initial sample images; Gaussian kernel similarity is calculated for any two initial sample images to obtain the Gaussian kernel similarity between the two initial sample images; The sample similarity between any two initial sample images is calculated based on the comprehensive similarity and the Gaussian kernel similarity between them.

4. The method of claim 3, wherein, The step of calculating the comprehensive similarity between any two initial sample images to obtain the comprehensive similarity between the two initial sample images includes: The cosine similarity between any two initial sample images is calculated to obtain the cosine similarity between the two initial sample images. The cosine similarity between any two initial sample images is processed by a preset logic to obtain the processed similarity between any two initial sample images. Clustering is performed on the processing similarity between any two initial sample images to obtain the comprehensive similarity between the two initial sample images.

5. The method of claim 4, wherein, The step of performing clustering processing on the processing similarity between any two initial sample images to obtain the comprehensive similarity between any two initial sample images includes: Determine the clustering coefficient, which reflects the clustering coefficient of the same image cluster when any two initial sample images are located in the same image cluster; The processing similarity between any two initial sample images is corrected based on the clustering coefficient to obtain the comprehensive similarity between any two initial sample images.

6. The method of claim 3, wherein, The step of calculating the Gaussian kernel similarity between any two initial sample images to obtain the Gaussian kernel similarity between any two initial sample images includes: The Gaussian kernel bandwidth of each initial sample image is determined based on the association between each initial sample image and each initial label; The similarity between any two initial sample images is calculated based on the Gaussian kernel bandwidth to obtain the Gaussian kernel similarity between the two initial sample images.

7. An image multi-label classification apparatus characterized by comprising: include: The acquisition module is used to acquire n initial sample images and m initial labels, where m and n are both positive integers; A construction module is used to construct a sample association network, a label association network, and a sample label network based on n initial sample images and m initial labels. The sample association network is used to reflect the similarity between the initial sample images, the label association network is used to reflect the association between the initial labels, and the sample label network is used to reflect the association between the initial sample images and the initial labels. The processing module is used to perform random walks on the sample association network, the label association network, and the sample label network to obtain the confidence level between each initial sample image and each initial label; The determination module is used to determine the multi-label classification result of each initial sample image based on the confidence level between each initial sample image and each initial label; The calculation module is used to calculate the similarity coefficient between each initial sample image and each initial label to obtain the similarity coefficient between each initial sample image and each initial label; Based on the similarity coefficient between each initial sample image and each initial label, calculate the walk step size for each initial sample image and the walk step size for each initial label; The processing module is specifically used to perform a random walk on the label association network and the sample label network according to the walk step size of each initial sample image, so as to obtain a first prediction value between each initial sample image and each initial label; According to the walk step size of each initial label, a random walk is performed on the sample association network and the sample label network to obtain a second predicted value between each initial sample image and each initial label; based on the first predicted value and the second predicted value between each initial sample image and each initial label, the confidence level between each initial sample image and each initial label is calculated.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images