Non-convex imbalanced multi-instance multi-label learning method based on dual-granularity annotation

Abstract

The present invention provides a non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation. The method comprises: obtaining a target multi-instance package set for a target multi-instance multi-label learning task, wherein each package in the multi-instance package set consists of a number of examples and corresponds to a number of label classes, wherein the examples can be divided into key examples and irrelevant examples according to their respective label classes; and detecting key examples of the target multi-instance packages in the test multi-instance package set using a DL-MIML model to obtain classification results at both the example and package levels. The present invention has the beneficial effects of improving annotation efficiency and annotation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer and artificial intelligence technology, and in particular to a non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation. Background Art

[0002] The large number of multi-semantic objects with complex structures has driven the development of the weakly supervised paradigm of multi-instance multi-label learning. This approach uses multiple examples to describe polysemantic objects with complex structures and learns a mapping from a set of examples to a set of class labels. Currently, many methods aim to efficiently recognize multi-semantic objects. However, in practical tasks such as image text annotation and drug screening, domain experts are required to manually inspect large amounts of images and text, or identify effective drug molecules from a vast library of drugs, a process that is time-consuming and labor-intensive.

[0003] Therefore, automatic example-level labeling is crucial. Existing methods that can achieve example-level label prediction usually ignore some issues:

[0004] (1) Not every instance in an object plays the same role. It is the diversity of features across different instances that gives the object its polysemantics. It is unreasonable for some methods to treat every instance in an object as identical.

[0005] (2) The reconstructed dataset suffers from a serious label imbalance problem. The label distribution in applications is often skewed, which inevitably leads to label imbalance. Some methods remove examples from the objects, which dramatically increases the number of samples in the reconstructed dataset, changing the label distribution and exacerbating the imbalance problem.

[0006] (3) Some methods simply degenerate the multi-instance multi-label learning task into a binary classification task, ignoring the label correlation, resulting in a decrease in model classification performance and poor generalization ability. Summary of the Invention

[0007] The main purpose of the embodiment of the present invention is to propose a non-convex unbalanced multi-instance multi-label learning method based on dual-granularity labeling, which improves labeling efficiency and labeling accuracy.

[0008] One aspect of the present invention provides a non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, characterized by comprising:

[0009] Obtain a target multi-instance bag set for a target multi-instance multi-label learning task, where each bag in the multi-instance bag set consists of several examples and corresponds to several label classes, where the examples can be divided into key examples and irrelevant examples according to their label classes;

[0010] The DL-MIML model is used to detect key examples of the target multi-instance package in the multi-instance package set to be tested, and classification results at both the example and package levels are obtained.

[0011] The DL-MIML model is obtained by the following steps:

[0012] Determining contribution values ​​of examples in a bag of a target multi-example bag set to a labeled label using an example weight vector, and determining key examples, and determining a bag representation according to the contribution values ​​of the labels and the key examples, wherein the key examples are used to characterize that the examples belong to the labeled label;

[0013] Calculate the label scores of key examples and irrelevant examples in the package, and use the reweighted loss function to determine the loss of each package based on the label scores;

[0014] The objective function based on the training set is processed with a low-rank constraint to obtain a second objective function, where the first objective function is to minimize the loss;

[0015] Processing the example weight vector using a sparse constraint to obtain a DL-MIML model;

[0016] Obtain the variables to be learned in the DL-MIML model and optimize and solve them using the block coordinate-Nesterov accelerated projection descent algorithm.

[0017] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the example weight vector u i，c Learning n through classification model i The contribution of examples to the i-th bag being labeled with the c-th label, where the example weight vector is used to align features and labels at the bag level; where the target multi-example bag set {X1,X2,...,X N}, N represents the total number of packets, and packet X i A set of examples Composition, n i is the total number of examples in the i-th bag, and has a label vector Y i =[y i,1 ,y i,2 ,...,y i,C ], when package X i When marked by the cth label, y i,c =1, otherwise 0, where C represents the dimension of the label space; at the same time, each bag includes at least one key example that characterizes the label features.

[0018] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the label scores of key examples and irrelevant examples in the package are calculated, and the loss of each package is determined using a reweighted loss function based on the label scores, including:

[0019] The classification model is trained by a reweighted loss function, wherein the reweighted loss function for:

[0020]

[0021] in, is the score of the i-th package belonging to the c-th label class calculated based on the key example, is the label score calculated based on the j-th irrelevant example, and T is the transposition operation;

[0022] Loss per packet for:

[0023]

[0024] is the reweighting weight, and ρ is a hyperparameter.

[0025] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the objective function based on the training bag set is processed with a low-rank constraint to obtain a second objective function, wherein the first objective function is to minimize the loss, including:

[0026] The objective function based on the training set is processed using low-rank constraints:

[0027]

[0028] Among them, ① is determined by low-rank decomposition, and ② is determined by and Determine, where λ is the regularization factor, W represents a low-rank matrix of K, and W = AB T ,in and Rank(W) can be decomposed into rank(A)+rank(B), and rank(W)=‖W‖ * , ||*||F represents the Frobenius norm, is a real number space, rank represents the rank, and d represents the matrix dimension.

[0029] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the example weight vector is processed using a sparse constraint to obtain a DL-MIML model, including:

[0030] Obtain the bag feature obtained by embedding the examples based on the weight vector, that is, the embedded bag feature is a convex combination of key examples;

[0031] Adding a sparse constraint to the weight vector according to the convex combination, where the sparse constraint is a non-convex sparsity-induced regularization term, where the regularization term includes l0-norm, l1-norm and non-negative constraint;

[0032] According to the weight vector with sparse constraints and the second loss function, the DL-MIML model is obtained as follows:

[0033]

[0034] where r is a hyperparameter regarding the number of critical examples.

[0035] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the variables to be learned in the DL-MIML model are obtained and optimized and solved using the block coordinate-Nesterov accelerated projection descent algorithm, including:

[0036] The variables to be learned are recorded as:

[0037] w=(A,{bc},{ui @ c})

[0038] Among them, in the constraint set Add for u i，c The constraint, that is

[0039] According to the variables to be learned, the optimization problem is determined as

[0040]

[0041] in Indicator function

[0042] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the optimization solution is performed by using the block coordinate-Nesterov accelerated projection descent algorithm, including:

[0043] The block coordinate descent algorithm fixes the non-target variable block at the latest updated value and iteratively adds j Minimize F and get ω after the kth update j The value of , that is:

[0044]

[0045] Among them, N v is the total number of variable blocks;

[0046] In each iteration, the DL-MIML model adjusts the variable block ω by minimizing the approximate linear surrogate functionj Update, that is:

[0047]

[0048] Among them, α k is the step length;

[0049] The algorithm passes through j Introducing auxiliary variables in the optimization process Initialize k=0, a0=1,

[0050] The auxiliary variable z j satisfy and In the kth iteration, DL-MIML first calculates the smallest i≥0 such that

[0051]

[0052] By using the smallest i, we can get Update the variables to be learned to

[0053]

[0054] According to the updated variables to be learned, the auxiliary variables for the next iteration are obtained, that is,

[0055]

[0056] Repeat the iteration until convergence.

[0057] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the optimization of the variable A to be learned includes:

[0058] Fixed {b c} and {u i，c}, determine the objective function is equivalent to:

[0059]

[0060] The gradient of the objective function f with respect to A is

[0061]

[0062] The Nesterov-accelerated gradient algorithm is used to calculate the optimization result of the variable A to be learned.

[0063] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the variable to be learned b c The optimizations include:

[0064] Fixed A, {b m|m≠c}, and {u i，c}, determine the objective function is equivalent to:

[0065]

[0066] The objective function f with respect to b c The gradient of

[0067]

[0068] The variable b to be learned is calculated using the Nesterov-accelerated gradient algorithm c Optimization results.

[0069] According to the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, the variable u to be learned is i，c The optimizations include:

[0070] Fixed A, {b c} and {u m，n |m≠i,n≠c}, the objective function is equivalent to

[0071]

[0072] The DL-MIML model is used to process the convex optimization problem min f(u i，c ), the optimal solution is In the collection The sparse Euclidean projection on

[0073]

[0074] in represents the projection operator, I * Yes Reserve The index set of the first r largest positive elements in , and isI * The complement of It can be obtained by gradient-based algorithm, the objective function f is about u i，c The gradient of

[0075]

[0076] The beneficial effects of the present invention are as follows: it takes into account the problem of key example identification in the multi-instance multi-label learning process, and can realize two-level data classification and labeling of packages and examples based on package labels and example features; at the same time, it takes into account the unbalanced label distribution and label correlation in processing multi-instance multi-label learning tasks, and balances the quantitative gap between key examples and irrelevant examples through the weighted loss function, enhances the model's attention to minority class samples, and effectively alleviates the problem of unbalanced label distribution. The kernel function low-rank approximation label correlation is used, which can be combined with the heavily weighted loss in a "convex" form, so that the method can train a multi-instance multi-label learning model that can simultaneously handle label imbalance and label correlation; the block coordinate-Nesterov accelerated projection descent algorithm can enable the model to converge to the optimal solution at a faster speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0078] Figure 1 1 is a flowchart of a non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to an embodiment of the present invention.

[0079] Figure 2 2 is a schematic diagram of the DL-MIML model framework of an embodiment of the present invention.

[0080] Figure 3 1 is a flow chart of a block coordinate-Nesterov accelerated projection descent algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. In the subsequent description, suffixes such as "module," "component," or "unit" used to represent elements are used solely to facilitate the description of the present invention and have no specific meaning in themselves. Therefore, "module," "component," or "unit" may be used interchangeably. "First," "second," and the like are used solely to distinguish technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the consecutive numbering of method steps is for ease of review and understanding. In conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the order of implementation of the steps does not affect the technical effects achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and should not be construed as limiting the present invention.

[0082] Figure 1 This is a flow chart of a non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to an embodiment of the present invention, which includes but is not limited to steps S100 to S700, wherein S100 to S200 are a non-convex unbalanced multi-instance multi-label learning method, and wherein S300 to S700 are a flow chart of a DL-MIML model construction process.

[0083] S100, obtaining a target multi-instance package set for a target multi-instance multi-label learning task, wherein each package in the multi-instance package set consists of several examples and corresponds to several label classes, wherein the examples can be divided into key examples and irrelevant examples according to the different label classes to which they belong.

[0084] S200, detecting key examples of the target multi-instance package in the multi-instance package set to be tested through the DL-MIML model, and obtaining classification results at both the example and package levels;

[0085] S300, using an example weight vector to determine the contribution value of the examples in the package of the target multi-example package set to the marked label, and determining the key examples, determining the package representation according to the contribution value of the label and the key examples, wherein the key examples are used to characterize that the examples belong to the marked label.

[0086] In some embodiments, in a multi-instance multi-label learning task, if a package has a certain label, then at least one example in it belongs to the label class, otherwise it means that all examples in the package do not belong to the label class. The examples belonging to the corresponding label class are called "key examples" or "Witness". For the multi-instance package set to be trained {X1, X2, ..., X N}, package X i A set of examples Composed of, and has a label vector Y i =[y i,1 ,y i,2 ,...,y i,C ], when package X i When marked by the cth label, y i,c =1, otherwise 0.

[0087] In some embodiments, reference Figure 2 : This is a schematic diagram of the DL-MIML model framework of an embodiment of the present invention, where an example weight vector is used to determine the packet representation of a packet. For the training of a general classification model, let y and Represent the true value and predicted value of the data point x respectively. The model minimizes the loss function in However, due to the inconsistency between the level of features and labels in multi-instance multi-label learning tasks, y and x cannot be obtained at the same time. DL-MIML introduces the example weight vector ui，c , to measure n i The contribution of examples to the cth label of the i-th package. It is worth noting that the example weight vector can not only detect key instances, but also eliminate the influence of irrelevant examples, which helps to enhance the model's discriminative ability. i，c , It can be used as a new representation of the i-th bag to identify the c-th label class, thereby aligning features and labels at the bag level.

[0088] S400 , calculating the label scores of the key examples and irrelevant examples in the package, and determining the loss of each package using a reweighted loss function based on the label scores.

[0089] In some embodiments, the model needs to select an appropriate Cross entropy loss is a classic loss function in the field of machine learning and can be extended to multi-label scenarios to train models. The cross entropy loss function is Where s is the score of the model output corresponding to the label class. Therefore, for the i-th package, DL-MIML calculates s based on the key examples as Calculate s based on the jth irrelevant example as Since there are n i instances, the loss function is expressed as:

[0090]

[0091] In order to balance the number of key examples and irrelevant examples, we introduce weights Where ρ is a hyperparameter, the loss of the i-th bag is

[0092] S500 , processing the objective function based on the training package set using a low-rank constraint to obtain a second objective function, wherein the first objective function is to minimize the loss.

[0093] In some embodiments, the training dataset is a multi-instance multi-label dataset for training.

[0094] Considering the label correlation in the multi-instance multi-label sample space, DL-MIML incorporates the low-rank constraint of the model into the objective function. Specifically, the low-rank matrix W with K can be expressed as W = AB through low-rank decomposition T ,in and The rank(W) can be decomposed into rank(A)+rank(B), and rank(W)=‖W‖ * , then Therefore, the loss of the i-th packet can also be expressed as Based on the goal of minimizing the loss function, we have

[0095]

[0096] Among them, ① is established by low-rank decomposition, ② is established by and And it is established, λ is the regularization factor, C: label space dimension (that is, the dataset involves a total of C label classes for classification problems); ||*|| F The hollow R represents the real space (i.e., A is a d*K-dimensional matrix, and B is a C*K-dimensional matrix), where d represents the matrix dimension. Rank represents the rank, meaning that the rank of matrix W can be decomposed into the rank of matrix A + the rank of matrix B.

[0097] S600: Process the example weight vector using a sparse constraint to obtain a DL-MIML model.

[0098] In some embodiments, in order to meet the needs of actual applications, the example weight vector must be constrained. On the one hand, based on the standard assumption, the bag contains at least one key example to characterize the label feature, while other examples are irrelevant examples. DL-MIML uses the l0-norm to constrain the number of selected key examples. On the other hand, by definition, the elements in the example weight vector represent the contribution of the example to the label. Therefore, all its elements must be non-negative. At the same time, the example weight vector plays an important role in embedding the original bag space into the discriminant feature space. The new feature representation of the bag after mapping can be regarded as a convex combination of the selected key examples. It is natural to require that the sum of all elements in the example weight vector is 1, which can be achieved by adding the l1-norm constraint. Therefore, DL-MIML includes a non-convex sparsity-inducing regularization term that adds the l0-norm, l1-norm and non-negative constraints to the example weight vector.

[0099] To summarize, DL-MIML modeling is as follows:

[0100]

[0101] Here, r is a hyperparameter regarding the number of critical examples.

[0102] S700: Obtain the variables to be learned in the DL-MIML model and optimize and solve them using the block coordinate-Nesterov accelerated projection descent algorithm.

[0103] In some embodiments, there are three sets of variables to be learned in the model. c},{u i，c})and The optimization problem is

[0104]

[0105] in The indicator function is defined as

[0106] Note: variable A, {b c} and {u i，c} is coupled in this framework that includes smooth and non-smooth functions; due to the use of non-convex sparsity-inducing regularization to constrain u i，c , so each piece of ω is non-convex and non-smooth, which leads to the objective being a non-trivial non-convex and non-smooth optimization problem. It is extremely challenging to directly use general gradient descent algorithms to solve such problems. Therefore, DL-MIML developed the BC-Nesterov-PD algorithm to handle this problem.

[0107] Specifically, DL-MIML uses the Block Coordinate Descent (BCD) algorithm to iteratively minimize F for each variable block while fixing the other variable blocks to their latest updated values. ω after the kth update j The value of In each iteration, DL-MIML updates the variable block ω by minimizing the approximate linear surrogate function j ,Right now

[0108]

[0109] Among them, α k is the step length.

[0110] DL-MIML adds a "momentum term" formed by the weighted accumulation of all past gradients on the basis of gradient descent, and performs a "forward" gradient calculation in the momentum direction before adjusting the update frontier, predicting the change of the objective function in the momentum direction and accelerating the convergence of this minimization problem. j Introduced in the optimization process Initialize k=0, a0=1, where z j satisfy and In the kth iteration, DL-MIML first calculates the smallest i≥0 such that

[0111]

[0112] The existence of i can be proved. Through the smallest i, we can get So the updated variable is

[0113]

[0114] Based on this, the auxiliary variables for the next iteration are obtained, namely

[0115]

[0116] Repeat the above iterations until convergence. The specific process is shown as follows Figure 3 The block coordinate-Nesterov accelerated projection descent algorithm flow diagram is shown.

[0117] In some embodiments, the variables to be optimized are as follows:

[0118] (1) Optimize A

[0119] Fixed {b c} and {u i，c}, the objective function is equivalent to:

[0120]

[0121] Therefore, the problem minf(A) is a convex optimization problem that can be solved by gradient-based methods. The gradient of the objective function f with respect to A is

[0122]

[0123] Next, follow the steps of the Nesterov Accelerated Gradient (NAG) algorithm to solve the problem.

[0124] (2) Optimize b c

[0125] Fixed A, {b m |m≠c}, and {u i，c}, the objective function is equivalent to

[0126]

[0127] Therefore, the problem min f(b c ) is a convex optimization problem that can be solved by gradient-based methods. The objective function f is related to b c The gradient of

[0128]

[0129] Next, follow the NAG algorithm steps to solve the problem.

[0130] (3) Optimize u i,c

[0131] Fixed A, {b c} and {um,n |m≠m≠i,n≠c}, the objective function is equivalent to

[0132]

[0133] DL-MIML uses the Projected Gradient Descent (PGD) algorithm to solve the convex optimization problem min f(u i,c ). Its optimal solution is In the collection The sparse Euclidean projection on

[0134]

[0135] in Represents the projection operator. I * Yes Reserve The index set of the first r largest positive elements in , and isI * The complement of . It can be obtained by gradient-based algorithm, the objective function f is about u i，c The gradient of

[0136]

[0137] In some embodiments, the DL-MIML model of the embodiments of the present invention is used to detect, identify, classify and record the data required for human activity recognition (HAR) tasks, such as the daily movement and behavior patterns of individuals detected, identified, classified and recorded by different sensors, such as movement and physical activity status (walking, running, driving or standing still) in different locations; by collecting these data through smartphones with sensors and communication facilities, preprocessing and extracting relevant features (such as time and frequency domain features); using these relevant features, a multi-instance package is constructed to represent a combination of various human activities. It is understandable that individual human activities are variable, so the DL-MIML model of the embodiments of the present invention identifies key instances, handles imbalance problems and low-rank approximation of label correlations, thereby improving the accuracy and recognition performance of individual activity recognition. For another example, the DL-MIML model of the embodiments of the present invention is used to perform a drug-target prediction task based on molecular conformation. A single drug can be regarded as a "package", which can have multiple molecular conformations or multiple chemical fragments, which are regarded as "instances". These different examples may be associated with different protein targets (labels), allowing the drug to be combined with multiple protein targets. It is understandable that the drug molecule library data is huge. Therefore, through the DL-MIML model of the embodiment of the present invention, key instance identification, processing imbalance problems and low-rank approximation of label correlation, the classification and detection tasks of molecular conformation (example) and drug (package) level data are realized, which greatly improves efficiency.

[0138] An embodiment of the present invention further provides an electronic device, the electronic device including a processor and a memory;

[0139] The memory stores a program;

[0140] The processor executes a program to perform the aforementioned non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation; the electronic device has the function of carrying and running the software system for glass flow control provided by an embodiment of the present invention, such as a personal computer, a minicomputer, a main frame, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicating with a charged particle tool or other imaging device, etc.

[0141] An embodiment of the present invention further provides a computer-readable storage medium storing a program, wherein the program is executed by a processor to implement the non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation as described above.

[0142] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0143] Embodiments of the present invention further disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned non-convex imbalanced multi-instance multi-label learning method based on dual-granularity annotation.

[0144] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0145] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0146] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0147] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0148] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0150] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0151] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation, characterized by: include: Obtain a target multi-instance bag set for a target multi-instance multi-label learning task, where each bag in the multi-instance bag set consists of several examples and corresponds to several label classes, where the examples can be divided into key examples and irrelevant examples according to their label classes; The DL-MIML model is used to detect key examples of the target multi-instance package in the multi-instance package set to be tested, and classification results at both the example and package levels are obtained. The DL-MIML model is obtained by the following steps: Determining contribution values ​​of examples in a bag of a target multi-example bag set to a labeled label using an example weight vector, and determining key examples, and determining a bag representation according to the contribution values ​​of the labels and the key examples, wherein the key examples are used to characterize that the examples belong to the labeled label; Calculate the label scores of key examples and irrelevant examples in the package, and use the reweighted loss function to determine the loss of each package based on the label scores; The objective function based on the training set is processed with a low-rank constraint to obtain a second objective function, where the first objective function is to minimize the loss; Processing the example weight vector using a sparse constraint to obtain a DL-MIML model; Obtain the variables to be learned in the DL-MIML model and optimize and solve them using the block coordinate-Nesterov accelerated projection descent algorithm.

2. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 1 is characterized in that The example weight vector u i，c Learning n through classification model i The contribution of examples to the i-th bag being labeled with the c-th label, where the example weight vector is used to align features and labels at the bag level; The target multi-instance bag set {X1,X2,...,X N }, N represents the total number of packets, and packet X i A set of examples Composition, n i is the total number of examples in the i-th bag, and has a label vector Y i =[y i,1 ,y i,2 ,...,y i,C ], when package X i When marked by the cth label, y i,c =1, otherwise 0, where C represents the dimension of the label space; at the same time, each bag includes at least one key example that characterizes the label features.

3. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 2 is characterized in that The label scores of the key examples and irrelevant examples in the package are calculated, and the loss of each package is determined using a reweighted loss function based on the label scores, including: The classification model is trained by a reweighted loss function, wherein the reweighted loss function for: in, is the score of the i-th package belonging to the c-th label class calculated based on the key example, is the label score calculated based on the j-th irrelevant example, and T is the transposition operation; Loss per packet for: is the reweighting weight, and ρ is a hyperparameter.

4. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 3 is characterized in that: The objective function based on the training bag set is processed using a low-rank constraint to obtain a second objective function, wherein the first objective function is to minimize the loss, including: The objective function based on the training set is processed using low-rank constraints: Among them, ① is determined by low-rank decomposition, and ② is determined by and Determine, where λ is the regularization factor, W represents a low-rank matrix of K, and W = AB T ,in and Rank(W) can be decomposed into rank(A)+rank(B), and rank(W)=‖W‖ * , ||*‖ F represents the Frobenius norm, is a real number space, rank represents the rank, and d represents the matrix dimension.

5. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 4 is characterized in that: The method of processing the example weight vector using a sparse constraint to obtain a DL-MIML model includes: Obtain the bag feature obtained by embedding the examples based on the weight vector, that is, the embedded bag feature is a convex combination of key examples; Adding a sparse constraint to the weight vector according to the convex combination, where the sparse constraint is a non-convex sparsity-induced regularization term, where the regularization term includes l0-norm, l1-norm and non-negative constraint; According to the weight vector with sparse constraints and the second loss function, the DL-MIML model is obtained as follows: where r is a hyperparameter regarding the number of critical examples.

6. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 5, characterized in that: The step of obtaining the variables to be learned in the DL-MIML model and optimizing and solving them using the block coordinate-Nesterov accelerated projection descent algorithm includes: The variables to be learned are recorded as: ω=(A,{b c },{u i，c }) Among them, in the constraint set C i，c Add for u i，c The constraint, that is According to the variables to be learned, the optimization problem is determined as in Indicator function 7. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 6, characterized in that: The optimization solution using the block coordinate-Nesterov accelerated projection descent algorithm includes: The block coordinate descent algorithm fixes the non-target variable block at the latest updated value and iteratively adds j Minimize F and get ω after the kth update j The value of , that is: Among them, N v is the total number of variable blocks; In each iteration, the DL-MIML model adjusts the variable block ω by minimizing the approximate linear surrogate function j Update, that is: Among them, α k is the step length; The algorithm passes through j Introducing auxiliary variables in the optimization process Initialize k=0, a0=1, The auxiliary variable z j satisfy and In the kth iteration, DL-MIML first calculates the smallest i≥0 such that By using the smallest i, we can get Update the variables to be learned to According to the updated variables to be learned, the auxiliary variables for the next iteration are obtained, that is, Repeat the iteration until convergence.

8. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 6, characterized in that: The optimization of the variable A to be learned includes: Fixed {b c } and {u i，c }, determine the objective function is equivalent to: The gradient of the objective function f with respect to A is The Nesterov-accelerated gradient algorithm is used to calculate the optimization result of the variable A to be learned.

9. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 6, characterized in that: The variable to be learned b c The optimizations include: Fixed A, {b m |m≠c}, and {u i，c }, determine the objective function is equivalent to: The objective function f with respect to b c The gradient of The variable b to be learned is calculated using the Nesterov-accelerated gradient algorithm c Optimization results.

10. The non-convex unbalanced multi-instance multi-label learning method based on dual-granularity annotation according to claim 6, characterized in that: Variable u to be learned i，c The optimizations include: Fixed A, {b c } and {u m，n |m≠i,n≠c}, the objective function is equivalent to The DL-MIML model is used to process the convex optimization problem min f(u i，c ), the optimal solution is In set C i，c The sparse Euclidean projection on in represents the projection operator, I * Yes Reserve The index set of the first r largest positive elements in , and isI * The complement of It can be obtained by gradient-based algorithm, the objective function f is about u i，c The gradient of

Images