A weakly supervised object detection method in images based on class-agnostic foreground mining
By generating foreground attention map and graph matching technology, the positioning and classification tasks are separated, and the problems of inaccurate positioning and difficulty in multi-instance detection in weak-supervised target detection are solved, and the detection performance is improved.
Patent Information
- Application Number
- CN202210133345.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-10
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2042-02-10
AI Technical Summary
The existing weakly supervised object detection model has inaccuracy in the positioning and classification tasks, and fails to effectively utilize the relationship between candidate boxes and labels, resulting in insufficient positioning performance and difficulty in multi-instance detection.
By generating a foreground attention map filtering candidate boxes, constructing instance spatial graphs and label semantic graphs for graph matching, semantic graphs, separating positioning and classification tasks, and generating pseudo-supervised boxes and pseudo-instance tags to improve detection performance.
The two-way improvement of positioning and classification performance is achieved, effectively alleviating the problems of local positioning and multi-instance loss, and improving the overall performance of weakly supervised target detection.
Smart Images
Figure CN114565752B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to an image weakly supervised target detection method based on class-agnostic foreground mining. Background Art
[0002] Object detection is an important research direction in the field of computer vision and has a wide range of applications in real life. With the development of deep learning, object detection technology has also made great progress. However, object detection in a fully supervised mode requires precise annotation boxes, which has a high annotation cost. On the contrary, image category information is easier to obtain, so people began to study weakly supervised object detection, which completes the object detection task given only image category information.
[0003] The Weakly Supervised Deep Detection Networks (WSDDN) proposed in 2015 designed weakly supervised object detection into an end-to-end structure for the first time. The Multiple Instance Detection Network with Online Instance Classifier Refinement (OICR) proposed in 2017 added an online refinement module based on WSDDN, which improved the detection performance of the model and became the main baseline model in the field of weakly supervised object detection. Deep learning methods for weakly supervised object detection face two main challenges: First, the detected boxes basically only cover the local salient areas of the object. This is because the category labels of the image can only be used for classification loss. The classification of the Convolutional Neural Network (CNN) only needs to rely on the most representative areas of the image, and the candidate boxes with high classification scores are often those small local boxes. Second, when there are multiple instances of the same category in the image, not all instances can be detected. This is because the number of different categories in the image is not equal, and the number of candidate boxes cannot be determined.
[0004] In order to solve the above problems, the work in recent years can be divided into the following directions.
[0005] 1. Optimize the candidate boxes before inputting into the network, such as filtering candidate boxes based on categories; using Class Activation Mapping (CAM) to assist in generating candidate boxes, etc.
[0006] 2. Optimize the output prediction box. Remove the salient box with the highest score and merge the remaining neighboring boxes; use a voting mechanism to calculate the contribution of each prediction box to the image, generate a heat map, and generate a larger box based on the heat map.
[0007] 3. Use the information contained in the image itself to mine high-quality candidate boxes. For example, use multi-task learning to combine with the segmentation task to find a more complete and compact prediction box; use the idea of graph to find candidate boxes with similar features to the most significant box, so as to mine more examples of the same type.
[0008] 4. Use features of different depths to represent the image more comprehensively. Among the features output by CNN, low-dimensional features represent more appearance information, and high-dimensional features represent semantic information. Fusion of the two can better represent the entire instance, thereby alleviating the problem of the network focusing more on local features. 5. Find more comprehensive visual patterns. Objects of the same category also have differences in shape, size, appearance, etc. If you want to better represent each category, you can use a data structure such as a dictionary to save one or more common features to minimize intra-class differences and maximize inter-class differences.
[0009] At present, the problems of weakly supervised target detection models in existing technologies are mainly:
[0010] First, using classification to guide the localization task, the classification and localization tasks themselves focus on different features. Using classification to guide the localization task will lead to inaccurate positioning. Decoupling classification and localization can improve the performance of the two subtasks respectively.
[0011] Secondly, most of the work does not consider the relationship between the candidate box and the label, and a few works only consider the relationship between the candidate box and the label. However, the relationship between the candidate box and the label also hides important information. An effective tool for mining this information is graph matching. Based on the candidate box and the label, the graphs are constructed separately, and the two graphs are matched. This can effectively mine the information between the candidate box and the label, thereby completing the classification task of the located object. Summary of the invention
[0012] In view of the above problems existing in the prior art, an embodiment of the present invention proposes a method for weakly supervised object detection in images based on class-agnostic foreground mining, so as to effectively improve the weakly supervised object detection performance of images.
[0013] In order to achieve the above object, the present invention adopts the following technical scheme.
[0014] A weakly supervised object detection method for images based on class-agnostic foreground mining, comprising:
[0015] Generate a foreground attention map based on the image to be detected by the convolutional neural network CNN;
[0016] Calculate the foreground relative confidence (FRC) of each candidate frame based on the foreground attention map, and select the foreground candidate frames according to the FRC of each candidate frame;
[0017] constructing an instance space graph based on the foreground candidate frame, constructing a label semantic graph based on the label of the data set, performing graph matching on the instance space graph and the label semantic graph, and classifying each foreground candidate frame according to the graph matching result;
[0018] A pseudo supervisory frame is generated according to the classification results of each foreground candidate frame, and the pseudo supervisory frame is merged with the spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and the pseudo instance label is used as the target detection result of the image to be detected.
[0019] Preferably, the method of generating a foreground attention map based on the image to be detected by a convolutional neural network (CNN) comprises:
[0020] Given an image I∈R to be detected H×W , where H and W represent the height and width of the image, respectively, and the candidate box set P∈R N×4 , where N represents the number of candidate boxes, the image is input into the convolutional neural network CNN, and the image feature F∈R is obtained h×w×c , where h, w, c represent the height, width and number of channels of the feature map respectively;
[0021] Add the image features F along the channel direction of CNN to obtain the feature attention map f mask ∈R h×w , the feature attention map f mask Restore to the size of the image and get the image attention map mask∈R H×W , and then convert the value of the image attention map mask to (0,1) after normalization to obtain the regularized attention map mask normed ∈R H×W , where the standardization process is shown in formula (1):
[0022]
[0023] Mask is a two-dimensional matrix, where min() is the function to get the minimum value and max() is the function to get the maximum value;
[0024] Calculate the regularized attention map mask normed The pixel mean of the regularized attention map mask normed The area where the pixel value is greater than the pixel mean is determined as the foreground attention map mask fore , the area smaller than the pixel mean is determined as the background part.
[0025] Preferably, the foreground relative confidence FRC of each candidate box is calculated based on the foreground attention map, and the foreground candidate boxes are screened out according to the FRC of each candidate box, including:
[0026] Given a candidate box p = x0, y0, x1, y1, in the candidate box area, first calculate the sum of the pixel values of the foreground attention map, and then divide it by the area of the candidate box to obtain the foreground relative confidence FRC of the candidate box p;
[0027]
[0028] Select a set number of candidate boxes whose FRC is above the mean as the foreground candidate box P fore ∈R M×4 , where M is the number of foreground candidate boxes.
[0029] Preferably, the step of constructing an instance space graph based on the foreground candidate frame, constructing a label semantic graph based on the label of the data set, performing graph matching on the instance space graph and the label semantic graph, and classifying each foreground candidate frame according to the graph matching result includes:
[0030] Take the feature f of the foreground candidate box as the node v of the graph o , the feature splicing between the foreground candidate boxes is used as the attribute of the edge, as shown in formula (3), to construct the instance space graph G o ;
[0031]
[0032] The semantics of the dataset is embedded in words to obtain the semantic feature vector w of each category, and the semantic feature vector is used as the node v of the label semantic graph l , the features of adjacent nodes are concatenated as the attributes of the edge, as shown in formula (4), to construct the label semantic graph G l ;
[0033]
[0034] f, w represent the feature vectors of the candidate box and the category label respectively, [x, y] refers to the concatenation function, which concatenates vectors x and y, v represents a node, and e represents an edge;
[0035] The instance space graph G o Each instance and label semantic graph G l Each category of is connected to form an instance-label matching graph G A , taking the concatenation of two nodes in the instance-label matching graph as the attribute of the edge;
[0036] The instance-label matching graph is input into a two-layer graph convolutional network, and the edge attributes of the instance-label matching graph are predicted. The predicted value of the edge represents the probability that the instance belongs to the category. The edge with the largest prediction score for each instance in the instance space graph is selected, and the category node corresponding to the edge is the category of the instance. The predicted scores of the edges between the instance and all category nodes are used as the classification prediction of the instance.
[0037] Preferably, the generating of a pseudo supervisory frame according to the classification results of each foreground candidate frame, merging the pseudo supervisory frame with a spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and using the pseudo instance label as the target detection result of the image to be detected, comprises:
[0038] Generate a pseudo supervisory frame according to the classification results of each foreground candidate frame, merge the pseudo supervisory frame with the spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and use the pseudo instance label as the target detection result of the image to be detected;
[0039] Select the candidate box P with the highest classification score in the base network top-score As pseudo-supervisory information, after classifying each foreground candidate frame, select the foreground candidate frame P whose score is greater than the mean of the category in each category. fore Also as pseudo supervision information, the pseudo supervision box P pseudo = {P top-score , P fore};
[0040] Then select the pseudo supervision box P pseudo The spatial neighbor box P pseudo-neighbor , the spatial neighbor box P pseudo-neighbo With the pseudo-supervision box P pseudo Merge, and use the merged result as a pseudo instance label, and use the pseudo instance label as the target detection result of the image to be subjected to target detection.
[0041] It can be seen from the technical solution provided by the above-mentioned embodiments of the present invention that the method of the present invention completes the screening of foreground candidate boxes through class-independent positioning, and then completes the classification of instances by constructing an instance graph and a label graph and performing graph matching on the two. This method separates the positioning and classification tasks, thereby achieving a two-way improvement in positioning and classification performance, thereby effectively improving the weakly supervised target detection performance of the image.
[0042] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0044] Figure 1 A processing flow chart of an image weakly supervised object detection method based on class-agnostic foreground candidate box generation and graph matching foreground candidate box classification provided in an embodiment of the present invention.
[0045] Figure 2 A class-agnostic foreground candidate frame generation process flow chart provided by an embodiment of the present invention.
[0046] Figure 3 A matching graph construction process for foreground candidate frame classification provided for the implementation of the present invention.
[0047] Figure 4 A flowchart of pseudo instance label generation process provided for the implementation of the present invention. DETAILED DESCRIPTION
[0048] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0049] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.
[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0051] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.
[0052] The embodiment of the present invention proposes a weakly supervised target detection method based on class-independent positioning and graph matching. The method completes the screening of foreground candidate frames through class-independent positioning, and then completes the classification of instances by constructing instance graphs and label graphs and performing graph matching on the instance graphs and label graphs. The method separates the positioning and classification tasks, thereby achieving a two-way improvement in positioning and classification performance. On the basis of the baseline model, the present invention divides the weakly supervised target detection task into three steps: the first is the generation of class-agnostic foreground candidate frames, the second is the classification of foreground candidate frames based on graph matching, and the third is the generation of pseudo-instance labels.
[0053] The processing flow chart of an image weakly supervised target detection method based on class-agnostic foreground candidate frame generation and graph matching foreground candidate frame classification provided by an embodiment of the present invention is as follows: Figure 1 As shown, the processing steps include the following:
[0054] Step S10: Generate a foreground attention map through CNN (Convolutional Neural Networks) based on the image to be detected.
[0055] Given an image I∈R to be detected H×W , where H and W represent the height and width of the image, respectively, and the candidate box set P∈R N×4 , where N represents the number of candidate boxes, the image is input into the CNN network to obtain the image feature F∈R h×w×c , where h, w, c represent the height, width and number of channels of the feature map respectively.
[0056] CNN has an inductive bias that focuses on objects, that is, a randomly initialized CNN pays more attention to objects. Using this inductive bias of CNN, the model can discover class-agnostic foreground areas. Figure 2 As shown, the processing includes the following:
[0057] By adding the image feature F along the channel direction, we can get the feature attention map f mask ∈R h×w , the attention map reflects the area that CNN focuses on. The location with a large feature attention map value is the part that CNN pays more attention to, which is most likely the foreground part of the image. Restore the feature attention map to the size of the image to obtain the attention map mask∈P on the image H ×W , and then after standardization, the value of the image attention map can be converted to (0,1) to obtain the regularized attention map mask normed ∈R H×W , where the standardization process is shown in formula (1):
[0058]
[0059] The mask here is a two-dimensional matrix, where min() is the function that takes the minimum value and max() is the function that takes the maximum value.
[0060] Get the regularized attention map mask normed After that, calculate the mask normed The pixel mean of the attention map is set to zero for the part smaller than the mean and larger than the mask. normed The area with the mean pixel value is roughly the foreground attention map mask. fore , smaller than mask normed The area with the average pixel value is the background part.
[0061] Step S20: Calculate the foreground relative confidence (FRC) of each candidate box based on the foreground attention map, and select the foreground candidate boxes according to the FRC of each candidate box.
[0062] After obtaining the foreground attention map, according to this foreground attention map mask fore Filter foreground candidate boxes from the candidate box set.
[0063] Here, the present invention introduces a new definition: FRC, which characterizes the probability of whether a candidate box is a foreground. Given a candidate box p = x0, y0, x1, y1, the sum of the pixel values of the foreground attention map in the candidate box area is first calculated, and then divided by the area of the candidate box to obtain the FRC of the candidate box p.
[0064] The specific calculation method is shown in formula (2):
[0065]
[0066] The FRC value is between 0 and 1. It is calculated based on the foreground attention map and can reflect whether the candidate box contains the foreground or the background, as well as the proportion of the foreground. The following is an analysis of the values of FRC in different ranges. If the FRC value of a candidate box is too small, it means that the candidate box basically contains the background, because the background mask fore The value is 0, or although the object is included, the candidate box is too large and still includes a large amount of background area. If the FRC value of a candidate box is too large, it means that it tightly includes part of the object, and the area of the box is relatively small. Because the generated foreground binary image cannot be very accurate, the candidate box with a large FRC only includes the local area of the object. When FRC is at the mean level, the visualization results show that the candidate box includes most of the object area. A set number (for example, 100) of candidate boxes with FRC above the mean are selected as the foreground candidate boxes P. fore ∈R M×4 , where M is the number of foreground candidate boxes. Here, the FRC values of the candidate boxes are sorted in ascending order, and the set number of candidate boxes above the mean are selected.
[0067] So far, class-agnostic foreground candidate boxes have been generated, which basically cover the foreground part of the image, so they are effective supervision information for the weakly supervised object detection task.
[0068] Step S30: construct an instance spatial graph based on the foreground candidate box, construct a label semantic graph based on the label of the data set, perform graph matching on the instance spatial graph and the label semantic graph, and classify each foreground candidate box according to the graph matching result.
[0069] After obtaining the foreground candidate frames, it is necessary to classify the instances enclosed by these candidate frames. Directly using the classification scores of the candidate frames to classify the foreground candidate frames will lose the frames containing the complete objects, because the candidate frames with high classification scores are generally more local frames. In order to classify the foreground candidate frames more accurately, the present invention constructs an instance space graph based on the foreground candidate frames, constructs a label semantic graph based on the labels of the data set, and performs graph matching on the instance space graph and the label semantic graph, thereby mining the category of each foreground candidate frame.
[0070] An example of a graph matching classification module construction provided by an embodiment of the present invention - the workflow of a label matching graph is as follows Figure 3 As shown, the processing includes the following:
[0071] Take the feature f of the foreground candidate box as the node v of the graph o , the feature splicing between the foreground candidate boxes is used as the attribute of the edge, as shown in formula (3), to construct the instance space graph Go The semantics of the dataset is embedded in words to obtain the semantic feature vector w of each category. The semantic feature vector is used as the node v of the label semantic graph. l , the features of adjacent nodes are concatenated as the attributes of the edge, as shown in formula (4), to construct the label semantic graph G l .
[0072] In order to further summarize the matching relationship between instances and labels, the present invention transforms the instance space graph G o Each instance and label semantic graph G l Each category of is connected to form an instance-label assignment graph G A , the concatenation of two nodes in the instance-label matching graph is used as the attribute of the edge. In this way, the complex graph matching problem is transformed into the problem of selecting appropriate edges in the matching graph.
[0073]
[0074]
[0075] In the formula, f and w represent the feature vectors of the candidate box and the category label respectively, [x, y] refers to the concatenation function, which concatenates the vectors x and y, v represents the node, and e represents the edge. The above instance-label matching graph is input into a two-layer graph convolutional network, and the edge attributes of the instance-label matching graph are predicted. The predicted value of the edge represents the probability that the instance belongs to the category. After the graph convolution, the multi-label classification loss is used to guide the graph convolution process. After predicting the edges of the instance-label matching graph, the highest prediction score of each category is selected to form the image-level prediction score predicted by the graph matching process, and the image-level prediction score is calculated with the true category label for cross entropy loss. Cross entropy loss is a common classification loss used for classification tasks.
[0076] Select the edge with the largest prediction score for each instance in the instance space graph, then the category node corresponding to the edge is the category of the instance, and the prediction scores of the edges between the instance and all category nodes are used as the classification prediction of the instance. So far, the classification task of the foreground candidate box has been completed.
[0077] Step S40, generating a pseudo supervisory frame according to the classification results of each foreground candidate frame, merging the pseudo supervisory frame with the spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and using the pseudo instance label as the target detection result of the image to be detected.
[0078] Generate pseudo instance labels for object detection models.
[0079] Figure 4The pseudo instance label generation process flow chart provided for the implementation of the present invention includes the following process: the basic network directly selects the candidate box P with the highest classification score top-score As pseudo-supervisory information. The above basic network consists of a CNN layer, ROIpooling, two fully connected layers, and two parallel fully connected layers. The outputs of the two parallel fully connected layers are normalized along two dimensions and then multiplied as the classification prediction of the image; the candidate boxes for classification are generated by selective search, that is, boxes of different sizes generated by sliding the candidate boxes.
[0080] After classifying each foreground candidate frame, select the foreground candidate frame P whose score is greater than the mean of the category in each category fore Also as pseudo supervision information, the pseudo supervision box P pseudo = {P top-score , P fore}. Then select the spatial neighbor box P of the pseudo-supervisory box pseudo-neighbor ,For each pseudo-supervisory box, select the candidate box whose intersection and union with it is greater than 0.5 from all candidate boxes as the neighboring box.
[0081] The spatial neighbor box P pseudo-neighbor With the pseudo-supervision box P pseudo Merge, and use the merged result as a pseudo instance label, and use the pseudo instance label as the target detection result of the image to be detected.
[0082] Because the foreground candidate boxes generated by the present invention can cover most areas of multiple instances, the local positioning and multi-instance loss problems of weakly supervised target detection can be effectively alleviated, thereby improving the performance of weakly supervised target detection.
[0083] In summary, compared with the existing weakly supervised target detection model, the embodiments of the present invention attempt for the first time to decompose weakly supervised target detection into a class-agnostic positioning task and a foreground classification task based on graph matching. Class-agnostic positioning can effectively alleviate the problem of weakly supervised target detection focusing on local salient areas and losing multiple instances because it can find as many and complete instances as possible. Graph matching can improve classification capabilities due to its powerful ability to mine hidden information. Finally, this method can effectively improve the weakly supervised target detection performance of images.
[0084] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.
[0085] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.
[0086] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0087] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A weakly supervised object detection method for images based on class-agnostic foreground mining, characterized in that: include: Generate a foreground attention map based on the image to be detected by the convolutional neural network CNN; Calculate the foreground relative confidence (FRC) of each candidate frame based on the foreground attention map, and select the foreground candidate frames according to the FRC of each candidate frame; constructing an instance space graph based on the foreground candidate frame, constructing a label semantic graph based on the label of the data set, performing graph matching on the instance space graph and the label semantic graph, and classifying each foreground candidate frame according to the graph matching result; Generate a pseudo supervisory frame according to the classification results of each foreground candidate frame, merge the pseudo supervisory frame with the spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and use the pseudo instance label as the target detection result of the image to be detected; The step of constructing an instance space graph based on the foreground candidate frame, constructing a label semantic graph based on the label of the data set, performing graph matching on the instance space graph and the label semantic graph, and classifying each foreground candidate frame according to the graph matching result includes: Take the feature f of the foreground candidate box as the node v of the graph o , the feature splicing between the foreground candidate boxes is used as the attribute of the edge, as shown in formula (3), to construct the instance space graph G o ; The semantics of the dataset is embedded in words to obtain the semantic feature vector w of each category, and the semantic feature vector is used as the node v of the label semantic graph l , the features of adjacent nodes are concatenated as the attributes of the edge, as shown in formula (4), to construct the label semantic graph G l ; f,w represent the feature vectors of candidate boxes and category labels respectively, [f i ,f j ] refers to the concatenation function, which converts the vector f i and f j Splicing, [w i ,w j ] refers to the concatenation function, which transforms the vector w i and w j Splicing, v represents node, e represents edge; The instance space graph G o Each instance of the label semantic graph G l Each category of is connected to form an instance-label matching graph G A , taking the concatenation of two nodes in the instance-label matching graph as the attribute of the edge; The instance-label matching graph is input into a two-layer graph convolutional network, and the edge attributes of the instance-label matching graph are predicted. The predicted value of the edge represents the probability that the instance belongs to the category. After predicting the edge of the instance-label matching graph, the highest prediction score of each category is selected to form the image-level prediction score predicted by the graph matching process. The image-level prediction score and the true category label are cross-entropy loss calculated, and the edge with the largest prediction score for each instance in the instance space graph is selected. The category node corresponding to the edge is the category of the instance, and the prediction score of the edge between the instance and all category nodes is used as the classification prediction of the instance.
2. The method according to claim 1, characterized in that The method of generating a foreground attention map based on the image to be detected by the convolutional neural network (CNN) includes: Given an image I∈R to be detected H×W , where H and W represent the height and width of the image, respectively, and the candidate box set P∈R N×4 , where N represents the number of candidate boxes, the image is input into the convolutional neural network CNN, and the image feature F∈R is obtained h ×w×c , where h, w, c represent the height, width and number of channels of the feature map respectively; Add the image features F along the channel direction of CNN to obtain the feature attention map f mask ∈R h×w , the feature attention map f mask Restore to the size of the image and get the image attention map mask∈R H×W , and then convert the value of the image attention map mask to (0,1) after normalization to obtain the regularized attention map mask normed ∈R H×W , where the standardization process is shown in formula (1): Mask is a two-dimensional matrix, where min() is the function to get the minimum value and max() is the function to get the maximum value; Calculate the regularized attention map mask normed The pixel mean of the regularized attention map mask normed The area where the pixel value is greater than the pixel mean is determined as the foreground attention map mask fore , the area smaller than the pixel mean is determined as the background part.
3. The method according to claim 2, characterized in that The foreground relative confidence FRC of each candidate box is calculated based on the foreground attention map, and the foreground candidate boxes are screened out according to the FRC of each candidate box, including: Given a candidate box p = x0, y0, x1, y1, in the candidate box area, first calculate the sum of the pixel values of the foreground attention map, and then divide it by the area of the candidate box to obtain the foreground relative confidence FRC of the candidate box p; Select a set number of candidate boxes whose FRC is above the mean as the foreground candidate box P fore ∈R M×4 , where M is the number of foreground candidate boxes.
4. The method according to claim 1, characterized in that: The method of generating a pseudo supervisory frame according to the classification results of each foreground candidate frame, merging the pseudo supervisory frame with the spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and using the pseudo instance label as the target detection result of the image to be detected, includes: Generate a pseudo supervisory frame according to the classification results of each foreground candidate frame, merge the pseudo supervisory frame with the spatial neighbor frame of the pseudo supervisory frame to obtain a pseudo instance label, and use the pseudo instance label as the target detection result of the image to be detected; Select the candidate box P with the highest classification score in the base network top―score As pseudo-supervisory information, after classifying each foreground candidate frame, select the foreground candidate frame P whose score is greater than the mean of the category in each category. fore Also as pseudo supervision information, the pseudo supervision box P pseudo = {P top―score ,P fore }; Then select the pseudo supervision box P pseudo The spatial neighbor box P pseudo―neighbor , the spatial neighbor box P pseudo―neighbor With the pseudo-supervision box P pseudo Merge, and use the merged result as a pseudo instance label, and use the pseudo instance label as the target detection result of the image to be subjected to target detection.