Target detection method based on graph structure and electronic device

Abstract

The application relates to a target detection method based on a graph structure and electronic equipment, and comprises the following steps: S1, acquiring a to-be-detected image; S2, constructing a target detection network based on a graph structure through an improved knowledge distillation method, wherein the improved knowledge distillation method is as follows: a student model is obtained by distilling a teacher model through a graph structure method, the teacher model is obtained by training a preset image data set, and the student model is a target detection network based on a graph structure; and S3, detecting a target in the to-be-detected image through the target detection network based on the graph structure. The application solves the problem that information of extracted feature knowledge is not rich in an existing knowledge distillation method, and the knowledge distillation method can effectively improve the performance of a student model in target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method and electronic device based on graph structure. Background Technology

[0002] With the development of computer technology, deep learning-based detection methods have become more accurate and faster. At the same time, powerful detectors also bring greater computational demands. Model compression methods have become another key research direction. Knowledge distillation is an effective method to reduce computational resource requirements by improving the capabilities of a smaller model. Typically, it involves transferring guiding knowledge from a pre-trained model (teacher) to a smaller model (student). However, most distillation methods focus on image classification tasks. Directly transferring classification distillation methods to object detection tasks remains a challenge. In object detection, due to the nature of the detector, the teacher and student models cannot predict the same object suggestions; the instance features of the teacher and student are inconsistent. Furthermore, multiple positive and negative instances will be predicted, with the number of negative instances far exceeding the number of positive instances. If all instances are simply selected for distillation, some complex instance information will be ignored, and the effectiveness of the student model will not be further improved. Therefore, a new selection mechanism is needed to guide the student model by selecting effective instances. Summary of the Invention

[0003] Therefore, the technical problem to be solved by the present invention is to overcome the problem that some complex instance information in the image is ignored when performing knowledge distillation on the teacher model in the prior art, and that the student model is not very effective.

[0004] To address the aforementioned technical problems, this invention provides a target detection method based on graph structure, comprising:

[0005] Step S1: Acquire the image to be detected;

[0006] Step S2: Construct a graph-based target detection network using an improved knowledge distillation method, wherein the improved knowledge distillation method is: distilling the teacher model using a graph structure method to obtain a student model, wherein the teacher model is trained using a preset image dataset, and the student model is a graph-based target detection network;

[0007] The graph structure method in step S2 specifically involves constructing a graph structure based on the instances detected in the teacher model, including:

[0008] Step S21: Construct similarity between instance features and construct pixel-level distance relationships between instance locations;

[0009] Step S22: Multiply the similarity between the instance features and the pixel-level distance relationship between the instance locations to obtain the instance weight;

[0010] Step S23: Filter out instances whose instance weights are higher than preset weights to form the graph structure;

[0011] Step S3: Detect the target in the image to be detected using the graph-based target detection network.

[0012] In one embodiment of the present invention, the graph structure in step S2 satisfies the formula: G = (N, E), where N and E represent the node set and edge set in the image during distillation, respectively, and the node set N includes region nodes N. r and class node N c Each of the class nodes N c , corresponding to instances in the image; each of the region nodes N r The corresponding instance location in the image; the edge set E includes region nodes N r Edge set between them, and class node N c Edge set between.

[0013] In one embodiment of the present invention, the similarity between instance features in step S21 is constructed using the following formula:

[0014] E c (V p V q ) = sim(V p V q )

[0015] in, sim represents similarity, V p V q The feature vectors represent different instances of the teacher model.

[0016] In one embodiment of the present invention, the pixel-level distance relationship between instance locations is constructed in step S21, and the formula is:

[0017] E r (V p V q = dist(Coordinate(V) p ),Coordinate(V q ))

[0018] in, dist represents the distance, Coordinate represents the coordinates of the instance's center point, and V p V qX1 and Y1 represent the feature vectors of different instances of the teacher model, respectively. p The x and y coordinates of the center point of the instance are represented by X2 and Y2, respectively. q The x and y coordinates of the center point of the represented instance.

[0019] In one embodiment of the present invention, when distilling the teacher model to obtain the student model in step S2, instances are also selected from the distilled images using an instance selection method to train the student model.

[0020] In one embodiment of the present invention, the formula for the instance selection method in step S2 is:

[0021]

[0022]

[0023] ALSI = NMS(P ALSI B ALSI )

[0024] Among them, P t and P s δ represents the probability scores of the teacher model and the student model. Pt P represents the standard deviation of the teacher model scores. ALSI P represents t and P s The average after subtraction, B t and B s B represents the corresponding suggestion box. ALSI The selected suggestion box is represented by α, which is a hyperparameter used to control the number of instances and remove instances with high uncertainty; NMS represents non-maximum suppression; and ALSI represents the instance selection result.

[0025] In one embodiment of the present invention, the overall loss formula for the student model in step S2 is: L = L D +λ f L F +λ g L G , where L D L represents the target detection loss; F L represents the loss between the output features of the teacher model and the output features of the student model in feature distillation (i.e., the loss between the backbone network output of the teacher model and the backbone network output of the student model in feature distillation); G The graph distillation loss is represented by λ, which consists of the node loss and edge loss in the graph structure. f , λ g Contribution to equilibrium distillation.

[0026] In one embodiment of the present invention, the distillation loss L is... G The formula is: Where λ1 and λ2 represent the penalty coefficients used for balancing in the distillation loss, L V L represents the node loss in the graph structure. E This represents the graph edge loss in the graph structure, where K represents the number of instances. The node representing the i-th instance of teacher model t, The node representing the i-th instance of student model s. This represents the edge between the i-th and j-th instances of the teacher model t. Let represent the edge between the i-th instance and the j-th instance of the student model s.

[0027] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described graph-based target detection method.

[0028] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the above-described graph-based target detection method are implemented.

[0029] The technical solution of the present invention has the following advantages compared with the prior art:

[0030] The graph-based target detection method described in this invention designs a selection mechanism based on an active learning strategy. By selecting effective instances to guide the student model, it solves the problem that simply selecting all instances for distillation will ignore some complex instance information and the effectiveness of the student model will not be further improved.

[0031] The graph-based target detection method described in this invention designs a unique structural graph to construct the complex relationships between instances in the model and the complex relationships between graph connections, thus solving the problem of insufficient information in the feature knowledge extracted during the knowledge distillation process. Attached Figure Description

[0032] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0033] Figure 1 This is a flowchart of the target detection method based on graph structure according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the network structure of the graph-based target detection method according to an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram illustrating the selection of active instances in the graph-based target detection method according to an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of graph structure generation for the graph-based target detection method according to an embodiment of the present invention;

[0037] Figure 5 This is a comparison chart of partial detection results from the GID method and the method of this invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0039] Example 1

[0040] Reference Figure 1 As shown, an embodiment of the present invention provides a target detection method based on graph structure, comprising the following steps:

[0041] Step S1: Acquire the image to be detected;

[0042] Step S2: Construct a graph-based target detection network using an improved knowledge distillation method, wherein the improved knowledge distillation method is: distilling the teacher model using a graph structure method to obtain a student model, wherein the teacher model is trained using a preset image dataset, and the student model is a graph-based target detection network;

[0043] The graph structure method in step S2 specifically involves constructing a graph structure based on the instances detected in the teacher model, including:

[0044] Step S21: Construct similarity between instance features and construct pixel-level distance relationships between instance locations;

[0045] Step S22: Multiply the similarity between the instance features and the pixel-level distance relationship between the instance locations to obtain the instance weight;

[0046] Step S23: Filter out instances whose instance weights are higher than preset weights to form the graph structure;

[0047] Step S3: Detect the target in the image to be detected using the graph-based target detection network.

[0048] In step S2, when the teacher model is distilled to obtain the student model, instances are selected from the distilled images using an instance selection method to train the student model.

[0049] This embodiment designs an instance selection method based on active learning for semantic information transfer in object detection tasks, ensuring that the most advantageous information is learned. During model training, the most meaningful instances are selected based on the active learning strategy. This embodiment uses a graph distillation structure to construct relationships, which can handle the lack of instance relationships. A common relationship is the similarity between instance features. In addition, a new relationship about pixel-level distance between instance locations is provided. The combination of these two relationships enhances knowledge transfer.

[0050] The following is a detailed description of this embodiment:

[0051] This invention proposes an instance selection method based on active learning, which adaptively selects the most distinctive instances between teachers and students for distillation. Furthermore, this method effectively improves the performance of modern detection frameworks with relation-based knowledge through structured graphs. Future experiments will be conducted under different detection frameworks. Related research indicates that a reasonable instance selection mechanism will be effective, providing insights for future distillation work.

[0052] In a graph-based target detection method disclosed in an embodiment of the present invention, such as Figure 2 Design an instance selection method, including:

[0053] First, one active learning strategy is to select a few of the most representative instances based on instance diversity. These instances are used for labeling and training to ensure the generalizability of the model. In the distillation task, the differences between predicted scores are calculated. Schemes with higher scores are selected and distilled using the corresponding features. The selected most representative instances will guide the student model toward better predictions.

[0054] Secondly, this method is based on instance uncertainty. In active learning, the model can quickly improve its performance by learning the labels of the definite instances retained after removing high-uncertainty instances. Therefore, instances with high uncertainty are not selected. This embodiment removes some suggestions represented by uncertain instances based on probability distributions. The specific formula is as follows:

[0055]

[0056]

[0057] ALSI = NMS(P ALSI B ALSI )

[0058] Among them, P t and P sThese are the probability scores for the teacher model and the student model; It is the standard deviation of the teacher model score; P ALSI It is P t and P s The average after subtraction; B t and B s The corresponding suggestion box; B ALSI The selected suggestion boxes are the ones that overlap because the selection mechanism creates a corresponding suggestion box for each pixel. Therefore, NMS (non-maximum suppression) is required. α is a hyperparameter that controls the number of uncertain instances, and then, based on P... ALSI The score selects the top K instances with the highest scores; ALSI represents the instance selection result.

[0059] In a graph-based target detection method disclosed in an embodiment of the present invention, such as Figure 3 In step S2, the method for designing a graph structure that enhances the relationships between instances includes:

[0060] Construct a graph G = (N, E), where N and E represent the set of nodes and the set of edges in the graph, respectively.

[0061] 1) Nodes: Two types of nodes are defined in N: region nodes N r For region R (R region is...) Figure 3 In the reg), class node N c For region C (region C is...) Figure 3 (cls in the structured graph). In the structured graph, each class node N c For each instance in the image, the vectorized features of that instance are represented. This embodiment constructs nodes based on the set of features selected by the activated instance selection module. Based on the difference between the proposal and the ground truth bounding box, they are assigned to either the foreground or background category. Unlike previous works, which selected all foreground and background instances or selected background instances proportionally, this approach neglects a reasonable instance selection rule. Whether foreground or background instances, the goal is to select the information most beneficial for distillation.

[0062] 2) Edges: For E, two sets of edges are defined between the nodes. First, for the first set of edges (E... r ), using a spatial map to encode the spatial relationships between regions (E r →r). Edges are defined based on positional relationships, such as "near or far," and by measuring the pixel-level distance between them. This embodiment uses Euclidean distance to define region edges:

[0063] E r (V p V q= dist(Coordinate(V) p ),Coordinate(V q ))

[0064]

[0065] Among them, V p V q This represents the feature vector of different instances of the teacher model, where dist represents the distance, Coordinate represents the coordinates of the instance's center point, and X1 and Y1 are respectively the feature vectors of V. p The x and y coordinates of the center point of the instance are represented by X2 and Y2, respectively. q The x and y coordinates of the center point of the represented instance.

[0066] The second set of edges is located in class (E) c The relationship between two instances forms an edge between two corresponding nodes, which is calculated using their similarity in the embedding space. This embodiment uses cosine similarity to define the edges of instances:

[0067] E c (V p V q ) = sim(V p V q )

[0068]

[0069] Where sim represents similarity, V p V q The feature vectors represent different instances of the teacher model.

[0070] The instance weight is obtained by multiplying the similarity between instance features and the pixel-level distance between instance locations:

[0071] E(V p V q ) = E c (V p V q )·E r (V p V q )

[0072] E(V p V q ) is a combination of two sets of edges. Finally, instances with weights higher than the preset weights are selected to form a graph structure.

[0073] It is easy to see that the addition of regional relationship edges provides a new supplementary relationship. The original instances only constructed the relationship between features, while this embodiment supplements the relationship between the pixel distances between instance boxes, which enhances the relationship between instances and makes the subsequent student model more reliable.

[0074] like Figure 4 As shown, cls in the input part represents class nodes, and reg in the input part represents region nodes. It is not difficult to see that the nodes located in... Figure 4 Between the input and output parts: class nodes are fully connected, and region nodes are also fully connected. In this embodiment, the similarity between instance features and the pixel-level distance between instance locations are multiplied to obtain the instance weights. Figure 4 The output is the final structure diagram.

[0075] Compared with existing technologies, the graph-based target detection method proposed in this invention emphasizes the selection of effective instances to guide the student model. It designs a unique structural graph to construct complex relationships between instances, between teacher and student features, and between graph connections. This solves the problems that simply selecting all instances for distillation will ignore some complex instance information, thus failing to further improve the effectiveness of the student model and resulting in insufficient information for general feature extraction.

[0076] When designing a distillation structure diagram with selected instances, the distillation loss is considered, and the distillation loss consists of multiple parts, including feature loss and graph loss.

[0077] 1) Backbone network feature loss: The feature-based distillation loss is as follows:

[0078] L F =L CE (y t ,y' s )

[0079] y'=f(y)

[0080] 2) Wherein, L CE Let y be the KL divergence loss function. t It is the feature vector output by the teacher backbone network, y s y is the feature vector output by the student backbone network, and f(y) is the mapping function to ensure consistent feature dimensions. Graph loss: Graph distillation loss L G Defined as the difference between the structure graphs of teachers and students, the loss L is calculated from the nodes in the graph structure. V Graph edge loss L in graph structure E The components are as follows:

[0081]

[0082] Where λ1 and λ2 represent the penalty coefficients for balancing the distillation losses in the figure. The node representing the i-th instance of teacher model t, Let represent the node of the i-th instance of student model s. This represents the edge between the i-th and j-th instances of the teacher model t. Let represent the edge between the i-th instance and the j-th instance of the student model s.

[0083] In the detection process, there are classification and bounding box lids, where the output log-matching uses the Kubbie divergence loss. The graph and head log-losses are combined into the detector loss, resulting in the student's overall training loss as follows:

[0084] L = L D +λ f L F +λ g L G

[0085] Among them, L D L represents the normal target detection loss. F L represents the loss between the output features of the teacher model and the output features of the student model in feature distillation (i.e., the loss between the backbone network outputs of the teacher model and the backbone network outputs of the student model in feature distillation). G The graph distillation loss is represented by λ, which consists of the node loss and edge loss in the graph structure. f , λ g This balances the contribution of distillation.

[0086] Furthermore, this embodiment also conducted experiments on the COCO dataset and verified the effectiveness of the graph-based object detection network (i.e., the student model).

[0087] Figure 5 The image below shows a visual comparison of the detection results of the GID method and the method of the present invention. The first row of images represents the results of the GID method, and the second row represents the detection results of the present invention. It is easy to see that the method of the present invention is superior to the GID method, indicating that the detection effectiveness of the present invention is higher.

[0088] Example 2

[0089] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph-based target detection method described in Embodiment 1.

[0090] Example 3

[0091] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the graph-based target detection method described in Embodiment 1.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0097] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A target detection method based on graph structure, characterized in that, include: Step S1: Acquire the image to be detected; Step S2: Construct a graph-based target detection network using an improved knowledge distillation method, wherein the improved knowledge distillation method is: to distill the teacher model using a graph structure method to obtain the student model, wherein the teacher model is trained using a preset image dataset, and the student model is a graph-based target detection network; The graph structure method in step S2 specifically involves constructing a graph structure based on the instances detected in the teacher model, including: Step S21: Construct similarity between instance features and construct pixel-level distance relationships between instance locations; In step S21, the similarity between instance features is constructed using the following formula: ； in, , Indicating similarity, , The feature vectors representing different instances of the teacher model; In step S21, the pixel-level distance relationship between instance locations is constructed using the following formula: ； in, , Indicates distance, Represents the coordinates of the instance's center point. , This represents the feature vectors of different instances of the teacher model. , They are The x and y coordinates of the center point of the representative instance. , They are The x and y coordinates of the center point of the represented instance; Step S22: Multiply the similarity between the instance features and the pixel-level distance relationship between the instance locations to obtain the instance weight; Step S23: Filter out instances whose instance weights are higher than preset weights to form the graph structure; Step S3: Detect the target in the image to be detected using the graph-based target detection network.

2. The target detection method based on graph structure according to claim 1, characterized in that: The graph structure in step S2 satisfies the following formula: ,in, and These represent the node set and edge set in the image during distillation, respectively. Including regional nodes and class nodes Each of the aforementioned class nodes , corresponding to instances in the image; each of the said region nodes The corresponding instance location in the image; the edge set Including regional nodes Edge sets and class nodes Edge set between.

3. The target detection method based on graph structure according to claim 1, characterized in that: In step S2, when the teacher model is distilled to obtain the student model, instances are selected from the distilled images using an instance selection method to train the student model.

4. The target detection method based on graph structure according to claim 3, characterized in that: The formula for the instance selection method in step S2 is: ； in, and Represents the probability scores for the teacher model and the student model. The standard deviation of the teacher model score is represented by the following: express and The average after subtraction and This indicates the corresponding suggestion box. The selected suggestion box This represents a hyperparameter used to control the number of instances and to delete instances with high uncertainty. Indicates non-maximum suppression; This indicates the result of instance selection.

5. The target detection method based on graph structure according to claim 1, characterized in that: The overall loss formula for the student model in step S2 is: ,in, Indicates the target detection loss; This represents the loss between the output features of the teacher model and the output features of the student model during feature distillation. The graph distillation loss is represented by the node loss and edge loss in the graph structure. , Contribution to equilibrium distillation.

6. The target detection method based on graph structure according to claim 5, characterized in that: The distillation loss is shown in the figure. The formula is: ,in, and This represents the penalty coefficient used to balance the distillation losses. This represents the node loss in the graph structure. This represents the graph edge loss in a graph structure. Indicates the number of instances. Teacher model The The nodes of each instance, Student model The The nodes of each instance, Teacher model The The first instance and the first Edges between instances, Student model The The first instance and the first Edges between instances.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the graph-based target detection method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the graph-based target detection method as described in any one of claims 1 to 6.

Images