A small target detection method based on multi-level relationship enhancement

Abstract

The present invention discloses a small target detection method based on multi-level relationship enhancement, which is intended to detect small defects on the appearance of a PCB circuit board, and belongs to the field of industrial quality inspection and machine vision. The present invention uses Faster RCNN as a benchmark model, including a feature extractor, a candidate frame generator, and an auxiliary detector, and the corresponding output features are image-level features, instance-level features, and category-level features, respectively; the present invention designs a set of image-level to instance-level contextual semantic relationship enhancement and instance-level to instance-level visual reasoning components, and fuses the outputs of the above components through a residual fusion module, and then transmits them to the detector to realize the detection of small defect targets on the surface of the input image data. The present invention enhances the characterization capability of the benchmark model for small defect targets, can accurately detect small defects on the appearance of PCB circuit boards, and improves the appearance quality of circuit boards and the production efficiency of enterprises.

Description

Technical Field

[0001] The invention belongs to the field of industrial quality inspection and machine vision, and specifically relates to a small target detection method based on multi-level relationship enhancement. Background Art

[0002] As the basic component of electronic products, PCB circuit boards play an important role in electrical connection, mechanical support and signal transmission. With the rapid development of packaging and testing technology, PCB circuit boards are gradually developing towards high integration and compact structure. However, this trend brings new challenges to the target detection of PCB circuit board appearance defects, namely: its surface defects are easier to produce but more difficult to be found. Therefore, it is urgent to invent a detection method suitable for minor defects in the appearance of PCB circuit boards.

[0003] The current mainstream target detection methods perform well in achieving large-size target detection, but when migrating to small target detection scenarios, the performance drops sharply. At present, some studies have adopted multi-scale feature fusion technology and small anchor boxes to detect small target defects. Although some progress has been made compared with mainstream target detection methods, there are still problems such as excessive computing resources and complex network structure, which cannot meet the actual working conditions. In addition, it should be noted that the appearance defects of PCB circuit boards only appear on specific components. For example, short circuit defects only occur between two circuit lines. However, most of the current mainstream defect target detection models use Faster RCNN as the benchmark model. Such detection models ignore the contextual semantic information of the candidate box (such as the circuit situation around the short circuit defect candidate box) and the correlation between the candidate boxes (such as the relationship between the pinhole defect candidate box and the specific component candidate box). Such information is crucial for the detection of weak defects. Summary of the invention

[0004] The present invention provides a method for detecting small targets based on multi-level relationship enhancement. The present invention aims to utilize the relationship between different-level features such as image level, instance level, and category level of Faster RCNN to enrich the contextual semantic information of instance-level features and the association information between instance-level features, thereby enhancing the ability to characterize small defects in the appearance of PCB circuit boards, thereby achieving accurate positioning and identification of small defects in the appearance of PCB circuit boards, greatly improving the appearance quality of PCB circuit boards and the production efficiency of enterprises. This is to solve the problems in the prior art that the mainstream Faster RCNN target detection model is difficult to accurately detect small defects in the appearance of PCB circuit boards, and the detection model ignores the contextual semantic information of candidate frames and the association relationship between candidate frames.

[0005] The present invention provides a small target detection method based on multi-level relationship enhancement, comprising the following steps:

[0006] Step S1: Acquire PCB circuit board defect image data.

[0007] Step S2: Use the Faster RCNN target detection model as a benchmark framework, which includes three modules: feature extractor, candidate box generator, and auxiliary detector. Input the PCB circuit board defect image data obtained in step S1 into the feature extractor of the benchmark framework to obtain image-level features. , image-level features Input candidate box generator to get instance-level features , instance-level features Input auxiliary detector to obtain category-level features .

[0008] Step S3: Construct an image-level to instance-level contextual semantic relationship enhancement component, and transform the image-level features obtained in step S2 into , instance-level features As the input of the contextual semantic relationship enhancement component, the instance-level features containing contextual semantic information are obtained .

[0009] Step S4: Construct an instance-level to instance-level visual reasoning component to transform the instance-level features obtained in step S2 , category-level features As the input of the visual reasoning component, the correlation features between instance-level features are obtained .

[0010] Step S5: Construct a residual fusion module to combine the instance-level features containing contextual semantic information obtained in step S3 The correlation features between the instance-level features obtained in step S4 After inputting the residual fusion module, new instance-level features are obtained .

[0011] Step S6: The new instance-level features obtained in step S5 The data is sent to the detector to obtain the test results, thus realizing the preparation for the detection of small defect targets on the surface of the input image data.

[0012] Furthermore, image-level features The dimension is [ m 1 × n 1 × c 1 ] , instance-level features The dimension is [ l 2 × m 2 × n 2 × c 2 ] , category-level features The dimension is [ l 3 ] ; Among them, image-level features First Dimension and the second dimension Represent image-level features Width and height , image-level features The third dimension Represents image-level features Number of channels ; Instance-level features The first dimension Represents the number of candidate boxes generated by the candidate box generator, instance-level features The second dimension and the third dimension Represent instance-level features Width and height , instance-level features The fourth dimension Represents instance-level features Number of channels ; Category-level features Dimensions Represents the total number of classification results of candidate boxes.

[0013] Further, step S3 includes the following steps:

[0014] Step S31: Instance-level features As a query vector , the image-level features As key vectors Sum value vector .

[0015] Step S32: Query vector Each channel of is globally averaged pooled along the second and third dimensions , get the features after global average pooling .

[0016] Step S33: The key vector Sum value vector Reset the dimension to get the key vector after dimension reset Sum value vector .

[0017] Step S34: Pool the features after global average pooling The query vector features are obtained by feature mapping of the first fully connected layer. , the key vector after resizing Sum value vector The key vector features are obtained by the feature mapping of the second fully connected layer and the third fully connected layer respectively. Sum value vector feature .

[0018] Step S35: Transform the key vector feature Perform matrix transposition operation to obtain the key vector features after transposition , the query vector feature and the transposed key vector features After batch normalization operation After matrix multiplication, the enhanced features are obtained after the Softmax activation layer. , the specific formula is:

[0019]

[0020] in, represents the activation function, Represents a normalization operation.

[0021] Step S36: Set the value vector feature and Enhanced Features Perform matrix multiplication, and then pass through the fourth fully connected layer to obtain the bias term containing contextual semantic information .

[0022] Step S37: Set the offset term The broadcasting mechanism in the NumPy library is used to expand the dimension and obtain the broadcast bias term of the dimension expansion. , the broadcast bias term And the instance-level features obtained in step S2 Add channel by channel to obtain instance-level features containing contextual semantic information .

[0023] Furthermore, the features after global average pooling The feature dimension is [ l 2 × c 2 ] , where each channel is globally averaged pooled The formula is:

[0024]

[0025] in, is the feature after global average pooling No. The value of the channel, c ∈ [ 0 , c 2 ] , The input feature map is at the height coordinate , the width coordinate is Location Place The value of each channel.

[0026] Further, step S4 includes the following steps:

[0027] Step S41: Instance-level features Flattened to dimensions [ l 2 × m '] Instance Flatten Feature ,in .

[0028] Step S42: Construct feature encoder , where the feature encoder It consists of two fully connected layers in series, and each fully connected layer is followed by a ReLU activation layer.

[0029] Step S43: Flatten the instance features Input to feature encoder Get the latent vector .

[0030] Step S44: Substituting latent vector and the latent vector Transpose Multiply them together to get the adjacency matrix .

[0031] Step S45: The adjacency matrix The numerical value of each row of elements is ranked first The values ​​are retained and the values ​​of the remaining elements are set to 0 to obtain a sparse adjacency matrix .

[0032] Step S46: Obtain each instance-level feature through a soft mapping function Visual embedding features corresponding to category-level features ; Instance-level features The corresponding category-level features are obtained through the soft mapping function to form the visual embedding matrix .

[0033] Step S47: Obtain the network layer weight matrix of the last layer of the auxiliary detector .

[0034] Step S48: embedding the visual matrix and the network layer weight matrix Matrix multiplication to obtain graph input .

[0035] Step S49: Use the PyTorch Geometric library to build a graph convolutional neural network ,in Indicates shared nodes, here refer to Candidate boxes, Represents the edge connection relationship between each node; the sparse adjacency matrix obtained in step S45 is As a graph convolutional neural network The edge connection relationship , input the graph obtained in step S48 into As a graph convolutional neural network of The category information of nodes is obtained through graph convolutional neural network Get graph association features .

[0036] Step S410: The Input a fully connected layer and ReLU activation function in sequence to obtain the correlation features between instance-level features .

[0037] Further, step S5 includes the following steps:

[0038] Step S51: The instance-level features containing contextual semantic information obtained in step S3 are Along the dimension [ m 2 × n 2 × c 2 ] Flatten to get the contextual semantic information instance flattened features ; And flatten the contextual semantic information instance features Input a fully connected layer with an output of 1024 to obtain the instance features of the new contextual semantic information .

[0039] Step S52: Associate the instance-level features obtained in step S4 with the And step S51 obtains the new context semantic information instance feature Splicing along the second dimension to obtain splicing features .

[0040] Step S53: The splicing features obtained in step S52 After passing through the fully connected layer with an output of 1024, the fully connected features are obtained .

[0041] Step S54: The new context semantic information instance feature obtained in step S51 And step S53 to obtain the fully connected features Add along the second dimension to get new instance-level features .

[0042] Compared with the prior art, the present invention has the following beneficial technical effects: the present invention adopts the mainstream Faster RCNN as the benchmark model, introduces a set of contextual semantic enhancement of candidate frames and visual reasoning components between candidate frames, and embeds the components into the mainstream benchmark model, further improving the benchmark model's ability to characterize small defects. The present invention can accurately locate and identify small defects in the appearance of PCB circuit boards, reduce the missed detection rate of defects, and greatly improve the appearance quality of PCB circuit boards and the production efficiency of enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0044] Figure 1 The figure is a flowchart of the overall framework of a small target detection method based on multi-level relationship enhancement according to an embodiment of the present invention.

[0045] Figure 2 This is a flow chart of a contextual semantic relationship enhancement component according to an embodiment of the present invention.

[0046] Figure 3 Flowchart of the visual reasoning component of an embodiment of the present invention.

[0047] Figure 4 This is a flow chart of the residual fusion module according to an embodiment of the present invention.

[0048] Figure 5 This is a diagram showing the effect of detecting small defects on the PCB appearance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The present invention proposes a small target detection method based on multi-level relationship enhancement. The method adopts the mainstream Faster RCNN as the benchmark model, introduces the context semantic enhancement component of the candidate box, the visual reasoning component between the candidate boxes and the residual module, and embeds the above components and modules into the benchmark model. The overall process is as follows Figure 1 As shown, the method comprises the following steps:

[0051] Step S1: Acquire PCB circuit board defect image data.

[0052] Step S2: The Faster RCNN target detection model is used as the benchmark framework of the present invention. The benchmark framework includes three modules: feature extractor, candidate box generator, and auxiliary detector. The output features corresponding to the three modules are image-level features, instance-level features, and category-level features, respectively.

[0053] Step S3: Construct an image-level to instance-level contextual semantic relationship enhancement component, and use the output of this component as a bias term and add it channel by channel to the instance-level features to obtain instance-level features containing contextual semantic information.

[0054] Step S4: Design an instance-level to instance-level visual reasoning component to obtain the correlation between each instance-level feature.

[0055] Step S5: Design a residual fusion module to fuse instance-level features containing contextual semantic information and the correlation relationship features between instance-level features to obtain new instance-level features.

[0056] Step S6: The new instance-level features are input into the detector to obtain the localization and classification results of minor defects in the appearance of the PCB circuit board.

[0057] The specific construction steps of a small target detection method based on multi-level relationship enhancement in an embodiment of the present invention are as follows:

[0058] Step S1: Acquire PCB circuit board defect image data.

[0059] Step S2: Use the Faster RCNN target detection model as a benchmark framework, which includes three modules: feature extractor, candidate box generator, and auxiliary detector. Input the PCB circuit board defect image data obtained in step S1 into the feature extractor of the benchmark framework to obtain image-level features. , image-level features Input candidate box generator to get instance-level features , instance-level features Input auxiliary detector to obtain category-level features Category-level features is the classification result, and the classification result is one of the classification categories, and the classification categories include open circuit, short circuit, rat bite, burr, pinhole, pseudo copper, and normal sample.

[0060] Image-level features The dimension is [ m 1 × n 1 × c 1 ] , instance-level features The dimension is [ l 2 × m 2 × n 2 × c 2 ] , category-level features The dimension is [ l 3 ] ; Among them, image-level features First Dimension and the second dimension Represent image-level features Width and height , image-level features The third dimension Represents image-level features Number of channels ; Instance-level features The first dimension Represents the number of candidate boxes generated by the candidate box generator, instance-level features The second dimension and the third dimension Represent instance-level features Width and height , instance-level features The fourth dimension Represents instance-level features Number of channels ; Category-level features Dimensions Represents the total number of classification results of candidate boxes. In this embodiment, there are candidate boxes, each of which generates a corresponding classification result, so the total number of classification results of candidate boxes is Exactly equal to the number of candidate boxes ; Among them, the number of channels , It can be reasonably set to 1024 or 512 or other parameters as needed. In this embodiment, the number of channels , Set to 1024; in this embodiment and The value of is set to 512. and The value of can be set to other reasonable parameters as needed; , The parameter value is less than , Parameter value of .

[0061] Step S3: Construct an image-level to instance-level contextual semantic relationship enhancement component, and transform the image-level features obtained in step S2 into , instance-level features As the input of the contextual semantic relationship enhancement component, the instance-level features containing contextual semantic information are obtained , instance-level features containing contextual semantic information Dimensions and instance-level features The dimensions are the same.

[0062] Specifically, if Figure 2 As shown, step S3 includes the following steps:

[0063] Step S31: Instance-level features As a query vector , the image-level features As key vectors Sum value vector .

[0064] Step S32: Query vector Each channel along the second dimension and the third dimension Perform global average pooling , get the features after global average pooling , the feature after global average pooling The feature dimension is [ l 2 × c 2 ] , where each channel is globally averaged pooled The formula is:

[0065]

[0066] in, is the feature after global average pooling No. The value of the channel, c ∈ [ 0 , c 2 ] , The input feature map is at the height coordinate , the width coordinate is Location Place The value of each channel.

[0067] Step S33: The key vector Sum value vector Reset the dimension to get the key vector after dimension reset Sum value vector , that is: the key vector Sum value vector The first dimension and the second dimension Multiply to get the key vector after dimension resetting Sum value vector , the key vector after re-dimensioning Sum value vector The dimensions are expressed as [ m 1 n 1 × c 1 ] .

[0068] Step S34: Pool the features after global average pooling The query vector features are obtained by feature mapping of the first fully connected layer. , the key vector after resizing Sum value vector The key vector features are obtained by the feature mapping of the second fully connected layer and the third fully connected layer respectively. Sum value vector feature , where the outputs of the first fully connected layer, the second fully connected layer, and the third fully connected layer are all 512, and the query vector feature , key vector features , value vector features The dimensions are [ l 2 × 512 ] , [ m 1 n 1 × 512 ] , [ m 1 n 1 × 512 ] .

[0069] Step S35: Transform the key vector feature Perform matrix transposition operation to obtain the key vector features after transposition , the query vector feature and the transposed key vector features After batch normalization operation After matrix multiplication, the enhanced features are obtained after the Softmax activation layer. , the specific formula is:

[0070]

[0071] in, represents the activation function, Represents a normalization operation; The dimension is [ l 2 × m 1 n 1 ] .

[0072] Step S36: Set the value vector feature and Enhanced Features Perform matrix multiplication, and then pass through the fourth fully connected layer to obtain the bias term containing contextual semantic information , where the output of the fourth fully connected layer is 1024 and the bias term The feature dimension is [ l 2 × 1024 ] .

[0073] Step S37: Set the offset term The broadcasting mechanism in the NumPy library is used to expand the dimension and obtain the broadcast bias term of the dimension expansion. , broadcast bias The dimension is [ l 2 × 1 × 1 × 1024 ] , the broadcast bias term And the instance-level features obtained in step S2 Add channel by channel to obtain instance-level features containing contextual semantic information , instance-level features containing contextual semantic information The dimension is [ l 2 × m 2 × n 2 × c 2 ] .

[0074] Step S4: Construct an instance-level to instance-level visual reasoning component to transform the instance-level features obtained in step S2 , the category-level features obtained in step S2 As the input of the visual reasoning component, the correlation features between instance-level features are obtained .

[0075] Specifically, Figure 3 As shown, step S4 includes the following steps:

[0076] Step S41: Instance-level features Flattened to dimensions [ l 2 × m '] Instance Flatten Feature ,in , .

[0077] Step S42: Construct feature encoder , where the feature encoder It consists of two fully connected layers in series, and each fully connected layer is followed by a ReLU activation layer. Both fully connected layers have an output of 512.

[0078] Step S43: Flatten the instance features Input to feature encoder Get the latent vector , the latent vector The feature dimension is [ l 2 × l ] , Represents the latent vector The feature length of each candidate box in this embodiment The value is 2048;

[0079] Step S44: Substituting latent vector and the latent vector Transpose Multiply them together to get the adjacency matrix , the adjacency matrix The feature dimension is [ l 2 × l 2 ] .

[0080] Step S45: Get the adjacency matrix Before each line values, and obtain the sparse adjacency matrix , sparse adjacency matrix The feature dimension is [ l 2 × t ] In this embodiment The value is 32.

[0081] Step S46: Through the soft mapping function Get each instance-level feature Visual embedding features corresponding to categories , where the soft mapping function expression is:

[0082]

[0083] in, Representing instance-level features The type-level features obtained by the auxiliary detector in step S2 Predicted to be The probability of the categories, Indicates the number of classification categories. In this embodiment . Instance-level features Visual embedding features obtained through soft mapping function Constructing the visual embedding matrix , the visual embedding matrix The dimension is .

[0084] Step S47: Obtain the network layer weight matrix of the last layer of the auxiliary detector , the network layer weight matrix The dimension is , Represents visual embedding features In this embodiment, .

[0085] Step S48: embedding the visual matrix and the network layer weight matrix Matrix multiplication to obtain graph input , graph input The dimension is .

[0086] Step S49: Use the PyTorch Geometric library to build a graph convolutional neural network ,in Indicates shared nodes, here refer to Candidate boxes, Represents the edge connection relationship between each node; the sparse adjacency matrix obtained in step S45 is As a graph convolutional neural network The edge connection relationship , input the graph obtained in step S48 into As a graph convolutional neural network of The category information of nodes is obtained through graph convolutional neural network Get graph association features , graph association features The feature dimension is [ l 2 × 512 ] .

[0087] Step S410: The Input a fully connected layer and ReLU activation function in sequence to obtain the correlation features between instance-level features , where the output dimension of the fully connected layer is 1024, and the correlation features between instance-level features The feature dimension is [ l 2 × 1024 ] .

[0088] Step S5: Construct a residual fusion module to combine the instance-level features containing contextual semantic information obtained in step S3 The correlation features between the instance-level features obtained in step S4 After inputting the residual fusion module, new instance-level features are obtained .

[0089] Specifically, Figure 4 As shown, step S5 includes the following steps:

[0090] Step S51: Instance-level features containing contextual semantic information obtained in step S3 The dimension is [ l 2 × m 2 × n 2 × c 2 ] , the instance-level features containing contextual semantic information Along the dimension [ m 2 × n 2 × c 2 ] Flatten to get the contextual semantic information instance flattened features , contextual semantic information instance flattening features The feature dimension is [ l 2 × m '] ,in Calculated as follows: ; And flatten the contextual semantic information instance features Input a fully connected layer with an output of 1024 to obtain the instance features of the new contextual semantic information , new contextual semantic information instance features The dimension is [ l 2 × 1024 ] .

[0091] Step S52: Associate the instance-level features obtained in step S4 with the And step S51 obtains the new context semantic information instance feature Splicing along the second dimension to obtain splicing features , where the splicing feature The dimension is [ l 2 × 2048 ] .

[0092] Step S53: The splicing features obtained in step S52 After passing through the fully connected layer with an output of 1024, the fully connected features are obtained , where the fully connected features The dimension is [ l 2 × 1024 ] .

[0093] Step S54: The new context semantic information instance feature obtained in step S51 And step S53 to obtain the fully connected features Add along the second dimension to get new instance-level features , new instance-level features The dimension is [ l 2 × 1024 ] .

[0094] Step S6: The new instance-level features obtained in step S5 The image data is sent to the detector to obtain the test results, which can locate and identify the small defects on the surface of the input image data. The test results are as follows: Figure 5 As shown, the method of the present invention can accurately detect small defects on PCB circuit boards, such as open circuits, short circuits, mouse bites, pinholes, burrs and other defects, with an identification accuracy rate of more than 99%.

[0095] The present invention provides a PCB circuit board appearance weak defect target detection algorithm based on multi-level relationship enhancement. The above-mentioned open source methods, image processing tools, parameter values, etc. are all to help readers understand the principles of the present invention. The protection scope of the present invention is not limited to such special statements and embodiments. Ordinary technicians in this field can also make improvements and deformations that do not deviate from the principles of the present invention, and these improvements and deformations still belong to the protection scope of the present invention.

Claims

1. A small target detection method based on multi-level relationship enhancement, characterized in that: The following steps are involved: Step S1: Acquire PCB circuit board defect image data; Step S2: Use the Faster RCNN target detection model as a benchmark framework, which includes three modules: feature extractor, candidate box generator, and auxiliary detector. Input the PCB circuit board defect image data obtained in step S1 into the feature extractor of the benchmark framework to obtain image-level features. , image-level features Input candidate box generator to get instance-level features , instance-level features Input auxiliary detector to obtain category-level features ; Step S3: Construct an image-level to instance-level contextual semantic relationship enhancement component, and transform the image-level features obtained in step S2 into , instance-level features As the input of the contextual semantic relationship enhancement component, the instance-level features containing contextual semantic information are obtained ; Step S4: Construct an instance-level to instance-level visual reasoning component to transform the instance-level features obtained in step S2 , category-level features As the input of the visual reasoning component, the correlation features between instance-level features are obtained ; Step S5: Construct a residual fusion module to combine the instance-level features containing contextual semantic information obtained in step S3 The correlation features between the instance-level features obtained in step S4 After inputting the residual fusion module, new instance-level features are obtained ; Step S6: The new instance-level features obtained in step S5 The data is sent to the detector to accurately detect small defect targets on the surface of the input image data.

2. The method for detecting small and weak targets based on multi-level relationship enhancement according to claim 1, characterized in that: Image-level features The dimension is , instance-level features The dimension is , category-level features The dimension is ; Among them, image-level features First Dimension and the second dimension Represent image-level features Width and height , image-level features The third dimension Represents image-level features Number of channels ; Instance-level features The first dimension Represents the number of candidate boxes generated by the candidate box generator, instance-level features The second dimension and the third dimension Represent instance-level features Width and height , instance-level features The fourth dimension Represents instance-level features Number of channels ; Category-level features Dimensions Represents the total number of classification results of candidate boxes.

3. The method for detecting small and weak targets based on multi-level relationship enhancement according to claim 2, characterized in that: Step S3 includes the following steps: Step S31: Instance-level features As a query vector , the image-level features As key vectors Sum value vector ; Step S32: Query vector Each channel of is globally averaged pooled along the second and third dimensions , get the features after global average pooling ; Step S33: The key vector Sum value vector Reset the dimension to get the key vector after dimension reset Sum value vector ; Step S34: Pool the features after global average pooling The query vector features are obtained by feature mapping of the first fully connected layer. , the key vector after resizing Sum value vector The key vector features are obtained by the feature mapping of the second fully connected layer and the third fully connected layer respectively. Sum value vector feature ; Step S35: Feature the key vector Perform matrix transposition operation to obtain the key vector features after transposition , the query vector feature and the transposed key vector features After batch normalization operation After matrix multiplication, the enhanced features are obtained after the Softmax activation layer. , the specific formula is: in, represents the activation function, Represents a normalization operation; Step S36: Set the value vector feature and Enhanced Features Perform matrix multiplication, and then pass through the fourth fully connected layer to obtain the bias term containing contextual semantic information ; Step S37: Set the offset term The broadcasting mechanism in the NumPy library is used to expand the dimension and obtain the broadcast bias term of the dimension expansion. , the broadcast bias term And the instance-level features obtained in step S2 Add channel by channel to obtain instance-level features containing contextual semantic information .

4. The method for detecting small and weak targets based on multi-level relationship enhancement according to claim 3, characterized in that: Features after global average pooling The feature dimension is , where each channel is globally averaged pooled The formula is: in, is the feature after global average pooling No. The value of the channel, , The input feature map is at the height coordinate , the width coordinate is The location of The value of each channel.

5. The method for detecting small and weak targets based on multi-level relationship enhancement according to claim 4, characterized in that: Step S4 includes the following steps: Step S41: Instance-level features Flattened to dimensions Instance Flatten Feature ,in ; Step S42: Construct feature encoder , where the feature encoder It consists of two fully connected layers in series, and each fully connected layer is followed by a ReLU activation layer; Step S43: Flatten the instance features Input to feature encoder Get the latent vector ; Step S44: Substituting latent vector and the latent vector Transpose Multiply them together to get the adjacency matrix ; Step S45: The adjacency matrix The numerical value of each row of elements is ranked first The values ​​are retained and the values ​​of the remaining elements are set to 0 to obtain a sparse adjacency matrix ; Step S46: Obtain each instance-level feature through a soft mapping function Visual embedding features corresponding to category-level features ; Instance-level features The corresponding category-level features are obtained through the soft mapping function to form the visual embedding matrix ; Step S47: Obtain the network layer weight matrix of the last layer of the auxiliary detector ; Step S48: embedding the visual matrix and the network layer weight matrix Matrix multiplication to obtain graph input ; Step S49: Use the PyTorch Geometric library to build a graph convolutional neural network ,in Indicates shared nodes, here refer to Candidate boxes, Represents the edge connection relationship between each node; the sparse adjacency matrix obtained in step S45 is As a graph convolutional neural network The edge connection relationship , input the graph obtained in step S48 into As a graph convolutional neural network of The category information of nodes is obtained through graph convolutional neural network Get graph association features ; Step S410: The Input a fully connected layer and ReLU activation function in sequence to obtain the correlation features between instance-level features .

6. The method for detecting small and weak targets based on multi-level relationship enhancement according to claim 5, characterized in that: Step S5 includes the following steps: Step S51: The instance-level features containing contextual semantic information obtained in step S3 are Along the dimension Flatten to get the contextual semantic information instance flattened features ; And flatten the contextual semantic information instance features Input a fully connected layer with an output of 1024 to obtain the instance features of the new contextual semantic information ; Step S52: Associate the instance-level features obtained in step S4 with the And step S51 obtains the new context semantic information instance feature Splicing along the second dimension to obtain splicing features ; Step S53: The splicing features obtained in step S52 After passing through the fully connected layer with an output of 1024, the fully connected features are obtained ; Step S54: The new context semantic information instance feature obtained in step S51 And step S53 to obtain the fully connected features Add along the second dimension to get new instance-level features .

Images