A target detection method and system for distinguishing foreground from background

Abstract

The present application discloses a target detection method and system for distinguishing foreground from background, wherein the method steps include: extracting a foreground feature map and a background feature map of a target to be detected based on multi-scale feature fusion; enhancing the foreground feature map and the background feature map to obtain a first foreground feature map and a first background feature map; generating a prediction frame based on the first foreground feature map; calculating a foreground feature vector using the prediction frame and the first foreground feature map; calculating a background feature vector using the first background feature map; calculating a true feature vector using the true frame and the first foreground feature map; constructing a contrast loss function, a foreground saliency guided loss function and a prediction frame loss function based on the foreground feature vector, the background feature vector, the true feature vector and the prediction frame; combining the contrast loss function, the foreground saliency guided loss function and the prediction frame loss function to improve the discrimination between the foreground and the background of the target and complete target detection.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a target detection method and system for distinguishing foreground from background. Background Art

[0002] With the continuous development of deep learning, object detection has evolved from traditional object detection algorithms that require manual feature extraction to deep learning-based object detection algorithms that can autonomously learn image features. The core of deep learning-based object detection is to identify objects of interest in images or videos and detect their position and size. The accuracy of object recognition directly affects the positioning accuracy of the predicted box. The complexity and variability of background features, as well as the uneven distribution of background and foreground object samples, are key factors affecting foreground object recognition. To improve the network model's ability to recognize objects of interest, the target is treated as the foreground and distinguished from the background in the image or video. This allows foreground-background feature comparison to improve object recognition and provide basic information for predicting box positioning.

[0003] Currently, most enhanced object detection networks achieve their ability to distinguish foreground from background by adding constraints to the network model. Among the existing technologies:

[0004] 1) Separate the foreground and background through local distillation, and then force the student model to learn the foreground features learned by the teacher model to improve the student model's foreground target recognition ability in the target detection task.

[0005] While this approach improves the network's object recognition capabilities by allowing the student model to learn only foreground features, the use of binary masks in local distillation forcibly removes background information, weakening the distinction between foreground and background. Although global distillation is subsequently used to supplement the information lost by local distillation, the importance of foreground information is not emphasized during this process. This causes the network model to prioritize foreground and background information equally during global distillation, which in turn affects the network's object recognition capabilities.

[0006] 2) By calculating the Euclidean distance difference between the actual grayscale value of the image and the global average grayscale value, the foreground pixels are highlighted, the background pixels are suppressed, the contrast between the foreground and background is increased, and the interference of the background on subsequent target detection is reduced.

[0007] However, the distribution of foreground and background in the image is unbalanced, resulting in the dominance of background samples during training. This method does not further highlight the importance of a small amount of foreground information, which may cause the network model to have poor learning effect on more difficult foreground samples. Summary of the Invention

[0008] In order to solve the technical problems in the above background, this application provides a deep learning target detection network model under foreground-background feature contrast learning, aiming to solve the problems of poor foreground target recognition ability of existing deep learning network models caused by complex and changeable background features and the number of samples far exceeding target samples, and effectively improve the target detection accuracy of deep learning network models.

[0009] To achieve the above objectives, the present application provides a method for detecting an object to distinguish between foreground and background, comprising the following steps:

[0010] Based on multi-scale feature fusion, the foreground feature map and background feature map of the target to be detected are extracted;

[0011] The foreground feature map and the background feature map are enhanced using an attention module to obtain a first foreground feature map and a first background feature map;

[0012] Generate a prediction box based on the first foreground feature map;

[0013] Calculating a foreground feature vector using the predicted frame and the first foreground feature map; calculating a background feature vector using the first background feature map; and calculating a true feature vector using the true frame and the first foreground feature map;

[0014] Constructing a contrast loss function, a foreground saliency guided loss function, and a prediction box loss function based on the foreground feature vector, the background feature vector, the true feature vector, and the prediction box;

[0015] The contrast loss function, the foreground saliency guided loss function and the prediction box loss function are combined to improve the distinction between the foreground and background of the target and complete target detection.

[0016] Preferably, the method for extracting the foreground feature map and the background feature map includes: first using a feature extraction network to extract features of the target to be detected; then using a multi-scale feature fusion network to perform multi-scale feature fusion on the extracted features to obtain the foreground feature map and the background feature map.

[0017] Preferably, the method for enhancing the foreground feature map and the background feature map includes: designing an interactive reverse attention module, using background attention to reversely solve the foreground feature map weight, and simultaneously using foreground attention to reversely solve the background feature map weight to obtain the first foreground feature map and the first background feature map; the formula for calculating the foreground feature map weight and the background feature map weight is as follows:

[0018]

[0019] Among them, F irepresents the foreground feature map; B represents the background feature map; att represents the self-attention operation; softmax represents the normalization operation; w i represents the weight of the foreground feature map; w represents the weight of the background feature map.

[0020] Preferably, the method for enhancing the foreground feature map and the background feature map further comprises: designing an interactive attention query module, obtaining feature mappings of the foreground feature map and the background feature map: query value, key value and value, using the query values ​​of the foreground feature map and the background feature map to query the key value of each other, using the query results to construct weights, and assigning the weights to the values ​​of the queried feature maps to obtain the first foreground feature map and the first background feature map; the formula includes: E

[0021]

[0022] Among them, Q i , K i and V i represents the query value, key value and value of the foreground feature map; Q, K and V represent the query value, key value and value of the background feature map; d ki and d k K i and the dimension of K; w i represents the weight of the foreground feature map; w represents the weight of the background feature map; F′ i represents the foreground feature map updated by the interactive attention module, and B′ represents the background feature map updated by the interactive attention module.

[0023] Preferably, the method for generating the prediction frame includes: predicting the foreground target center point and the target bounding box on the first foreground feature map to obtain the prediction frame.

[0024] Preferably, the foreground feature vector is calculated using the predicted box and the first foreground feature map; the background feature vector is calculated using the first background feature map; and the real feature vector is calculated using the real box and the first foreground feature map.

[0025] Preferably, the contrast loss function is constructed based on the similarity between the foreground feature vector and the background feature vector; the foreground saliency guided loss function is constructed based on the foreground feature vector and the true feature vector; and the prediction box loss function is constructed using the foreground target center point and the target bounding box.

[0026] Preferably, a total loss function is constructed based on the contrast loss function, the foreground saliency guided loss function and the prediction box loss function; a network model is trained using the total loss function; and target recognition is completed using the network model.

[0027] The present application also provides an object detection system for distinguishing foreground from background, comprising: an extraction module, an enhancement module, a generation module, a calculation module, a construction module, and a prediction module;

[0028] The extraction module is used to extract the foreground feature map and the background feature map of the target to be detected based on multi-scale feature fusion;

[0029] The enhancement module is used to enhance the foreground feature map and the background feature map using an attention module to obtain a first foreground feature map and a first background feature map;

[0030] The generating module is used to generate a prediction frame based on the first foreground feature map;

[0031] The calculation module is used to calculate a foreground feature vector using the predicted frame and the first foreground feature map; calculate a background feature vector using the first background feature map; and calculate a real feature vector using the real frame and the first foreground feature map;

[0032] The construction module is used to construct a contrast loss function, a foreground saliency guided loss function and a prediction box loss function based on the foreground feature vector, the background feature vector, the true feature vector and the prediction box;

[0033] The prediction module is used to combine the contrast loss function, the foreground saliency guided loss function and the prediction box loss function to improve the distinction between the foreground and background of the target and complete target detection.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] By designing an interactive attention module, the object detection algorithm learns foreground and background information. Based on this, a contrastive loss function is constructed using the similarity between the foreground object information and the background information. This proposed method guides the network model's learning from two aspects: weights and feature differences between the foreground and background. This allows the network model to learn not only the foreground and background information itself, but also the differences between the two. By constraining the network model with the contrastive loss function, it better learns the differences between foreground and background information, improving the network model's ability to distinguish between foreground and background. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present application;

[0038] Figure 2 A schematic diagram of an interactive attention module according to an embodiment of the present application;

[0039] Figure 3 This is a schematic diagram of an interactive attention query module according to an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the overall network flow of an embodiment of the present application;

[0041] Figure 5 Schematic diagram of the system structure of an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0044] Example 1

[0045] like Figure 1 FIG. 1 is a flow chart of the method of this embodiment, and the steps include:

[0046] S1. Based on multi-scale feature fusion, extract the foreground feature map and background feature map of the target to be detected.

[0047] In order to obtain the location and category information of the target, the feature extraction network is first used to extract the necessary feature information from the input image, laying the foundation for the subsequent tasks of the network; then the multi-scale fusion network is used to fuse the features extracted from the feature extraction network, and the feature maps of different depths in the fusion network are fused to enrich the feature information extracted by the multi-scale fusion network; finally, the features fused by the multi-scale fusion network are distinguished as the foreground feature map F by convolution. i And background feature map B, where i is the category.

[0048] In this embodiment, the feature extraction network can be selected from VGG, Resnet, Darknet, etc.; in this embodiment, the multi-scale feature fusion network can be selected from FPN, PANet, etc.

[0049] S2. Use the attention module to enhance the foreground feature map and the background feature map to obtain a first foreground feature map and a first background feature map.

[0050] In this embodiment, the enhancement method uses one of two interactive attention modules to enhance the foreground feature map and the background feature map; the two interactive attention modules are respectively as follows: Figure 2 The interactive reverse attention module and Figure 3 The interactive attention query module is shown in Figure 5.

[0051] Using interactive reverse attention modules (such as Figure 2 The steps include: using background attention to reversely solve the weight of the foreground feature map, and using foreground attention to reversely solve the weight of the background feature map; specifically, performing self-attention operations on the foreground feature map Fi and the background feature map B respectively to obtain the weight w of the foreground feature map i And the weight w of the background feature map, the process formula is as follows:

[0052]

[0053] Among them, F i represents the foreground feature map; B represents the background feature map; att represents the self-attention operation; softmax represents the normalization operation; w i represents the weight of the foreground feature map; w represents the weight of the background feature map.

[0054] On this basis, interactive attention calculation is performed on the foreground feature map and the background feature map. In order to make the background feature more significant, the weight of the foreground feature map is smaller; the more significant the foreground feature, the smaller the weight of the background feature map. The following formula is constructed:

[0055]

[0056] Among them, among them, F′ i represents the foreground feature map after the interactive attention module updates; B′ represents the background feature map after the interactive attention module updates.

[0057] Using interactive attention query modules (such as Figure 3 As shown in FIG, obtaining feature maps of the foreground feature map and the background feature map respectively: query value (Query), key value (Key) and value (Value), using the query value of the background feature map and the key value and value of the foreground feature map to build an attention mechanism to obtain a first foreground feature map; using the query value of the foreground feature map and the key value and value of the background feature map to build an attention mechanism to obtain a first background feature map. The steps include:

[0058] Calculate the feature map query value Q of the foreground feature mapi (Query i ), key value K i (Key i ) and value V i (Value i ); Calculate the feature map query value Q (Query), key value K (Key) and value V (Value) of the background feature map. First, use the convolution module to calculate the query value, key value and value of the foreground and background feature maps:

[0059]

[0060] Among them, w i Q 、 and w i V Respectively represent the calculation of Q i , K i and V i The weight value of the convolution; w Q 、w K and w V Represent the weight values ​​of the convolution when calculating Q, K and V respectively.

[0061] Use the query values ​​Q and Q of the background feature map and foreground feature map i , the key value K of the foreground feature map and background feature map i To prevent the query result from being too large, the dimension value of the key value is used to reduce the query result, and then the softmax is used to normalize the query result to obtain the weight w of the foreground feature map. i and the weight w of the background feature map; finally, use the weight w of the foreground feature map i And the value V of the foreground feature map i Multiply the weight w of the background feature map and the value V of the background feature map to get the first background feature map. The process formula is as follows:

[0062]

[0063] Among them, d ki and d k K i and the dimension of K; F′ i represents the foreground feature map updated by the interactive attention module, and B′ represents the background feature map updated by the interactive attention module.

[0064] Through the above steps, a first foreground feature map and a first background feature map are obtained.

[0065] S3. Generate a prediction box based on the first foreground feature map.

[0066] Calculate the position of each pixel on the first foreground feature map mapped back to the original image. If the position is in the center of the ground-truth bounding box, the point is considered the predicted center point. The centrality of the center point is calculated based on the distance from the predicted center point to the center point of the ground-truth bounding box. A 4D vector is predicted at the predicted center point, representing the distance from the predicted center point to the top, bottom, left, and right sides of the predicted bounding box, to obtain the predicted bounding box.

[0067] Predict the foreground object center point and the object bounding box on the first foreground feature map to obtain the predicted box. The specific steps include:

[0068] For each point (x, y) of the foreground feature map of the interactive attention module, if the point falls in the center area of ​​the true box, the point is regarded as a positive sample and the classification probability p of the point is predicted. x,y Otherwise, it is regarded as a negative sample, and p x,y = 0, where the center area of ​​the real frame is the center of the real frame (c x , c y ) as the center point and a rectangular area with r as the radius (c x -r,c y -r,c x +r,c y +r).

[0069] In order to screen high-quality center points, the straight-line distance between each predicted center point and the true center point is calculated, and all distances are mapped to (0,1) to obtain d x,y , use the following formula to calculate the center point parameter o x,y :

[0070] o x,y =1-d x,y (5)

[0071] Center point parameter o x,y Indicates the distance between the positive sample point on the feature map and the true center of the target. The closer it is to the true center of the target, the closer the parameter is to 1; the farther it is from the true center of the target, the closer the parameter is to 0. According to the center point parameter o of each position x,y And the classification probability p of the position x,y Calculate the centrality s of the location x,y , the process formula is as follows:

[0072]

[0073] If a point on a feature map falls into the center of multiple ground-truth boxes simultaneously, it is assigned to the ground-truth box with the smaller area by default. Based on this, scale thresholds are set for the first foreground feature maps of different scales, so that feature maps of different scales are only responsible for predicting objects within the corresponding scale threshold.

[0074] The ground truth box can be represented as a 4D vector Among them, l T , t T 、r T and b T Represents the distance from the point to the four sides of the real box. The formulas for calculating them are as follows:

[0075]

[0076] Among them, x0, y0, x1, y1 are the horizontal and vertical coordinates of the upper left corner and the lower right corner of the real box, and s is the downsampling rate of the original image to the current scale feature map. The network model generates a 4D vector when predicting: x,y =(l, t, r, b), which is used as the target bounding box information predicted by the network model. x,y The formula is as follows:

[0077]

[0078] Among them, w l 、w t 、w r and w b is the weight value of the convolution when calculating l, t, r and b.

[0079] Use the predicted center point and predicted target bounding box information t x,y The prediction box can be obtained during the network model inference process.

[0080] S4. Calculate the foreground feature vector using the predicted box and the first foreground feature map; calculate the background feature vector using the first background feature map; and calculate the true feature vector using the true box and the first foreground feature map.

[0081] The foreground feature vector is calculated using the predicted box and the first foreground feature map, and the background feature vector is calculated using the first background feature map. On the first foreground feature map, the real box is used to extract the pixels corresponding to the real target to calculate the real feature vector for the subsequent construction of the saliency guided loss function.

[0082] Specifically, use the prediction box in S3 to extract the pixels in the prediction box in the first foreground feature map, and use its mean and variance as the feature vector X of the foreground i; Take the mean variance of all pixels in the first background feature map as the background feature vector X; Use the real frame to extract the pixels in the real frame of the first foreground feature map, and use its mean and variance as the real feature vector X T The above three feature vectors are updated with iteration to ensure that each vector can describe the global data characteristics.

[0083] S5. Based on the foreground feature vector, background feature vector, true feature vector and prediction box, construct the contrast loss function, foreground saliency guided loss function and prediction box loss function.

[0084] Calculate the similarity between the foreground feature vector and the background feature vector, and use this to construct a contrast loss function that maximizes the gap between the foreground feature and the background feature. Use the foreground feature vector and the true feature vector to construct a foreground saliency guided loss function that allows the network to focus on samples with lower foreground saliency responses. Use the foreground target center point and the target bounding box to construct a prediction box loss function that allows the network to more accurately locate and classify targets. Specifically, calculate the cosine similarity of the foreground feature vector and the background feature vector of the same type and construct a contrast loss function to make the similarity of similar targets in the foreground as large as possible and the similarity between the foreground and the background as small as possible. Use the foreground feature vector and the true feature vector to calculate the foreground saliency response, and construct a foreground saliency guided loss function to guide the network to focus on targets with lower foreground saliency responses. Use the predicted value and the true value of the prediction box center point and 4D vector to construct the classification loss, regression loss and center point loss to form the prediction box loss function. The steps include:

[0085] When the foreground feature vector calculated in a certain iteration belongs to the same category as the foreground feature vector saved in the previous iteration, the cosine similarity between the two is calculated to obtain the similarity between the foreground of category i and the foreground. The foreground feature vector calculated in this iteration is used to calculate the cosine similarity with the background feature vector saved previously, and the similarity sim between the i-th foreground and background is obtained. b i. The formula for calculating cosine similarity is as follows:

[0086]

[0087] Among them, X′ i For X i represents the foreground feature vector saved from the previous iteration of the same class, and X′ represents the background feature vector saved from the previous iteration.

[0088] In order to make the difference between the foreground object and the background as large as possible, the following contrast loss function is constructed:

[0089]

[0090] Among them, N I The number of categories that belong to the same category as the foreground feature vector calculated in this iteration and the foreground feature vector saved in the previous iteration.

[0091] In order to make the foreground have a guiding role in network learning, the following foreground saliency guided loss function is constructed:

[0092]

[0093] Among them, p x,y and y x,y Respectively represent the predicted probability and true value of the point (x, y) on the first foreground feature map; L CE represents the cross entropy loss function; It represents the foreground saliency response of the corresponding category at point (x, y) on the first foreground feature map. The calculation formula of the foreground saliency response is:

[0094]

[0095] In order to make the network positioning and classification more accurate, a prediction box composed of classification loss L is constructed. cls , positioning loss L reg and center point loss L cen The loss function of the composed prediction box is as follows:

[0096]

[0097] Among them, L cls is the focal loss function combined with binary cross entropy; L reg is the GIOU loss function; L cen is the binary cross entropy loss function; N is the number of positive samples, p x,y is the score of each type of target predicted at the point (x, y) in the feature map; is the true category label at the point (x, y) in the feature map; When the feature map (x, y) point is matched as a positive sample, it is 1, otherwise it is 0; t x,y The target bounding box information predicted at the feature map (x, y) point; is the true target bounding box information at the feature map (x, y) point; s x,y is the predicted centrality of the feature map (x, y) point; is the true centrality at the feature graph (x, y) point. The true centrality calculation formula at the feature graph (x, y) point is as follows:

[0098]

[0099] S6. Combine the contrast loss function, the foreground saliency guided loss function, and the prediction box loss function to improve the distinction between the foreground and background of the target and complete the target detection.

[0100] The network model is trained by combining the above three loss functions (the overall network flow chart of this embodiment is as follows Figure 4 As shown in the figure, the model can distinguish between foreground and background features, thereby improving the target recognition ability. The steps include:

[0101] The objective function is used as the loss function of the neural network. The weights in the network model are updated through reverse gradient calculation so that the neural network can fully distinguish the foreground and background features. The overall loss function is as follows:

[0102] L total =L anchor +λL sim +γL FSG (15)

[0103] Among them, L total is the total loss function, λ and γ are adjustable parameters, which play an appropriate role in the prediction box loss function, the ratio loss function and the foreground saliency guided loss function.

[0104] Finally, the target is detected through the above monitoring model.

[0105] Example 2

[0106] like Figure 5 As shown, it is a schematic diagram of the system structure of this embodiment, including: an extraction module, an enhancement module, a generation module, a calculation module, a construction module and a prediction module. Among them, the extraction module is used to extract the foreground feature map and the background feature map of the target to be detected based on multi-scale feature fusion; the enhancement module is used to enhance the foreground feature map and the background feature map using the attention module to obtain the first foreground feature map and the first background feature map; the generation module is used to generate a prediction frame based on the first foreground feature map; the calculation module is used to calculate the foreground feature vector using the prediction frame and the first foreground feature map; the background feature vector using the first background feature map; the true feature vector using the true frame and the first foreground feature map; the construction module is used to construct a contrast loss function, a foreground saliency guided loss function and a prediction frame loss function based on the foreground feature vector, the background feature vector, the true feature vector and the prediction frame; the prediction module is used to combine the contrast loss function, the foreground saliency guided loss function and the prediction frame loss function to improve the discrimination between the foreground and background of the target and complete target detection.

[0107] The following will explain in detail how this application solves technical problems in real life in conjunction with this embodiment.

[0108] First, the extraction module is used to extract the foreground feature map and background feature map of the target to be detected based on multi-scale feature fusion.

[0109] In order to obtain the location and category information of the target, the feature extraction network is first used to extract the necessary feature information from the input image, laying the foundation for the subsequent tasks of the network; then the multi-scale fusion network is used to fuse the features extracted from the feature extraction network, and the feature maps of different depths in the fusion network are fused to enrich the feature information extracted by the multi-scale fusion network; finally, the features fused by the multi-scale fusion network are distinguished as the foreground feature map F by convolution. i And background feature map B, where i is the category.

[0110] In this embodiment, the feature extraction network can be selected from VGG, Resnet, Darknet, etc.; in this embodiment, the multi-scale feature fusion network can be selected from FPN, PANet, etc.

[0111] Then, the enhancement module is used to enhance the foreground feature map and the background feature map using the attention module to obtain a first foreground feature map and a first background feature map.

[0112] In this embodiment, the enhancement module uses one of two interactive attention modules to enhance the foreground feature map and the background feature map; the two interactive attention modules are respectively as follows: Figure 2 The interactive reverse attention module and Figure 3 The interactive attention query module is shown in Figure 5.

[0113] Using interactive reverse attention modules (such as Figure 2 The process includes: using the background attention to reversely solve the weight of the foreground feature map, and using the foreground attention to reversely solve the weight of the background feature map; specifically, the foreground feature map F i Perform self-attention operation on the background feature map B to obtain the weight w of the foreground feature map i And the weight w of the background feature map, the process formula is as follows:

[0114]

[0115] Among them, F i represents the foreground feature map; B represents the background feature map; att represents the self-attention operation; softmax represents the normalization operation; w i represents the weight of the foreground feature map; w represents the weight of the background feature map.

[0116] On this basis, interactive attention calculation is performed on the foreground feature map and the background feature map. In order to make the background feature more significant, the weight of the foreground feature map is smaller; the more significant the foreground feature, the smaller the weight of the background feature map. The following formula is constructed:

[0117]

[0118] Among them, among them, F′ i represents the foreground feature map after the interactive attention module updates; B′ represents the background feature map after the interactive attention module updates.

[0119] Using interactive attention query modules (such as Figure 3 As shown in FIG, the feature maps of the foreground feature map and the background feature map are obtained respectively: query value (Query), key value (Key) and value (Value), and the query value of the background feature map and the key value and value of the foreground feature map are used to build an attention mechanism to obtain a first foreground feature map; the query value of the foreground feature map and the key value and value of the background feature map are used to build an attention mechanism to obtain a first background feature map. The process includes:

[0120] Calculate the feature map query value Q of the foreground feature map i (Query i ), key value K i (Key i ) and value V i (Value i ); Calculate the feature map query value Q (Query), key value K (Key) and value V (Value) of the background feature map. First, use the convolution module to calculate the query value, key value and value of the foreground and background feature maps:

[0121]

[0122] Among them, w i Q 、 and w i V Respectively represent the calculation of Q i , K i and V i The weight value of the convolution; w Q 、w K and w V Represent the weight values ​​of the convolution when calculating Q, K and V respectively.

[0123] Use the query values ​​Q and Q of the background feature map and foreground feature map i , the key value K of the foreground feature map and background feature map iTo prevent the query result from being too large, the dimension value of the key value is used to reduce the query result, and then the softmax is used to normalize the query result to obtain the weight w of the foreground feature map. i and the weight w of the background feature map; finally, use the weight w of the foreground feature map i And the value V of the foreground feature map i Multiply the weight w of the background feature map and the value V of the background feature map to get the first background feature map. The process formula is as follows:

[0124]

[0125] Among them, d ki and d k K i and the dimension of K; F′ i represents the foreground feature map updated by the interactive attention module, and B′ represents the background feature map updated by the interactive attention module.

[0126] Through the above process, a first foreground feature map and a first background feature map are obtained.

[0127] The generation module generates a prediction box based on the first foreground feature map.

[0128] Calculate the position of each pixel on the first foreground feature map mapped back to the original image. If the position is in the center of the ground-truth bounding box, the point is considered the predicted center point. The centrality of the center point is calculated based on the distance from the predicted center point to the center point of the ground-truth bounding box. A 4D vector is predicted at the predicted center point, representing the distance from the predicted center point to the top, bottom, left, and right sides of the predicted bounding box, to obtain the predicted bounding box.

[0129] Predict the foreground target center point and target bounding box on the first foreground feature map to obtain the predicted box. The specific process includes:

[0130] For each point (x, y) of the foreground feature map of the interactive attention module, if the point falls in the center area of ​​the true box, the point is regarded as a positive sample and the classification probability p of the point is predicted. x,y Otherwise, it is regarded as a negative sample, and p x,y = 0, where the center area of ​​the real frame is the center of the real frame (c x , c y ) as the center point and a rectangular area with r as the radius (c x -r,c y -r,c x +r,c y +r).

[0131] In order to screen high-quality center points, the straight-line distance between each predicted center point and the true center point is calculated, and all distances are mapped to (0,1) to obtain d x,y , use the following formula to calculate the center point parameter o x,y :

[0132] o x,y =1-d x,y (5)

[0133] Center point parameter o x,y Indicates the distance between the positive sample point on the feature map and the true center of the target. The closer it is to the true center of the target, the closer the parameter is to 1; the farther it is from the true center of the target, the closer the parameter is to 0. According to the center point parameter o of each position x,y And the classification probability p of the position x,y Calculate the centrality s of the location x,y , the process formula is as follows:

[0134]

[0135] If a point on a feature map falls into the center of multiple ground-truth boxes simultaneously, it is assigned to the ground-truth box with the smaller area by default. Based on this, scale thresholds are set for the first foreground feature maps of different scales, so that feature maps of different scales are only responsible for predicting objects within the corresponding scale threshold.

[0136] The ground truth box can be represented as a 4D vector Among them, l T , t T 、r T and b T Represents the distance from the point to the four sides of the real box. The formulas for calculating them are as follows:

[0137]

[0138] Among them, x0, y0, x1, y1 are the horizontal and vertical coordinates of the upper left corner and the lower right corner of the real box, and s is the downsampling rate of the original image to the current scale feature map. The network model generates a 4D vector when predicting: x,y =(l, t, r, b), which is used as the target bounding box information predicted by the network model. x,y The formula is as follows:

[0139]

[0140] Among them, w l 、w t 、w r and w b is the weight value of the convolution when calculating l, t, r and b.

[0141] Use the predicted center point and predicted target bounding box information t x,y The prediction box can be obtained during the network model inference process.

[0142] The calculation module calculates a foreground feature vector using the predicted frame and the first foreground feature map; calculates a background feature vector using the first background feature map; and calculates a real feature vector using the real frame and the first foreground feature map.

[0143] The foreground feature vector is calculated using the predicted box and the first foreground feature map, and the background feature vector is calculated using the first background feature map. On the first foreground feature map, the real box is used to extract the pixels corresponding to the real target to calculate the real feature vector for the subsequent construction of the saliency guided loss function.

[0144] Specifically, the prediction box is used to extract the pixels within the prediction box in the first foreground feature map, and its mean and variance are used as the feature vector X of the foreground. i ; Take the mean variance of all pixels in the first background feature map as the background feature vector X; Use the real frame to extract the pixels in the real frame of the first foreground feature map, and use its mean and variance as the real feature vector X T The above three feature vectors are updated with iteration to ensure that each vector can describe the global data characteristics.

[0145] The construction module constructs the contrast loss function, foreground saliency guided loss function and prediction box loss function based on the foreground feature vector, background feature vector, true feature vector and prediction box.

[0146] Calculate the similarity between the foreground feature vector and the background feature vector, and use this to construct a contrast loss function that maximizes the gap between the foreground feature and the background feature. Use the foreground feature vector and the true feature vector to construct a foreground saliency guided loss function that allows the network to focus on samples with lower foreground saliency responses. Use the foreground target center point and the target bounding box to construct a prediction box loss function that allows the network to more accurately locate and classify targets. Specifically, calculate the cosine similarity of the foreground feature vector and the background feature vector of the same type and construct a contrast loss function to make the similarity of similar targets in the foreground as large as possible and the similarity between the foreground and the background as small as possible. Use the foreground feature vector and the true feature vector to calculate the foreground saliency response, and construct a foreground saliency guided loss function to guide the network to focus on targets with lower foreground saliency responses. Use the predicted value and the true value of the prediction box center point and 4D vector to construct the classification loss, regression loss and center point loss to form the prediction box loss function. The process includes:

[0147] When the foreground feature vector calculated in a certain iteration belongs to the same category as the foreground feature vector saved in the previous iteration, the cosine similarity between the two is calculated to obtain the similarity between the foreground of category i and the foreground. Use the foreground feature vector calculated in this iteration and the previously saved background feature vector to perform cosine similarity calculation to obtain the similarity between the i-th foreground and background The formula for calculating cosine similarity is as follows:

[0148]

[0149] Among them, X′ i For X i represents the foreground feature vector saved from the previous iteration of the same class, and X′ represents the background feature vector saved from the previous iteration.

[0150] In order to make the difference between the foreground object and the background as large as possible, the following contrast loss function is constructed:

[0151]

[0152] Among them, N I The number of categories that belong to the same category as the foreground feature vector calculated in this iteration and the foreground feature vector saved in the previous iteration.

[0153] In order to make the foreground have a guiding role in network learning, the following foreground saliency guided loss function is constructed:

[0154]

[0155] Among them, p x,y and y x,y Respectively represent the predicted probability and true value of the point (x, y) on the first foreground feature map; L CE represents the cross entropy loss function; It represents the foreground saliency response of the corresponding category at point (x, y) on the first foreground feature map. The calculation formula of the foreground saliency response is:

[0156]

[0157] In order to make the network positioning and classification more accurate, a prediction box composed of classification loss L is constructed. cls , positioning loss L reg and center point loss L cen The prediction box loss function is as follows:

[0158]

[0159] Among them, L clsis the focal loss function combined with binary cross entropy; L reg is the GIOU loss function; L cen is the binary cross entropy loss function; N is the number of positive samples, p x,y is the score of each type of target predicted at the point (x, y) in the feature map; is the true category label at the point (x, y) in the feature map; When the feature map (x, y) point is matched as a positive sample, it is 1, otherwise it is 0; t x,y The target bounding box information predicted at the feature map (x, y) point; is the true target bounding box information at the feature map (x, y) point; s x,y is the predicted centrality of the feature map (x, y) point; is the true centrality at the feature graph (x, y) point. The true centrality calculation formula at the feature graph (x, y) point is as follows:

[0160]

[0161] The prediction module combines the contrast loss function, the foreground saliency guided loss function and the prediction box loss function to improve the distinction between the foreground and background of the target and complete the target detection.

[0162] The network model is trained by combining the above three loss functions (the overall network flow chart of this embodiment is as follows Figure 4 As shown in the figure, the model can distinguish between foreground and background features, thereby improving the target recognition ability. The process includes:

[0163] The objective function is used as the loss function of the neural network. The weights in the network model are updated through reverse gradient calculation so that the neural network can fully distinguish the foreground and background features. The overall loss function is as follows:

[0164] L total =L anchor +λL sim +γL FSG (15)

[0165] Among them, L total is the total loss function, λ and γ are adjustable parameters, which play an appropriate role in the prediction box loss function, the ratio loss function and the foreground saliency guided loss function.

[0166] Finally, the target is detected through the above monitoring model.

[0167] The embodiments described above are merely descriptions of the preferred embodiments of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present application.

Claims

1. A method for detecting an object to distinguish between foreground and background, characterized in that the steps include: Based on multi-scale feature fusion, the foreground feature map and background feature map of the target to be detected are extracted; The foreground feature map and the background feature map are enhanced using an attention module to obtain a first foreground feature map and a first background feature map; Generate a prediction box based on the first foreground feature map; Calculating a foreground feature vector using the prediction box and the first foreground feature map; use The first background feature map is used to calculate a background feature vector; and the real feature vector is calculated using the real frame and the first foreground feature map; Constructing a contrast loss function, a foreground saliency guided loss function, and a prediction box loss function based on the foreground feature vector, the background feature vector, the true feature vector, and the prediction box; The contrast loss function, the foreground saliency guided loss function and the prediction box loss function are combined to improve the distinction between the foreground and background of the target and complete target detection.

2. The target detection method for distinguishing foreground from background according to claim 1, characterized in that: The method for extracting the foreground feature map and the background feature map includes: first using a feature extraction network to extract features of the target to be detected; then using a multi-scale feature fusion network to perform multi-scale feature fusion on the extracted features to obtain the foreground feature map and the background feature map.

3. The target detection method for distinguishing foreground from background according to claim 1, characterized in that: The method for enhancing the foreground feature map and the background feature map includes: designing an interactive reverse attention module, using background attention to reversely solve the foreground feature map weight, and simultaneously using foreground attention to reversely solve the background feature map weight, to obtain the first foreground feature map and the first background feature map; the formula for calculating the foreground feature map weight and the background feature map weight is as follows: Among them, F i represents the foreground feature map; B represents the background feature map; att represents the self-attention operation; softmax represents the normalization operation; w i represents the weight of the foreground feature map; w represents the weight of the background feature map.

4. The object detection method for distinguishing foreground from background according to claim 1, characterized in that: The method for enhancing the foreground feature map and the background feature map further includes: designing an interactive attention query module to obtain feature mappings of the foreground feature map and the background feature map: a query value, a key value, and a value; using the query values ​​of the foreground feature map and the background feature map to query the key values ​​of each other; using the query results to construct weights; and assigning the weights to the values ​​of the queried feature maps to obtain the first foreground feature map and the first background feature map; the formula includes: Among them, Q i , K i and V i represents the query value, key value and value of the foreground feature map; Q, K and V represent the query value, key value and value of the background feature map; d ki and d k K i and the dimension of K; w i represents the weight of the foreground feature map; w represents the weight of the background feature map; F′ i represents the foreground feature map updated by the interactive attention module, and B′ represents the background feature map updated by the interactive attention module.

5. The target detection method for distinguishing foreground from background according to claim 1, wherein: The method for generating the prediction frame includes: predicting a foreground target center point and a target bounding box on the first foreground feature map to obtain the prediction frame.

6. The object detection method for distinguishing foreground from background according to claim 1, characterized in that: The foreground feature vector is calculated using the predicted box and the first foreground feature map; the background feature vector is calculated using the first background feature map; and the real feature vector is calculated using the real box and the first foreground feature map.

7. The object detection method for distinguishing foreground from background according to claim 6, characterized in that: Based on the similarity between the foreground feature vector and the background feature vector, the contrast loss function is constructed; based on the foreground feature vector and the true feature vector, the foreground saliency guided loss function is constructed; and using the foreground target center point and the target bounding box, the prediction box loss function is constructed.

8. The object detection method for distinguishing foreground from background according to claim 7, characterized in that: Based on the contrast loss function, the foreground saliency guided loss function and the prediction box loss function, a total loss function is constructed; a network model is trained using the total loss function; and target recognition is completed using the network model.

9. An object detection system for distinguishing foreground from background, characterized in that: include: Extraction module, enhancement module, generation module, calculation module, construction module and prediction module; The extraction module is used to extract the foreground feature map and the background feature map of the target to be detected based on multi-scale feature fusion; The enhancement module is used to enhance the foreground feature map and the background feature map using an attention module to obtain a first foreground feature map and a first background feature map; The generating module is used to generate a prediction frame based on the first foreground feature map; The calculation module is used to calculate a foreground feature vector using the prediction frame and the first foreground feature map; use The first background feature map is used to calculate a background feature vector; and the real feature vector is calculated using the real frame and the first foreground feature map; The construction module is used to construct a contrast loss function, a foreground saliency guided loss function and a prediction box loss function based on the foreground feature vector, the background feature vector, the true feature vector and the prediction box; The prediction module is used to combine the contrast loss function, the foreground saliency guided loss function and the prediction box loss function to improve the distinction between the foreground and background of the target and complete target detection.

Images