A weakly supervised detection method based on multi-level feature fusion

Abstract

The present invention provides a weakly supervised detection method of multi-level feature fusion, including: obtaining a picture to be analyzed; inputting the picture to be analyzed into a preset target classification module, obtaining the feature maps of Layer1, Layer2, Layer3 and Layer4 in the target classification module; generating a corresponding class activation map based on the feature maps of Layer1, Layer2, Layer3 and Layer4; fusing the class activation maps of Layer1, Layer2 and Layer3 to form a foreground area; locating the target in the picture to be analyzed based on the class activation map of the foreground area and Layer4. The weakly supervised detection method of multi-level feature fusion of the present invention improves the Resnet50 backbone network by adopting a coordinate attention mechanism to capture the key features of the target, and provides a calculation basis for the target locator while improving the fine-grained classification performance of the model on the target.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-level feature fusion weak supervision detection method. Background Art

[0002] In recent years, with the rapid development of the aviation industry and the increasing number of air routes, the competition between aircraft and birds for airspace has become increasingly fierce. The probability of bird strike accidents around the world has shown a clear upward trend, causing serious loss of life and property. Faced with the severe situation of bird strike prevention, domestic and foreign airports have carried out extensive and in-depth bird surveys, adopted a variety of bird strike prevention measures, and equipped with bird detection radars, gas cannons, ultrasonic and laser bird repellents and other bird repellent equipment. However, in the process of bird repelling at the airport, it is difficult for bird detection radars to obtain bird species information. Usually, birds existing at low altitudes are identified by manual observation, which has the problems of low detection efficiency and high labor costs. It makes it impossible for airports to adopt targeted bird repellent strategies for birds, resulting in birds easily developing tolerance to bird repellent equipment, reducing the bird repellent effect of the equipment. To this end, a method based on visual target detection is used to detect birds at the airport and provide ground staff with bird species and location information, which is of great significance to improving the existing bird repellent mode and ensuring the flight safety of civil aircraft.

[0003] Bird detection at airports is a challenging visual task that requires fine-grained recognition and positioning of birds in the low altitude of airports, and has high requirements for the accuracy and efficiency of model detection. Generally speaking, airport bird detection faces the following three difficulties:

[0004] (1) The same bird has different movement postures during its activities. Different birds have similar appearances and forms, with large differences within the same bird group and small differences between different bird groups.

[0005] (2) The collection and labeling of bird images is relatively difficult and requires expert knowledge and a certain industry threshold. Small amounts of data can easily lead to network overfitting problems.

[0006] (3) The environment in which birds live is relatively complex and is seriously affected by background noise, which increases the difficulty of the bird detection task. Summary of the invention

[0007] One of the purposes of the present invention is to provide a weakly supervised detection method of multi-level feature fusion, by adopting the coordinate attention mechanism to improve the Resnet50 backbone network, to capture the key features of the target, while improving the model's fine-grained classification performance of the target, to provide a calculation basis for the target locator. In the weakly supervised positioning task, the predicted score of the target classifier for the category of the target is used as the starting point of back propagation, and the contribution of each position in the shallow, middle and deep feature maps of the network to the network decision is obtained by comparing the hierarchical correlation propagation rules, and then the class activation mapping algorithm is used to highlight the important features of the network in the target image extraction in the form of a heat map, and a high-resolution class activation map is generated by fusion, which is used to indicate the contour and spatial position of the target, and accurately and efficiently completes the positioning task of the target. Provide the type and location information of the target for the airport ground staff, so as to adopt a targeted bird-repelling strategy and improve the existing bird-repelling mode of the airport.

[0008] The embodiment of the present invention provides a multi-level feature fusion weak supervision detection method, including:

[0009] Get the image to be analyzed;

[0010] Input the image to be analyzed into the preset target classification module, and obtain the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4 in the target classification module;

[0011] Based on the feature maps of Layer 1, Layer 2, Layer 3, and Layer 4, generate the corresponding class activation maps;

[0012] The class activation maps of Layer 1, Layer 2, and Layer 3 are fused to form the foreground area;

[0013] Based on the foreground area and the class activation map of Layer 4, the target in the image to be analyzed is located.

[0014] Preferably, the target classification module includes:

[0015] Resnet50 network is used as the backbone;

[0016] In the pooling layer, each channel of the given input is encoded according to the horizontal and vertical coordinates to obtain a direction-aware feature map and embed the coordinate information;

[0017] The extracted direction information is concatenated, transformed using 1×1 convolution, decomposed into two separate tensors along the spatial dimension, and made to have the same number of channels using 1×1 convolution, generating a coordinate information mask as the attention weight.

[0018] The attention weight is multiplied pixel by pixel by the input vector as the output.

[0019] Preferably, the target classification module is obtained by training and converging using a data set in advance;

[0020] The dataset is constructed through the following steps:

[0021] According to the types of common targets near the airport, a dataset is created using target category labels and bounding box annotations.

[0022] Preferably, during the training of the target classification module, a joint loss function is used to guide the training;

[0023] The joint loss function is as follows:

[0024] L=L ce +λL bp ;

[0025] Where L represents the joint loss function; L ce represents the cross entropy loss function; L bp represents the perceptual loss function; λ represents the preset hyperparameter for controlling the weight of the perceptual loss function;

[0026] The perceptual loss function is as follows:

[0027]

[0028] Where N represents the number of image samples; M FA represents the class activation mask in the class activation map fusion process; M HR Represents the activation mask generated by all features of the last convolutional layer of the target classification module;

[0029] The cross entropy loss function is as follows:

[0030]

[0031] In the formula, g indicates the category; G indicates the total number of categories; u jg Represents the symbolic function. When the true category of the image sample j is equal to g, u jg =1; otherwise, u jg =0;v jg Represents the probability that image sample j belongs to category g.

[0032] Preferably, based on the feature map of Layer 1, a corresponding class activation map is generated, including:

[0033] The calculation steps of the class activation map are as follows:

[0034]

[0035]

[0036] In the formula, A represents the class activation map corresponding to the feature map of Layer 1; k Represents the kth feature map of Layer1; It represents the contribution of position (x, y) in the k-th feature map of Layer1 to the target category c obtained by back propagation using contrastive hierarchical correlation propagation theory. Represents the kth feature map A of Layer1 k Channel-wise weights for target category c.

[0037] Preferably, the contribution of the position (x, y) in the k-th feature map of Layer 1 obtained by back propagation of contrast hierarchical correlation theory to the target category c includes:

[0038] Use contrastive hierarchical correlation propagation to obtain the contribution of each position in the feature map to the target classification decision;

[0039] Among them, the comparison of hierarchical correlation propagation includes:

[0040] The network output value before the initialization softmax operation is used to distinguish the ratio of target category neurons and non-target category neurons, and serves as the starting point for back propagation of comparison hierarchical correlation. The calculation steps of the initialization function are as follows:

[0041]

[0042] In the formula, represents the score corresponding to the i-th neuron in the l-th layer before the Softmax operation, and P represents the total number of neurons in the l-th layer;

[0043] In the back propagation phase of contrast hierarchical correlation, the correlation score between each variable and the network decision is measured by decomposing from the output layer of the network to each variable in the input space;

[0044] The commonly used hierarchical correlation propagation rule is z + and z β The rules are defined as follows:

[0045]

[0046]

[0047] In the formula, and They represent the positive and negative weights connecting the neurons in the lth and l+1th layers respectively; [U, V] represents the value range of the neuron activation value.

[0048] Preferably, when determining the correlation score, it is also standardized, and the value of the correlation score is fixed in the interval [0-1]; the standardized calculation formula is as follows:

[0049]

[0050] In the formula, is the correlation score of position (x, y) in the k-th feature map of Layer1 before normalization with respect to category c; is the set of relevance scores for each position in the kth feature map with respect to the target category c.

[0051] Preferably, based on the foreground area and the class activation map of Layer 4, the target in the image to be analyzed is located, including:

[0052] Multiply the foreground area by the class activation map of Layer 4 element by element to obtain the class activation map for positioning;

[0053] Adopt adaptive threshold to generate mask map;

[0054] The target frame map is determined by using the bounding box to cover the largest connected area in the class activation map to achieve target positioning;

[0055] Among them, the generation of the class activation map used for positioning is as follows:

[0056]

[0057] In the formula, Represents the class activation map corresponding to the feature map of Layer2; Represents the class activation map corresponding to the feature map of Layer3; Represents the class activation map corresponding to the feature map of Layer4; M HR represents the class activation map used for localization; It means element-by-element addition; Represents element-wise multiplication;

[0058] The mask map is calculated as follows:

[0059]

[0060] In the formula, B x,y represents the mask image, (x, y) represents the coordinates; M HR (x, y) represents the pixel value of the class activation map used for positioning at (x, y); th represents the adaptive threshold; th = μ + δ, μ is the mean of the pixel values ​​in the class activation map used for positioning; δ is the variance of the pixel values ​​in the class activation map used for positioning.

[0061] Preferably, the multi-level feature fusion weak supervision detection method further includes:

[0062] Obtain images of targets near the airport from the big data platform;

[0063] Screening the acquired target images;

[0064] Add the target category labels and bounding box annotations of the filtered images to the dataset;

[0065] Based on the updated dataset, the target classification module is retrained;

[0066] Among them, images of targets near the airport are obtained from the big data platform, including:

[0067] Obtain images of targets and their associated positioning information on the big data platform;

[0068] Analyze the positioning information to determine the shooting location corresponding to the image;

[0069] Calculate the distance between the shooting location and the airport; when the distance is less than a preset distance threshold, acquire the image.

[0070] Preferably, before acquiring an image, the validity of the associated positioning information needs to be verified; the verification steps are as follows:

[0071] Determine whether the positioning information is of a preset type;

[0072] When the positioning information is of a preset type, the first time when the image is taken and the second time when the positioning information is uploaded are obtained; when the difference between the first time and the second time is within a preset time threshold range, the verification is passed;

[0073] When the positioning information is not of a preset type, determine the first confidence value of the uploading user and the second confidence value corresponding to the image; when the first confidence value is greater than the preset first threshold or the second confidence value is greater than the preset second threshold, pass the verification; when the first confidence value is between the preset first threshold and the third threshold and the second confidence value is between the preset second threshold and the fourth threshold, pass the verification; wherein the first threshold is greater than the third threshold, and the second threshold is greater than the fourth threshold;

[0074] The first confidence value is determined by the following steps:

[0075] Obtain the first credit value configured for the user by the big data platform;

[0076] Obtain the first evaluation information of other users on the big data platform;

[0077] Determine a first evaluation value of the user by other users based on a preset first scoring template and the first evaluation information;

[0078] Determine the average of the first evaluation values, query a preset correspondence table of evaluation values ​​and second credit values, and determine the second credit value;

[0079] Determining a first confidence value based on the first credit value and the second credit value;

[0080] The second confidence value is determined by the following steps:

[0081] Get the conversion coefficient configured by the big data platform for the user;

[0082] Determining a first relationship value based on the conversion coefficient and the first credit value;

[0083] Obtain the second evaluation information of other users on the big data platform for the image;

[0084] Determining a second evaluation value of the image by other users based on a preset second scoring template and second evaluation information;

[0085] Determine the first credit value of other users whose first credit value is greater than a preset first evaluation threshold or less than a preset second evaluation threshold, and delete the second evaluation value when the first credit value is less than the preset credit threshold;

[0086] Calculate the average of the processed second evaluation values, query a preset correspondence table of the second evaluation values ​​and the second relationship values, and determine the second relationship value;

[0087] Based on the first relationship value and the second relationship value, a second confidence value is determined.

[0088] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0089] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0091] Figure 1 Schematic diagram of a multi-level feature fusion weak supervision detection method in an embodiment of the present invention;

[0092] Figure 2It is a modular schematic diagram corresponding to a weakly supervised detection method of multi-level feature fusion in an embodiment of the present invention;

[0093] Figure 3 Schematic diagram of the coordinate attention mechanism and residual module improvement;

[0094] Figure 4 Schematic diagram of the process of generating class activation maps for localization. DETAILED DESCRIPTION

[0095] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0096] The embodiment of the present invention provides a weakly supervised detection method of multi-level feature fusion, such as Figure 1 As shown, including:

[0097] Step S1: Obtain the image to be analyzed;

[0098] Step S2: Input the image to be analyzed into a preset target classification module, and obtain the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4 in the target classification module;

[0099] Step S3: Based on the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4, generate corresponding class activation maps;

[0100] Step S4: Fuse the class activation maps of Layer 1, Layer 2, and Layer 3 to form a foreground area;

[0101] Step S5: Based on the foreground area and the class activation map of Layer 4, locate the target in the image to be analyzed.

[0102] The working principle and beneficial effects of the above technical solution are:

[0103] By processing the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4 in the target classification module, a class activation map is generated; then the class activation map is fused, and the target is located through the fused class activation map. The image to be analyzed is an image taken in real time by the image acquisition equipment set up in the airport; Figure 2 It is a modular schematic diagram;

[0104] The present invention uses the predicted score of the target category by the target classification module as the starting point of back propagation, and uses the comparative hierarchical correlation propagation rule to obtain the contribution of each position in the shallow, middle and deep feature maps of the network to the network decision, and then uses the class activation mapping algorithm to highlight the important features extracted by the network in the target image in the form of a heat map, and fuses them to generate a high-resolution class activation map to indicate the contour and spatial position of the target, and accurately and efficiently complete the target positioning task. The targets at the airport are generally birds.

[0105] In one embodiment, the target classification module includes:

[0106] Resnet50 network is used as the backbone;

[0107] In the pooling layer, each channel of the given input is encoded according to the horizontal and vertical coordinates to obtain a direction-aware feature map and embed the coordinate information;

[0108] The extracted direction information is concatenated, transformed using 1×1 convolution, decomposed into two separate tensors along the spatial dimension, and made to have the same number of channels using 1×1 convolution, generating a coordinate information mask as the attention weight.

[0109] The attention weight is multiplied pixel by pixel by the input vector as the output.

[0110] The working principle and beneficial effects of the above technical solution are:

[0111] Resnet50 is selected as the backbone network of the target classifier, and the residual module of the network is improved by using the coordinate attention mechanism. The CA-Resnet50 network model is proposed so that the target classification module can effectively extract the key features of the target and improve the fine-grained classification performance of the target. The coordinate attention mechanism and the improved residual network module are shown in Figure 2. Figure 3 As shown. Figure 3 As can be seen in (a), the coordinate attention mechanism can be divided into two steps. The first step is to use the pooling layer to encode each channel of the given input according to the horizontal and vertical coordinates, obtain the direction-aware feature map, and embed the coordinate information. The second step is to concatenate the extracted direction information, transform the information using 1×1 convolution, and then decompose it into two separate tensors along the spatial dimension, and use 1×1 convolution to make them have the same number of channels, generate the coordinate information mask as the attention weight, and finally perform pixel-by-pixel multiplication with the input vector as the output of the module. This attention mechanism can not only capture cross-channel information, but also capture direction-aware and position-aware information. It is lightweight and flexible, and can provide an important computational basis for subsequent weakly supervised target positioning tasks.

[0112] In one embodiment, the target classification module is trained and converged using a data set in advance;

[0113] The dataset is constructed through the following steps:

[0114] According to the types of common targets near the airport, a dataset is created using target category labels and bounding box annotations.

[0115] The working principle and beneficial effects of the above technical solution are:

[0116] By identifying the common types of targets near airports, a dataset is created using target category labels and bounding box annotations to achieve both the applicability and classification efficiency of the classification network.

[0117] In one embodiment, when training the target classification module, a joint loss function is used to guide the training; the joint loss function is as follows:

[0118] L=L ce +λL bp ;

[0119] Where L represents the joint loss function; L ce represents the cross entropy loss function; L bp represents the perceptual loss function; λ represents the preset hyperparameter for controlling the weight of the perceptual loss function;

[0120] The perceptual loss function is as follows:

[0121]

[0122] Where N represents the number of image samples; M FA represents the class activation mask in the class activation map fusion process; M HR Represents the activation mask generated by all features of the last convolutional layer of the target classification module;

[0123] The cross entropy loss function is as follows:

[0124]

[0125] In the formula, g indicates the category; G indicates the total number of categories; u jg Represents the symbolic function. When the true category of the image sample j is equal to g, u jg =1; otherwise, u jg =0;v jg Represents the probability that image sample j belongs to category g.

[0126] The working principle and beneficial effects of the above technical solution are:

[0127] In order to improve the model's target detection performance and constrain the features extracted by the network, a spatial-level perception loss function is proposed on the basis of the target classification module, and it is introduced into the fine-grained classification task of the target to suppress the background noise in the image. While improving the model's feature extraction ability, a multi-level feature fusion strategy can be used to generate a higher-quality class activation map, further improving the target positioning performance; in the training process of the target classification module, the proposed perception loss function and the cross entropy loss function are used as joint loss functions to jointly supervise the network training process, improve the network target classification performance, and further generate high-quality class activation maps, improving the performance of the network weakly supervised target positioning. Among them, the classification accuracy and positioning accuracy are used to evaluate the performance of the trained target classification module. When the weight parameter λ=1.5, the target classification accuracy and weakly supervised positioning accuracy are increased by 1.64% and 5.40% respectively, and the network performance reaches the best.

[0128] In one embodiment, based on the feature map of Layer 1, a corresponding class activation map is generated, including:

[0129] The calculation steps of the class activation map are as follows:

[0130]

[0131]

[0132] In the formula, A represents the class activation map corresponding to the feature map of Layer 1; k Represents the kth feature map of Layer1; It represents the contribution of position (x, y) in the k-th feature map of Layer1 to the target category c obtained by back propagation using contrastive hierarchical correlation propagation theory. Represents the kth feature map A of Layer1 k Channel-wise weights for target category c.

[0133] The working principle and beneficial effects of the above technical solution are:

[0134] The hierarchical correlation propagation theory and class activation mapping algorithm are used to obtain the contribution of each position in the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4 of the Resnet50 network to the target classification, and the corresponding class activation map is generated; the generation steps of the class activation map of Layer 2, Layer 3 and Layer 4 are the same as those of Layer 1, so they will not be repeated here.

[0135] In order to realize the generation of the class activation map, in one embodiment, the contribution of the position (x, y) in the k-th feature map of Layer 1 obtained by back-propagation of the contrast hierarchical correlation propagation theory to the target category c includes:

[0136] Use contrastive hierarchical correlation propagation to obtain the contribution of each position in the feature map to the target classification decision;

[0137] Among them, the comparison of hierarchical correlation propagation includes:

[0138] Convolutional neural networks are composed of several nested nonlinear functions. While they have high nonlinearity and excellent learning ability, it is difficult to explain the decisions made by the network. To this end, a comparative hierarchical correlation propagation theory is proposed, which uses correlation scores to measure the contribution of neurons to model decisions, which provides an important theoretical basis for weakly supervised target localization tasks.

[0139] The network output value before the initialization softmax operation is used to distinguish the ratio of target category neurons and non-target category neurons, and serves as the starting point for back propagation of comparison hierarchical correlation. The calculation steps of the initialization function are as follows:

[0140]

[0141] In the formula, represents the score corresponding to the i-th neuron in the l-th layer before the Softmax operation, and P represents the total number of neurons in the l-th layer;

[0142] In the back propagation phase of contrast hierarchical correlation, the correlation score between each variable and the network decision is measured by decomposing from the output layer of the network to each variable in the input space;

[0143] The commonly used hierarchical correlation propagation rule is z + and z β The rules are defined as follows:

[0144]

[0145]

[0146] In the formula, and They represent the positive and negative weights connecting the neurons in the lth and l+1th layers respectively; [U, V] represents the value range of the neuron activation value.

[0147] In one embodiment, when determining the correlation score, it is also standardized, and the value of the correlation score is fixed in the interval [0-1]. The standardized calculation formula is as follows:

[0148]

[0149] In the formula, is the correlation score of the position (x,u) in the k-th feature map of Layer1 before normalization with respect to the target category c; is the set of relevance scores for each position in the kth feature map with respect to the target category c.

[0150] In one embodiment, Figure 4 As shown in the figure, based on the foreground area and the class activation map of Layer 4, the target in the image to be analyzed is located, including:

[0151] Multiply the foreground area by the class activation map of Layer 4 element by element to obtain the class activation map for positioning;

[0152] Adopt adaptive threshold to generate mask map;

[0153] The target frame map is determined by using the bounding box to cover the largest connected area in the class activation map to achieve target positioning;

[0154] Among them, the generation of the class activation map used for positioning is as follows:

[0155]

[0156] In the formula, Represents the class activation map corresponding to the feature map of Layer2; Represents the class activation map corresponding to the feature map of Layer3; Represents the class activation map corresponding to the feature map of Layer4; M HR represents the class activation map used for localization; It means element-by-element addition; Represents element-wise multiplication;

[0157] The mask map is calculated as follows:

[0158]

[0159] In the formula, B x,y represents the mask image; (x, y) represents the coordinates; M HR (x, y) represents the pixel value of the class activation map used for positioning at (x, y); th represents the adaptive threshold; th = μ + δ, μ is the mean of the pixel values ​​in the class activation map used for positioning; δ is the variance of the pixel values ​​in the class activation map used for positioning.

[0160] The working principle and beneficial effects of the above technical solution are:

[0161] The high-resolution class activation map fusion strategy is divided into two stages: the generation and fusion of multi-level class activation maps. In the first stage, this paper uses the hierarchical correlation propagation theory and the class activation mapping algorithm to obtain the contribution of each position in the feature map of Layer 1, Layer 2, Layer 3 and Layer 4 of the Resnet50 network to the target classification, generates the corresponding class activation map, and upsamples it to the same size as the input image. Among them, the generation of the class activation map belongs to the first stage; in the second stage, the class activation maps of Layer 1, Layer 2 and Layer 3, which are rich in detail information such as target contour and texture, are fused in a direct fusion manner to obtain the foreground area of ​​the target. However, this process also highlights the background noise that is irrelevant to the target, and the target cannot be accurately located. To this end, the class activation map of Layer 4, which is rich in target abstract semantic information, is multiplied element by element with the fused class activation map to suppress the background noise, generate a class activation map that can effectively indicate the target contour and position, and use the adaptive threshold to generate the target mask map, and then use the bounding box to cover the largest connected area in the class activation map to determine the target bounding box and achieve target positioning.

[0162] In one embodiment, the multi-level feature fusion weak supervision detection method further includes:

[0163] Obtain images of targets near the airport from the big data platform;

[0164] Screening the acquired target images;

[0165] Add the target category labels and bounding box annotations of the filtered images to the dataset;

[0166] Based on the updated dataset, the target classification module is retrained;

[0167] Among them, images of targets near the airport are obtained from the big data platform, including:

[0168] Obtain images of targets and their associated positioning information on the big data platform;

[0169] Analyze the positioning information to determine the shooting location corresponding to the image;

[0170] Calculate the distance between the shooting location and the airport; when the distance is less than a preset distance threshold, acquire the image.

[0171] The working principle and beneficial effects of the above technical solution are:

[0172] By acquiring images of targets near the airport from the big data platform, the data set is updated so that the target classification module can always adapt to the airport and continuously ensure the flight safety of the aircraft; the acquired target images are screened and mainly classified by the existing target classification module. When the classification can be completed, there is no need to add them to the data set; or, the similarity between the acquired image and the original image in the data set is calculated. When the similarity is greater than the preset similarity threshold (for example: 0.98), there is no need to add it, otherwise, add it for update. In addition, the location where the image was taken is determined through the associated positioning information to ensure that the image is acquired near the airport, so as to achieve targeted updates.

[0173] In one embodiment, before acquiring an image, the validity of the associated positioning information needs to be verified; the verification steps are as follows:

[0174] Determine whether the positioning information is of a preset type; the preset type is the data type sent by the positioning module of the shooting device, that is, when the image on the big data platform is shot by the shooting device and uploaded in real time, if the verification is passed, it means that the positioning information is the location of the shooting location corresponding to the image;

[0175] When the positioning information is of a preset type, the first time when the image is taken and the second time when the positioning information is uploaded are obtained; when the difference between the first time and the second time is within the preset time threshold range (for example, 3 minutes), the verification is passed; if it is uploaded in real time, the first time and the second time are generally the same, so a threshold is set to prevent the delay upload caused by the failure of the shooting equipment from confusing the real-time uploaded data;

[0176] When the positioning information is not of a preset type, determine the first confidence value of the uploading user and the second confidence value corresponding to the image; when the first confidence value is greater than the preset first threshold or the second confidence value is greater than the preset second threshold, pass the verification; when the first confidence value is between the preset first threshold and the third threshold and the second confidence value is between the preset second threshold and the fourth threshold, pass the verification; wherein the first threshold is greater than the third threshold, and the second threshold is greater than the fourth threshold; for example: when the photographer uploads the image through a computer terminal, the positioning information is manually input; the positioning information is the address information of the shooting;

[0177] The first confidence value is determined by the following steps:

[0178] Obtaining a first credit value configured for the user by the big data platform; the first credit value may be assigned by the staff of the big data platform according to the authenticity and validity of the information uploaded by the user on the big data platform, and is the evaluation of the user by the big data platform;

[0179] Obtain the first evaluation information of other users on the big data platform;

[0180] Based on the preset first scoring template and the first evaluation information, the first evaluation value of the user by other users is determined; the first evaluation information can be divided into four levels, namely very unreliable, reliable, very reliable, and trustworthy; the corresponding scores of each level are -2, 1, 2, and 3, respectively.

[0181] Determine the average of the first evaluation values, query a preset correspondence table of evaluation values ​​and second credit values, and determine the second credit value;

[0182] Based on the first credit value and the second credit value, determining a first confidence value; the first confidence value is the sum of the first credit value and the second credit value;

[0183] The second confidence value is determined by the following steps:

[0184] Obtaining a conversion coefficient configured for the user by the big data platform; the conversion coefficient is a relationship coefficient between the first credit value and the first relationship value;

[0185] Determining a first relationship value based on the conversion coefficient and the first credit value;

[0186] Obtain the second evaluation information of other users on the big data platform for the image;

[0187] Determining a second evaluation value of the image by other users based on a preset second scoring template and second evaluation information;

[0188] Determine the first credit value of other users whose first credit value is greater than a preset first evaluation threshold or less than a preset second evaluation threshold, and delete the second evaluation value when the first credit value is less than the preset credit threshold;

[0189] Calculate the average of the processed second evaluation values, query a preset correspondence table of the second evaluation values ​​and the second relationship values, and determine the second relationship value;

[0190] Based on the first relationship value and the second relationship value, a second confidence value is determined. The second confidence value is a sum of the first relationship value and the second relationship value.

[0191] The working principle and beneficial effects of the above technical solution are:

[0192] When the image on the big data platform is uploaded by a shooting device in real time, the associated positioning information is the location of the image shooting, and the type of positioning information is used to distinguish whether it is shot in real time; when the image on the big data platform is uploaded by the user using a computer terminal afterwards, it is necessary to screen the image according to the user's first confidence value and the image's second confidence value; when determining the first confidence value, the user's own and other users' evaluations are comprehensively considered to ensure the accuracy of the first confidence value; when determining the second confidence value of the image, the evaluations of other users of the uploading user are also comprehensively considered; this realizes the determination of whether the image is accurate based on the recognition of the user himself and other users, thereby ensuring the accuracy of the data set update.

[0193] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A weakly supervised detection method with multi-level feature fusion, characterized in that: include: Get the image to be analyzed; Input the image to be analyzed into a preset target classification module, and obtain the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4 in the target classification module; Based on the feature maps of Layer 1, Layer 2, Layer 3 and Layer 4, generate corresponding class activation maps; The class activation maps of Layer 1, Layer 2, and Layer 3 are merged to form a foreground area; Based on the foreground area and the class activation map of Layer 4, locating the target in the image to be analyzed; Wherein, based on the feature map of the Layer 1 layer, the corresponding class activation map is generated, including: The calculation steps of the class activation map are as follows: In the formula, Represents the class activation map corresponding to the feature map of Layer1; Indicates the first feature maps; Represents the first layer of Layer1 obtained by backpropagation using contrast hierarchical correlation propagation theory Position in feature map About target categories Contribution Indicates the first Feature Map About target categories The channel-level weights of Among them, the first layer of Layer1 obtained by backpropagation using contrast level correlation propagation theory Position in feature map About target categories Contributions include: Use contrastive hierarchical correlation propagation to obtain the contribution of each position in the feature map to the target classification decision; Among them, the comparison of hierarchical correlation propagation includes: The network output value before the initialization softmax operation is used to distinguish the ratio of target category neurons and non-target category neurons, and serves as the starting point for back propagation of comparison hierarchical correlation. The calculation steps of the initialization function are as follows: In the formula, Indicates the first Layer The score corresponding to each neuron is Indicates The total number of neurons in the layer; In the back propagation phase of contrast hierarchical correlation, the correlation score between each variable and the network decision is measured by decomposing from the output layer of the network to each variable in the input space; The commonly used hierarchical correlation propagation rules are and The rules are defined as follows: In the formula, and Respectively represent the connection and The positive and negative weights of the layer neurons; Represents the value range of neuron activation value.

2. The multi-level feature fusion weak supervision detection method according to claim 1, characterized in that: The target classification module comprises: Resnet50 network is used as the backbone; In the pooling layer, each channel of the given input is encoded according to the horizontal and vertical coordinates to obtain a direction-aware feature map and embed the coordinate information; The extracted direction information is concatenated, transformed using 1×1 convolution, decomposed into two separate tensors along the spatial dimension, and made to have the same number of channels using 1×1 convolution, generating a coordinate information mask as the attention weight. The attention weight is multiplied pixel by pixel by the input vector as the output.

3. The multi-level feature fusion weak supervision detection method according to claim 1, characterized in that: The target classification module is obtained by training and convergence using a data set in advance; The data set is constructed through the following steps: According to the types of common targets near the airport, a dataset is created using target category labels and bounding box annotations.

4. The multi-level feature fusion weak supervision detection method according to claim 1, characterized in that: When training, the target classification module uses a joint loss function to guide the training; The joint loss function is as follows: ; In the formula, represents the joint loss function; represents the cross entropy loss function; represents the perceptual loss function; Represents the preset hyperparameters that control the weights of the perception loss function; The perceptual loss function is as follows: ; In the formula, Indicates the number of image samples; Represents the class activation mask in the class activation map fusion process; Represents the activation mask generated by all features of the last convolutional layer of the target classification module; The cross entropy loss function is as follows: ; In the formula, Indicate the category; Indicates the total number of categories; Represents a symbolic function, when the image sample The true category is equal to , ; otherwise, ; Indicates image sample Belongs to category probability.

5. The multi-level feature fusion weak supervision detection method according to claim 1, characterized in that: When determining the relevance score, it is also standardized to fix the value of the relevance score in the interval The standardized calculation formula is as follows: ; In the formula, is the first layer of Layer 1 before normalization Position in feature map About target categories the relevance score of For the Each position in the feature map has a relation to the target category A set of said relevance scores.

6. The multi-level feature fusion weak supervision detection method according to claim 1, characterized in that: The positioning of the target in the image to be analyzed based on the foreground area and the class activation map of Layer 4 includes: Multiply the foreground area by the class activation map of Layer 4 element by element to obtain a class activation map for positioning; Adopt adaptive threshold to generate mask map; The target frame map is determined by using the bounding box to cover the largest connected area in the class activation map to achieve target positioning; Among them, the generation of the class activation map used for positioning is as follows: ； In the formula, Represents the class activation map corresponding to the feature map of Layer2; Represents the class activation map corresponding to the feature map of Layer3; Represents the class activation map corresponding to the feature map of Layer4; represents the class activation map used for localization; It means element-by-element addition; Represents element-wise multiplication; The mask map is calculated as follows: ; In the formula, represents the mask image, Represents coordinates; Represents the class activation map used for positioning in The pixel value of represents the adaptive threshold; , is the mean of the pixel values ​​in the class activation map used for positioning; is the variance of pixel values ​​in the class activation map used for localization.

7. The multi-level feature fusion weak supervision detection method according to claim 3, characterized in that: Also includes: Obtain images of targets near the airport from the big data platform; Screening the acquired target images; The filtered images are labeled with target categories and annotated with bounding boxes and added to the dataset; Retraining the target classification module based on the updated data set; Among them, images of targets near the airport are obtained from the big data platform, including: Acquire an image of a target on the big data platform and its associated positioning information; Analyze the positioning information to determine the shooting position corresponding to the image; The distance between the shooting location and the airport is calculated; when the distance is less than a preset distance threshold, the image is acquired.

8. The multi-level feature fusion weak supervision detection method according to claim 7, characterized in that: Before acquiring the image, the validity of the associated positioning information needs to be verified; the verification steps are as follows: Determining whether the positioning information is of a preset type; When the positioning information is of a preset type, obtaining a first time when the image is captured and a second time when the positioning information is uploaded; when the difference between the first time and the second time is within a preset time threshold range, passing the verification; When the positioning information is not of a preset type, determining a first confidence value of the uploading user and a second confidence value corresponding to the image; when the first confidence value is greater than a preset first threshold or the second confidence value is greater than a preset second threshold, passing verification; when the first confidence value is between a preset first threshold and a third threshold and the second confidence value is between a preset second threshold and a fourth threshold, passing verification; wherein the first threshold is greater than the third threshold, and the second threshold is greater than the fourth threshold; Wherein, the first confidence value is determined by the following steps: Obtaining a first credit value configured by the big data platform for the user; Obtaining first evaluation information of the user by other users on the big data platform; Determine a first evaluation value of the user by other users based on a preset first scoring template and the first evaluation information; Determine the average of the first evaluation values, query a preset correspondence table of evaluation values ​​and second credit values, and determine the second credit value; Determining the first confidence value based on the first credit value and the second credit value; The second confidence value is determined by the following steps: Obtaining a conversion coefficient configured by the big data platform for the user; determining a first relationship value based on the conversion coefficient and the first credit value; Obtaining second evaluation information of the image by other users on the big data platform; Determining a second evaluation value of the image by other users based on a preset second scoring template and the second evaluation information; Determine the first credit value of other users whose first credit value is greater than a preset first evaluation threshold or less than a preset second evaluation threshold, and when the first credit value is less than the preset credit threshold, delete the second evaluation value; Calculate the average of the processed second evaluation values, query a preset correspondence table of second evaluation values ​​and second relationship values, and determine the second relationship value; Based on the first relationship value and the second relationship value, the second confidence value is determined.

Images