Feature map self-supervised dense crowd small target detection method and electronic equipment

Abstract

The application provides a feature map self-supervised dense crowd small target detection method and electronic equipment, the feature map self-supervised dense crowd small target detection method includes steps: S1, using target detection network to carry out multi-scale feature extraction to training image, and generate the feature map of different levels;S2, the first stage uses the main loss function of detection task to pretrain target detection network;S3, the second stage joins the self-supervised loss function and carries out joint training target detection network, wherein including steps: S3.1, the feature map of adjacent level is calculated structure similarity index SSIM as self-supervised signal;S3.2, based on the structure similarity index SSIM constructs self-supervised loss function and forcedly restricts deep feature map to retain the detail information of shallow feature map, and the main loss function of detection task is combined to optimize network parameters.The feature map self-supervised dense crowd small target detection method and electronic equipment of the application can comprehensively improve the detection performance of small target in dense crowd.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of dense crowd small target detection, and more particularly to a feature map self-supervised dense crowd small target detection method and electronic equipment. BACKGROUND

[0002] As a basic algorithm of computer vision, target detection aims to determine the position coordinate information of the target of interest in an image, and provides strong support for subsequent target analysis, so the accuracy of detection directly affects the feasibility of subsequent analysis results.

[0003] However, in the real situation of a dense crowd environment, the human target is not only small, but also limited by occlusion, light, motion blur, low image resolution, etc., which directly leads to the small human target carrying less information in the region of interest, and is easy to cause missed detection and false detection. SUMMARY

[0004] The purpose of the embodiment of the application is to provide a feature map self-supervised dense crowd small target detection method and electronic equipment to solve the technical problem of insufficient detection capability in the prior art in the process of detecting dense crowd small targets.

[0005] To achieve the above purpose, the technical solution adopted by the application is to provide a feature map self-supervised dense crowd small target detection method, which comprises the following steps:

[0006] S1, using a target detection network to perform multi-scale feature extraction on a training image to generate feature maps of different levels;

[0007] S2, using a main loss function of a detection task to pre-train the target detection network in the first stage;

[0008] S3, in the second stage, a self-supervised loss function is added to jointly train the target detection network, comprising the following steps:

[0009] S3.1, calculating the structural similarity index SSIM of the adjacent level feature maps as a self-supervised signal;

[0010] S3.2, constructing a self-supervised loss function based on the structural similarity index SSIM to force the deep feature maps to retain the detail information of the shallow feature maps, and jointly optimizing the network parameters with the main loss function of the detection task.

[0011] In a preferred embodiment, the formula for calculating the structural similarity index SSIM of the adjacent level feature maps is:

[0012] SSIM(F1, F2) = (l(F1, F2) x c(F1, F2) x s(F1, F2))α

[0013] wherein, F1, F2 represent two feature maps of adjacent levels, SSIM(F1, F2) represents the structural similarity index SSIM of two feature maps of adjacent levels, l(F1, F2) represents the brightness similarity between two images, c(F1, F2) represents the contrast similarity of two images, s(F1, F2) represents the structural similarity between two images, and a is a dependent variable, which is proportional to the crowd density of the target population.

[0014] In a preferred embodiment, the method for calculating the structural similarity index SSIM of feature maps of adjacent levels comprises the steps of:

[0015] S3.11, calculating the mean value of the feature maps of each level along the channel dimension to generate a single-channel mean feature map F mean (x, y), and the calculation formula is:

[0016] ,

[0017] wherein, C is the number of feature maps of the corresponding level, F i (x, y) is the value of the pixel coordinate (x, y) in the i-th feature map;

[0018] S3.12, spatially aligning the mean feature maps of adjacent levels.

[0019] In a preferred embodiment, the method for calculating the structural similarity index SSIM of feature maps of adjacent levels further comprises the steps of:

[0020] S3.13, dividing the spatially aligned mean feature maps into s x s grids, respectively applying different weights to each grid, and calculating the structural similarity index SSIM by weighting.

[0021] In a preferred embodiment, the grid weight is dynamically adjusted according to the center point position of the real annotation box, and the weight of the grid where the center of the annotation box falls is increased.

[0022] In a preferred embodiment, the calculation formula of the weight coefficient k of each grid is:

[0023]

[0024] wherein, n is the number of real annotation boxes in the mean feature map, and m is the number of center points of real annotation boxes in the corresponding grid.

[0025] In a preferred embodiment, the calculation formula of the weight coefficient k of each grid is:

[0026]

[0027] wherein n is the number of real annotation boxes of the current image, m is the number of center points of the real annotation boxes in the corresponding grid, a is the basic weight of each grid, and a is a positive integer.

[0028] In a preferred embodiment, the s takes the value of 4, and the a takes the value of 1.

[0029] In a preferred embodiment, the method for obtaining the self-supervised loss function L comprises the steps of:

[0030] S3.21, the structural similarity index SSIM is uniformly adjusted to [-1, 1], and the closer to 1 indicates that the structural similarity of the two images is higher, and the value closer to -1 indicates that the structural similarity is lower;

[0031] S3.22, the formula for calculating the self-supervised loss function L is:

[0032] ,

[0033] wherein X = 1-SSIM(F1, F2).

[0034] In a preferred embodiment, after constructing the self-supervised loss function, the method further comprises the steps of:

[0035] Suppose the feature extraction layers of the target detection network are T1 to Tn in turn. n ;

[0036] Based on the analysis of the structural similarity index SSIM of the feature maps of adjacent layers, the feature extraction layer in which the boundary structure of the small target in the dense crowd occurs qualitative change is determined as Tm. m wherein 1 < m < n.

[0037] The total self-supervised loss function L is established as: total = L 1&2 + L 2&3 + … + L （m-1）&m wherein L 1&2 represents the self-supervised loss of the feature map of the feature extraction layer T1 and the feature map of the feature extraction layer T2, L 2&3 represents the self-supervised loss of the feature map of the feature extraction layer T2 and the feature map of the feature extraction layer T3, L （m-1）&m represents the self-supervised loss of the feature map of the feature extraction layer Tm-1 and the feature map of the feature extraction layer Tm. m-1 m

[0038] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method as described above when executing the computer program.

[0039] ​​The feature map self-supervision dense crowd small target detection method and the electronic equipment provided by the application have the beneficial effects that: by adopting the two-stage training mode, the low-level consistency constrained by the self-supervision loss function and the high-level features constrained by the main loss function form a clear division of labor, synergistic effect, and complementary cooperation relationship, and the shallow detail information is forced to be reserved in the deep network, so as to solve the information attenuation and gradient imbalance problem in small target detection, reduce the missed detection and false detection probability, and comprehensively improve the detection performance of small targets in dense crowds. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 The flowchart of the feature map self-supervision dense crowd small target detection method provided by the embodiments of the application;

[0042] Figure 2 The schematic diagram of the Mosaic data enhancement provided by the embodiments of the application;

[0043] Figure 3 The flowchart of step S3 provided by the embodiments of the application;

[0044] Figure 4 The schematic diagram of dividing the spatially aligned mean feature map into s x s grid provided by the embodiments of the application;

[0045] Figure 5 The flowchart of establishing the total self-supervision loss function provided by the embodiments of the application;

[0046] Figure 6 The schematic diagram of the target detection network provided by the embodiments of the application;

[0047] Figure 7 The visualization schematic diagram of each level feature map output by the target detection network in the first stage in Figure 6

[0048] The visualization schematic diagram of each level feature map output by the target detection network in the second stage in Figure 8 Figure 6 DETAILED DESCRIPTION

[0049] ​​In order to make the technical problems, technical solutions and beneficial effects of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0050] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0051] It should be understood that the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0052] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly and specifically limited.

[0053] Please refer to Figure 1 , now the feature map self-supervised dense crowd small target detection method provided by the embodiment of the present application will be described. The feature map self-supervised dense crowd small target detection method comprises the steps of:

[0054] S1, using a target detection network to perform multi-scale feature extraction on a training image to generate feature maps of different levels.

[0055] In S1, the present embodiment preferably uses CNN, YOLOX and SSD as the basic detection network. In another embodiment, the YOLOv8 network architecture in real-time dense detection scenarios, the DETR network architecture in complex occlusion scenarios, etc. can also be adaptively replaced.

[0056] The training image size of the present embodiment is 640x640. In another embodiment, the training image size of 640x640 can also be adaptively replaced by 320x320, 1280x1280, etc.

[0057] When performing multi-scale feature extraction on a training image of 640x640, feature maps of 320x320, 160x160, 80x80, 40x40, and 20x20 can be obtained step by step.

[0058] In S2, the target detection network is pre-trained using a main loss function of a detection task in the first stage.

[0059] In S2, it can be understood that the training of the first stage using the main loss function of the detection task is essentially a conventional supervised training, without adding a self-supervised loss, and only through conventional supervised training can the network parameters be preliminarily converged.

[0060] Please refer to Figure 2 , for example, pre-training using the COCO dataset, using Mosaic data augmentation, which includes random scaling, cropping, flipping, etc. The number of training rounds can be determined according to actual conditions, and is usually set to 10 to 30 rounds. Preferably, the number of training rounds is set to 15 rounds, the optimizer is SGD, the initial learning rate is 0.01, and the cosine annealing strategy is used to adjust the learning rate. The main loss function includes target detection loss, such as Obj Loss, classification loss, and IoU Loss.

[0061] In this way, the essence of the first stage is to take advantage of the direct optimization of the task target of supervised learning, to let the target detection network directly learn the mapping relationship from input to output through labeled data (such as classification labels and detection boxes), to clearly define the task target, and to preliminarily and quickly learn the basic human detection capability of the target detection network. Although the pre-trained target detection network has the disadvantages of limited generalization ability (if the training data distribution is different from the real scene, such as annotation bias, the model performance may decrease), difficulty in handling unknown classes (unable to automatically identify classes not present in the training set), and single feature level (classification labels and detection boxes are both abstract semantic information, i.e., only high-level features).

[0062] It is worth noting that the classification of the feature level includes:

[0063] (1) Low-level features (edges / textures):

[0064] Small targets (such as pedestrians with a size of less than 20x20 pixels) have very little pixel information, and their recognizability is extremely dependent on edge sharpness (such as limb outlines) and local textures (such as clothing wrinkles). These information belong to low-level visual features.

[0065] (2) Mid-level features (geometric structures):

[0066] The local geometric relationship of small targets (such as the relative position of the head and the torso) is the key to distinguishing targets from noise, and belongs to mid-level features.

[0067] (3) High-level features (semantics):

[0068] The semantic features extracted by the deep network (such as the "person" and "car" categories labeled by the detection box) are often ineffective for small targets, because the small target may only have a few pixels in the deep feature map, and the semantic information has been severely lost.

[0069] In S3, the second stage also adds a self-supervised loss function to jointly train the target detection network, including the following steps:

[0070] S3.1, calculate the structural similarity index SSIM as a self-supervised signal for the feature maps of adjacent levels;

[0071] S3.2, based on the structural similarity index SSIM, construct a self-supervised loss function to force the deep feature map to retain the detail information of the shallow feature map, and jointly optimize the network parameters with the main loss function of the detection task.

[0072] In S3, the second stage also adds a self-supervised loss function to jointly train the target detection network, that is, the training process of the second stage simultaneously exists under the supervision of the main loss function and the self-supervision of the self-supervised loss function.

[0073] Specifically, in S3.1, the structural similarity index SSIM (Structural Similarity) is calculated as a self-supervised signal for the feature maps of adjacent levels, where the structural similarity index SSIM is an index for evaluating the visual similarity of two images. It not only considers the brightness similarity and contrast similarity of the image, but also considers the structural similarity of the overall image.

[0074] It is worth noting that the brightness similarity aims to ensure the consistency of the overall intensity distribution of the target, and the contrast similarity aims to preserve the edge and texture differences of the target, where the overall intensity distribution, edge, and texture are all low-level features. The structural similarity aims to ensure that the key geometric structure (such as the outline of a small target) is not destroyed, where the geometric structure and local target outline are combined features belonging to middle-level features. That is, the structural similarity index SSIM can force the alignment of adjacent level feature maps in the following dimensions: edge consistency (low-level), texture consistency (low-level), and geometric consistency (middle-level).

[0075] In this way, when jointly optimizing the network parameters with the main loss function of the detection task in S3.2, the middle-low level consistency constrained by the self-supervised loss function and the high-level features constrained by the main loss function form the following cooperative relationship:

[0076] (1) Clear division of labor: the self-supervised loss function uses the self-supervised loss function to force constrain the deep feature map to retain the detail information of the shallow feature map, force constrain the consistency of low-level features and middle-level features, and is responsible for retaining the "physical existence" (edge / shape) of the target, solving the problem of "where is the target" (positioning). The semantic features under the constraint of the main loss function are responsible for judging the "identity" (category) of the target, solving the problem of "what is it" (classification).

[0077] (2) Synergistic effect: in the target detection process, the structure-enhanced feature map under the constraint of the self-supervised loss function provides accurate position information, while the deep semantic features under the constraint of the main loss function provide category confidence, and the combination of the two can simultaneously reduce missed detection (relying on structure) and false detection (relying on semantics).

[0078] (3) complementary advantages: first, use the main loss function of the detection task to pre-train the target detection network, so that the network can preliminarily learn the basic detection ability (such as positioning and classification of large targets), and preliminarily narrow the gap between adjacent feature maps. On the basis of keeping the network parameters relatively stable, the self-supervised loss function is introduced again to prevent training convergence difficulties or "gradient explosion" and other situations, and to avoid gradient conflicts caused by early introduction of self-supervision.

[0079] In this way, the feature map self-supervised dense crowd small target detection method provided by the present application can form a cooperative relationship of clear division of labor, synergistic effect, and complementary advantages between the low-level consistency constrained by the self-supervised loss function and the high-level features constrained by the main loss function, and can force constrain the explicit retention of the details of the shallow layer in the deep network, so as to solve the information attenuation and gradient imbalance problems in small target detection, reduce the missed detection and false detection probabilities, and comprehensively improve the detection performance of small targets in dense crowds.

[0080] Since in a sparse personnel scene, the whole human body can usually be seen; while in a dense crowd, the human body occlusion problem will be infinitely magnified, and most human body targets only exist as a head, which will cause the human body target detection to be missed. To solve the above problems, the following embodiments are adopted in the present application:

[0081] The formula for calculating the structural similarity index SSIM of adjacent levels of feature maps is:

[0082] SSIM(F1, F2) = (l(F1, F2) x c(F1, F2) x s(F1, F2)) α

[0083] Wherein, F1, F2 represent two feature maps of adjacent levels, SSIM(F1, F2) represents the structural similarity index SSIM of two feature maps of adjacent levels, l(F1, F2) represents the brightness similarity between two images, c(F1, F2) represents the contrast similarity of two images, s(F1, F2) represents the structural similarity between two images, and a is a dependent variable, which is proportional to the target crowd density.

[0084] It can be understood that the structural similarity index SSIM of the embodiment is different from the calculation method of the conventional structural similarity index SSIM, l(F1, F2) x c(F1, F2) x s(F1, F2) ∈ [0, 1], and the closer to 1 represents the higher structural similarity of two feature maps of adjacent levels, and the closer to 0 represents the lower structural similarity of two feature maps of adjacent levels. Since a is proportional to the target crowd density, when the crowd is dense, a also increases, resulting in the decrease of SSIM(F1, F2), and the decrease of SSIM(F1, F2) is fed back to the self-supervised loss function, that is, the constraint that the deep feature map retains the detail information of the shallow feature map is strengthened; similarly, when the crowd is sparse, the constraint that the deep feature map retains the detail information of the shallow feature map is weakened. In this way, the adaptive scene is achieved, the constraint degree is dynamically adjusted, and the improvement of small target detection performance and training efficiency is considered.

[0085] Since the number of feature maps under each level is inconsistent, and the feature map scales of different levels are inconsistent. However, the structural similarity index SSIM generally measures the structural similarity between two images, so there are problems of large calculation amount and inaccuracy when calculating the structural similarity index SSIM of the feature maps of adjacent levels. To solve the above problems, the following embodiment is adopted in the present application:

[0086] Please refer to Figure 3 The method for calculating the structural similarity index SSIM of the feature maps of adjacent levels comprises the following steps:

[0087] S3.11, calculating the mean value of the feature maps of each level along the channel dimension to generate a single-channel mean feature map F mean (x, y), and the calculation formula is:

[0088] ,

[0089] Wherein, C is the number of feature maps of the corresponding level, F i (x, y) is the value of the pixel coordinate (x, y) in the i-th feature map;

[0090] S3.12, spatially aligning the mean feature maps of adjacent levels.

[0091] It can be understood that the single-channel mean feature map F of each level is calculated by the above formula in the embodiment mean (x, y), so that the feature maps at each scale are unified. Then, the mean feature maps of adjacent levels are spatially aligned, and thus two images of the same size can be obtained, each of which can represent the feature structure of the level it belongs to. In this way, the technical barriers of large calculation amount and inaccuracy when calculating the structural similarity index SSIM using the feature maps of adjacent levels are solved.

[0092] Please refer to Figure 3 and Figure 4 Further, the method for calculating the structural similarity index SSIM of the feature maps of adjacent levels further comprises the steps of:

[0093] S3.13. Divide the spatially aligned mean feature maps into s x s grids, and apply different weights to each grid respectively to calculate the structural similarity index SSIM.

[0094] It can be understood that the small target crowd image is usually formed by long-distance shooting, so when different weights are applied to each grid respectively, the weights of the grids in the upper region of the image can be increased and / or the weights of the grids in the lower region of the image can be reduced to enhance the attention of small targets at a distance. Of course, the weights of the grids in the central region of the image can also be increased and / or the weights of the grids in the edge region of the image can be reduced according to experience values to enhance the attention of small targets in the central region of interest and reduce the attention of the edge background region.

[0095] In this way, when the structural similarity index SSIM is calculated using the two spatially aligned mean feature maps, the factor of large overall pixel value variance of the two images can be overcome, the crowd small target detail supervision capability of the feature map can be improved, and further the detection performance of the dense crowd small target can be improved.

[0096] In a preferred embodiment, the grid weights are dynamically adjusted according to the center point positions of the real annotation boxes, and the weights of the grids where the center of the annotation box falls are increased.

[0097] Since the annotation boxes are set for the training images in the pre-training process, in the embodiment, the grid weights can be directly adjusted adaptively and dynamically according to the center points of the real human small target annotation boxes, the grid weights of the crowd dense grids are increased, and the grid weights of the crowd sparse grids are reduced, so that the image of any scene can be accurately adapted, the crowd small target detail supervision capability of the feature map can be optimized, and further the detection performance of the dense crowd small target can be improved.

[0098] In a specific embodiment, the calculation formula of the weight coefficient k of each grid is:

[0099]

[0100] wherein n is the number of real labeled boxes in the mean feature map, and m is the number of center points of the real labeled boxes in the corresponding grid.

[0101] It can be understood that, based on the calculation formula of the weight coefficient k in the embodiment, only the grid containing the center point of the labeled box will obtain the weight, and the remaining grids have a weight of 0. Therefore, the embodiment has the following effects:

[0102] (1) The self-supervised loss completely ignores the background area, and only forces the feature map around the target center to maintain structural consistency, thereby improving the constraint force of the deep feature map to retain the detailed information of the shallow feature map.

[0103] (2) Since the grid weight of the center point of the real labeled box is 0, only part of the grid needs to be calculated for SSIM, thereby reducing the computational amount, improving the training efficiency, and accelerating the convergence of the self-supervised loss function.

[0104] (3) The gradient is completely driven by the target area, thereby improving the anti-interference ability.

[0105] However, since only the grid containing the target is calculated for loss in the above embodiment, the model will completely ignore the feature expression of the background area, and the target edge with slight deviation will be ineffective, and the unconstrained background area may be misjudged as an unknown target, that is, the robustness is reduced.

[0106] In an improved embodiment, the calculation formula of the weight coefficient k of each grid is:

[0107]

[0108] wherein n is the number of real labeled boxes in the current image, m is the number of center points of the real labeled boxes in the corresponding grid, a is the basic weight of each grid, and a is a positive integer.

[0109] It can be understood that, by setting a basic weight for each grid, it is ensured that the target-free area (such as pure background) can still participate in self-supervised learning, avoiding excessive bias, maintaining uniform initialization, ensuring that the network continuously learns the global feature consistency, thereby improving the robustness, for example, the center of the labeled box may have a deviation of 1-2 pixels (especially in dense scenes), or part of the small target labeling may be missed (such as a severely occluded pedestrian).

[0110] In order to balance the extreme dense crowd small target detection performance and robustness, in a best specific embodiment, the value of s is 4, and the value of a is 1.

[0111] In a preferred embodiment, the method for obtaining the self-supervised loss function L comprises the following steps:

[0112] S3.21, the structural similarity index SSIM is uniformly adjusted to [-1, 1]. The closer the value is to 1, the higher the structural similarity between the two images. The closer the value is to -1, the lower the structural similarity.

[0113] S3.22, the formula for calculating the self-supervised loss function L is:

[0114] ,

[0115] Where X = 1 - SSIM(F1, F2).

[0116] It is understood that the calculation formula of the self-supervised loss function L in this embodiment combines the advantages of L1 loss and L2 loss. When the SSIM(F1,F2) values ​​of two feature maps F1 and F2 are 1, that is, when the two are very similar, the loss reaches the minimum.

[0117] In a preferred embodiment, please refer to the following: Figure 5 After constructing the self-supervised loss function, the following steps are also included:

[0118] Let the feature extraction layers of the object detection network be T1 to T2 respectively. n ;

[0119] Based on the analysis of the Structural Similarity Index (SSIM) of feature maps from adjacent layers, the feature extraction layer T is determined to be the layer where the boundary structure of small targets in dense crowds undergoes qualitative changes. m , where 1 < m < n;

[0120] Establish the total self-supervised loss function L total =L 1&2 +L 2&3 +……+L （m-1）&m L 1&2 L represents the self-supervised loss between the feature maps of feature extraction layer T1 and feature extraction layer T2. 2&3 L represents the self-supervised loss between the feature maps of feature extraction layer T2 and feature extraction layer T3. （m-1）&m Indicates the feature extraction layer T m-1 Feature maps and feature extraction layer T m Self-supervised loss of feature maps.

[0121] Understandably, with Figure 6 Taking the object detection network shown as an example, when pre-training the object detection network using the main loss function of the detection task in the first stage, a 640×640 training image is input, and the feature maps of each level of the output are visualized as follows. Figure 7As shown in the figure, it can be seen that as the level gradually deepens, the feature map resolution size gradually becomes smaller, and the details of the small target (human body) in the picture gradually disappear. The dark3 layer (T3) feature map size is 80x80, and the small target information can still be distinguished relatively clearly, but the dark4 layer (T4) and the dark5 layer (T5) cannot distinguish the small target boundary information at all. It is worth noting that in the 80x80 feature map of the PAFPN (feature fusion module), the small target boundary information is not as obvious as in the dark3 feature map. After analyzing a large number of different training images, it is found that each training image has a node where the small target boundary structure changes in quality during the continuous feature extraction process, and the node where the small target boundary structure changes in quality may be different depending on the size of the detection frame and the degree of crowd density.

[0122] Therefore, in this embodiment, based on the analysis of the structural similarity index SSIM of the feature maps of adjacent levels, the feature extraction layer where the small target boundary structure changes in quality in each training image can be determined. In the target detection network, the small target is mainly detected by the large resolution feature map, so in order to improve the detection ability of the model for the small target in the dense crowd, the detail information of the large resolution feature map needs to be improved. Therefore, according to the method of this embodiment, the total self-supervised loss function L total After that, in the training process of the second stage, the visualized feature maps of each level output by the network are as shown in Figure 8 As shown in the figure, it can be seen that as the level gradually deepens, the feature map of the stem layer (T1) to the dark3 layer (T3) has no obvious change in the small target boundary structure compared with the corresponding feature map in the first stage, but when it reaches the dark4 layer (T4), the small target boundary structure still maintains a relative gradient loss in clarity, and has a significant improvement compared with the corresponding feature map in the first stage.

[0123] It can be seen that the total self-supervised loss function L total After that, there are two beneficial effects:

[0124] (1) The feature extraction layer where the small target boundary structure changes in quality is determined as T m After that, only the upper feature map is forced to be constrained, and the invalid forced constraint of the lower feature map is discarded, thereby improving the self-supervision effect.

[0125] (2) According to the calculation formula of the total self-supervised loss function L total , the self-supervised losses between multiple consecutive levels are associated and constrained, which is beneficial to the gradientization of the losses between levels and further prevents training convergence difficulties or "gradient explosion" and the like.

[0126] The application also provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory, and the processor implements the method as described above when executing the computer program.

[0127] The electronic device can be a smart camera, a mobile phone, a computer, a smart car or the like.

[0128] The above merely provides the preferred embodiments of the application, and is not intended to limit the application. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A feature map self-supervised dense crowd small target detection method, characterized in that, The method comprises the steps of: S1, using a target detection network to perform multi-scale feature extraction on a training image to generate feature maps of different levels; S2, pre-training the target detection network using a main loss function of a detection task in a first stage; S3, jointly training the target detection network by adding a self-supervised loss function in a second stage, comprising the steps of: S3.1, calculating a structural similarity index SSIM of the feature maps of adjacent levels as a self-supervised signal; S3.2, constructing a self-supervised loss function based on the structural similarity index SSIM to force the deep feature maps to retain the detail information of the shallow feature maps, and jointly optimizing the network parameters with the main loss function of the detection task; the method for obtaining the self-supervised loss function L comprises the steps of: S3.21, adjusting the structural similarity index SSIM to [-1, 1], and the closer to 1, the higher the structural similarity of the two images, and the closer to -1, the lower the structural similarity; S3.22, the formula of the self-supervised loss function L is: ， Wherein, X=1-SSIM(F1,F2); After constructing the self-supervised loss function, the method further comprises the steps of: The feature extraction layers of the target detection network are T1 to T n ; Based on the structural similarity index SSIM of the feature maps of adjacent layers, it is determined that the feature extraction layer T at which the boundary structure of the small target in the dense crowd occurs qualitative change m where 1 < m < n; Establish a total self-supervised loss function L total = L 1&2 + L 2&3 + … + L （m-1）&m , wherein L 1&2 represents the self-supervised loss of the feature map of the feature extraction layer T1 and the feature map of the feature extraction layer T2, L 2&3 represents the self-supervised loss of the feature map of the feature extraction layer T2 and the feature map of the feature extraction layer T3, and L （m-1）&m represents the self-supervised loss of the feature map of the feature extraction layer T m-1 and the feature map of the feature extraction layer T m . The formula for calculating the structural similarity index SSIM of the feature maps of adjacent levels is: SSIM(F1,F2) = (l(F1,F2) x c(F1,F2) x s(F1,F2)) α Wherein, F1, F2 represent two feature maps of adjacent levels, SSIM(F1,F2) represents the structural similarity index SSIM of the two feature maps of adjacent levels, l(F1,F2) represents the brightness similarity between the two images, c(F1,F2) represents the contrast similarity between the two images, s(F1,F2) represents the structural similarity between the two images, and a is the dependent variable, which is proportional to the density of the target population.

2. The feature map self-supervised dense crowd small target detection method of claim 1, wherein, The method for calculating the structural similarity index SSIM of the feature maps of adjacent levels comprises the steps of: S3.11, compute mean along channel dimension for each level of feature maps, generate single channel mean feature maps F mean (x, y), the calculation formula is: ， where C is the number of feature maps of the corresponding level, F i (x, y) is the value of the pixel coordinate (x, y) in the i-th feature map. S3.12, spatially aligning the mean feature maps of adjacent levels.

3. The feature map self-supervised dense crowd small target detection method of claim 2, wherein, The method for calculating the structural similarity index SSIM of the feature maps of adjacent levels further comprises the steps of: S3.13, dividing the spatially aligned mean feature maps into s×s grids, and respectively applying different weights to each grid to calculate the structural similarity index SSIM.

4. The feature map self-supervised dense crowd small target detection method of claim 3, wherein, The grid weight is dynamically adjusted according to the center point position of the real labeled box, and the weight of the grid where the center of the labeled box falls is increased.

5. The feature map self-supervised dense crowd small target detection method of claim 4, wherein, The calculation formula of the weight coefficient k of each grid is: ， Wherein, n is the number of real labeled boxes in the mean feature map, and m is the number of center points of real labeled boxes in the corresponding grid.

6. The feature map self-supervised dense crowd small target detection method of claim 4, wherein, The calculation formula of the weight coefficient k of each grid is: ， Wherein, n is the number of real labeled boxes in the current image, m is the number of center points of real labeled boxes in the corresponding grid, a is the basic weight of each grid, and a is a positive integer.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-6. The processor executes the computer program to realize the method of any one of claims 1 to 6.

Images