A real-time multi-target detection system and method based on deep learning

Abstract

The application discloses a real-time multi-target detection system and method based on deep learning, and relates to the technical field of deep learning.The application utilizes interframe correlation multiplexing of overlapping area features through a space-time multiplexing module, thereby reducing redundant calculation; a feature enhancement module differentiates processing of dynamic and overlapping stable area features, strengthens basic dynamic features of a motion area, simplifies basic static features of an overlapping stable area, and performs weighted fusion on transition area features; and a window generation module dynamically adjusts detection window parameters based on target density.The application solves the problem that real-time performance and detection accuracy are difficult to be considered in the prior art, and improves processing efficiency and detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a real-time multi-target detection system and method based on deep learning. BACKGROUND

[0002] The real-time multi-target detection technology is applied in many fields such as security monitoring, intelligent transportation and automatic driving. Its core requirement is to ensure detection accuracy while meeting high frame rate real-time processing requirements. However, the existing technology still has the following defects in practical application:

[0003] Firstly, the existing method adopts a frame-by-frame independent processing mode, which does not fully utilize the correlation between video frames, resulting in repeated calculation of features in overlapping areas and waste of computing resources.

[0004] Secondly, the existing technology lacks differentiated strategies for feature processing of dynamic and static targets, and mostly uses unified feature extraction. For static targets, too much redundant texture and other non-key features are retained, increasing the computational burden. For dynamic targets, the core features such as motion trajectory and edge change cannot be targeted to be strengthened, resulting in loss of dynamic target details.

[0005] Thirdly, the existing detection window is mostly of fixed size and step, which cannot adapt to the dynamic changes of target density in the scene. In the target dense area, the target may be missed due to window oversize, and in the sparse area, a large amount of invalid calculation may be generated due to window undersize.

[0006] Therefore, there is an urgent need for a real-time multi-target detection system and method that can efficiently utilize inter-frame features and dynamically adapt window parameters. SUMMARY

[0007] In view of the deficiencies of the existing technology, the present application provides a real-time multi-target detection system and method based on deep learning, which solves the problems of difficulty in balancing real-time performance and detection accuracy and feature processing redundancy in the existing technology.

[0008] To achieve the above purpose, the present application realizes the following technical scheme: a real-time multi-target detection system based on deep learning, comprising:

[0009] The space-time multiplexing module sets a key frame every T frames for the collected video stream, obtains the overlapping region and the motion region through inter-frame feature point matching and difference calculation for the key frame and the previous frame image, and obtains the overlapping stable region by combining the two.

[0010] The feature enhancement module generates a binary mask image based on the motion region and the overlap stable region, divides a transition region, extracts and enhances dynamic basic features of the motion region, extracts and simplifies static basic features of the overlap stable region, thereby obtaining a transition region feature image, and splices the transition region feature image into a mixed feature image, wherein the mask value of the motion region is 1, the mask value of the overlap stable region is 0, and the mask value of the transition region is 0.5.

[0011] The window generation module calculates a feature response mean of the motion region, a contour matching degree of the overlap stable region, and a comprehensive response value of the transition region based on the mixed feature image, and marks a potential dynamic target area, a potential static target area, and a potential transition target area according to the feature response mean, the contour matching degree, and the comprehensive response value, and dynamically adjusts the size and step length of a detection window to realize multi-target detection.

[0012] As a further scheme of the application, the specific steps for calculating the overlap region are as follows:

[0013] ORB feature point matching is used to realize pixel-level alignment between the key frame and the previous frame image, and pixel-by-pixel difference calculation is performed.

[0014] When the pixel value difference of the corresponding region of the two frames is less than or equal to a difference threshold Dif1, the region is determined as an overlap region, and the coordinate range thereof is recorded.

[0015] The overlap degree D is calculated as overlap region area / key frame total area, and the overlap region is divided into three levels.

[0016] If D is greater than or equal to Dmax, the overlap region is classified as high overlap, and the complete features of the previous frame are directly reused, wherein the complete features refer to all basic visual features related to target detection in the overlap region, including edge features, texture features, and color features.

[0017] If Dmin is less than or equal to D and D is less than Dmax, the overlap region is classified as medium overlap, and the core features of the previous frame need to be reused, and only the detailed features of the changed region are recalculated, wherein the core features refer to features that remain stable in slight changes and play a key role in target detection, including contour features, dominant color features, and spatial position features, and the detailed features refer to local microscopic features that change with slight changes in the scene, including dynamic texture features, light and shadow detail features, and local edge features.

[0018] If D is less than Dmin, the overlap region is classified as low overlap, and no features are reused, and the entire region is recalculated.

[0019] Dmin and Dmax are the lower limit and upper limit of the overlap degree.

[0020] As a further scheme of the present application, the region where the pixel change in the inter-frame difference is greater than or equal to the difference threshold Dif2 is divided into a motion region, and the motion vector of each pixel is calculated by combining the optical flow method, and the region where the motion vectors are continuous and consistent is marked as a real dynamic target.

[0021] As a further scheme of the present application, the specific steps for obtaining the overlapping stable region are as follows:

[0022] selecting a fixed time window T;

[0023] For the i-th frame in the time window, i∈[adjacent key frame, key frame], the overlapping region with the (i-1)-th frame is calculated, and the high-overlapping region and the medium-overlapping region are screened out, obtaining T-1 frame pair overlapping regions, denoted as R 1,2 , R 2,3 ,..., R T-1,T ;

[0024] The region that continuously maintains high and medium overlap with the previous frame in the T frames is calculated, and is denoted as the preliminary stable region, i.e. 1,2 ∩R 2,3 ∩...∩R T-1,T ;

[0025] The real dynamic target in the preliminary stable region is removed to obtain the overlapping stable region;

[0026] The motion vector of each pixel is calculated by using the optical flow method for the key frame of the overlapping stable region and the previous frame image, and a speed threshold Vth is set, and the pixels with speed < Vth are marked as potential static pixels;

[0027] For the potential static pixels, the previous N frames are continuously observed: if the speed of each frame is < Vth and the direction is irregular, then the connected region composed of these pixels is marked as a real static target.

[0028] As a further scheme of the present application, the specific operation of extracting and enhancing the dynamic basic features of the motion region is as follows:

[0029] The spatio-temporal Transformer model is selected to extract 256-dimensional dynamic basic features;

[0030] The SE attention module is used to calculate the channel weight of the 256-dimensional dynamic basic features, and the weight of the motion trajectory and the edge change is increased to 1.2-1.5 times, and the weight of other channels is kept at 0.8-1.0;

[0031] The feature map is spatially refined by a 3x3 convolution layer, the activation function is LeakyReLU, and the output is a 256-dimensional feature map.

[0032] As a further scheme of the present application, the specific operation of extracting the static basic features of the overlapping stable area and simplifying is:

[0033] The ResNet50 model is selected to extract 256-dimensional static basic features.

[0034] The 256-dimensional static basic features are processed by principal component analysis (PCA), and the first 140 principal components with a cumulative variance contribution rate of greater than or equal to 95% are retained to obtain 140-dimensional features.

[0035] The 1x1 convolution layer is further used for dimension reduction, and the global maximum pooling is used to extract the semantic peak value of each channel to output a 128-dimensional feature map.

[0036] The 1x1 convolution layer is further used for dimension reduction, and the global maximum pooling is used to extract the semantic peak value of each channel to output a 128-dimensional feature map.

[0037] As a further scheme of the present application, the specific steps for obtaining the transition area feature map are:

[0038] The coordinate range of the mask value 0.5 in the original image is obtained.

[0039] The features corresponding to the position are cropped from the motion area feature map, denoted as Fd_transition, and the size is transition area width x transition area height x 256.

[0040] The features in the same coordinate range are cropped from the overlapping stable area feature map, denoted as Fs_transition, and the size is the same as Fd_transition.

[0041] The Euclidean distance d1 of each pixel in the transition area to the boundary of the motion area and the Euclidean distance d2 to the boundary of the overlapping stable area are calculated.

[0042] The dynamic weight w1 = d2 / (d1+d2) and the static weight w2 = d1 / (d1+d2) are calculated.

[0043] The transition area feature map Ft is calculated according to the formula Ft = Fd_transition x w1 + Fs_transition x w2, and the output size is transition area width x transition area height x 256, which is consistent with the dimension of the motion / overlapping stable area feature map.

[0044] As a further scheme of the present application, the specific operation of marking the potential dynamic target area, the potential static target area and the potential transition target area is:

[0045] The edge channel and the motion trajectory channel are selected from the motion area feature map, and the weighted average of the feature values of the two types of channels is performed, the edge channel weight is 0.6, the motion trajectory channel weight is 0.4, the feature response mean value is obtained, and if the mean value is greater than or equal to R1, it is marked as a potential dynamic target area.

[0046] Extracting contour feature sub-vectors from the overlapping stable region feature map, calculating the cosine similarity with the off-line trained static background contour feature vector to obtain the contour matching degree, if the matching degree is less than or equal to R2, it is marked as a potential static target area;

[0047] Calculating the arithmetic mean of the feature values from the transition region feature map as a comprehensive response value, if the comprehensive response value is greater than or equal to R3, it is marked as a potential transition target area;

[0048] Wherein, R1, R2, R3 are threshold values.

[0049] As a further scheme of the application, the specific operation of dynamically adjusting the size and step length of the detection window is as follows:

[0050] Dividing the potential dynamic target area, the potential static target area and the potential transition target area into statistical units of MxM pixels, and counting the number of candidate targets in the statistical units;

[0051] Calculating the target density p = number of candidate targets in the statistical unit / unit area;

[0052] If p >= p max , it is classified as high density, if p min < p < p max , it is classified as medium density, and if p < p min , it is classified as low density, wherein p min , p max are the lower and upper limits of the density.

[0053] According to the target density level, the size and step length of the detection window are dynamically adjusted:

[0054] If it is high density, the window size is set to 32x32 and the step length is 8 pixels;

[0055] If it is medium density, the window size is set to 48x48 and the step length is 12 pixels;

[0056] If it is low density, the window size is set to 64x64 and the step length is 16 pixels;

[0057] The candidate target judgment method is as follows: if it is a potential dynamic target area, the number of candidate targets is the number of real dynamic targets; if it is a potential static target area, the number of candidate targets is the number of real static targets; if it is a potential transition target area, the number of candidate targets is the number of real dynamic targets and real static targets.

[0058] A real-time multi-target detection method based on deep learning, comprising:

[0059] S1, for the collected video stream, set a key frame every T frame, and obtain the overlap region, the motion region by inter-frame feature point matching and difference calculation, and obtain the overlap stable region by combining the two;

[0060] S2, generate a binary mask map based on the motion region and the overlap stable region, divide the transition region, extract the dynamic basic features of the motion region and enhance them, extract the static basic features of the overlap stable region and simplify them, to obtain the transition region feature map, and splice it into a mixed feature map, the mask value of the motion region is 1, the mask value of the overlap stable region is 0, and the mask value of the transition region is 0.5;

[0061] S3, based on the mixed feature map, calculate the feature response mean of the motion region, the contour matching degree of the overlap stable region, and the comprehensive response value of the transition region, and mark out the potential dynamic target area, the potential static target area and the potential transition target area, and dynamically adjust the size and step length of the detection window, to realize multi-target detection.

[0062] The application provides a real-time multi-target detection system and method based on deep learning, which has the following advantages compared with the prior art:

[0063] (1) The application utilizes the inter-frame correlation of video frames to reuse the overlap region features, solving the problem of waste of computing resources caused by traditional independent frame-by-frame processing, and laying a foundation for real-time detection;

[0064] (2) The application differentiates and enhances the features of dynamic and static regions, and simplifies the static region by reducing the dimension to retain the core information and reduce the amount of calculation, and enhances the key features of the dynamic region by using the attention mechanism to improve the detail capture ability, and the transition region features are weighted and fused to avoid boundary jump, which balances the calculation efficiency and detection accuracy;

[0065] (3) The application dynamically adjusts the detection window parameters based on the target density, uses a small window with high overlap rate in the dense area to ensure low miss detection rate, and uses a large window with large step length in the sparse area to improve efficiency, solving the problem of insufficient adaptation of fixed window to uneven target. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The system principle frame of the application is shown in the figure;

[0067] Figure 2 The step flow chart of the application is shown in the figure. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0069] As Figure 1 The present application provides a real-time multi-target detection system based on deep learning, comprising:

[0070] A space-time multiplexing module sets a key frame every T frames for the collected video stream, and the rest are non-key frames.

[0071] The video stream contains dynamic targets such as cars, pedestrians and electric vehicles, and static backgrounds such as traffic lights and buildings.

[0072] The setting of the key frame is based on the actual scene change:

[0073] If it is a static / low dynamic scene, such as an office or a warehouse shot by a fixed monitoring camera, the target moves slowly in such a scene, and the key frame interval can be appropriately increased, which can be set to 10-30 frames in general.

[0074] If it is a high dynamic scene, such as a vehicle-mounted camera or a highway monitoring camera, the vehicles move at high speed in such a scene, and the number of targets changes frequently, so the key frame interval can be appropriately reduced, which can be set to 3-8 frames in general.

[0075] For the key frame and the previous frame image, first, the pixel-level alignment is realized through the ORB feature point matching to eliminate the frame-to-frame offset caused by the slight shaking or displacement of the camera.

[0076] Then, the overlapping area of the two is calculated by pixel-by-pixel difference, and the specific judgment rule is: when the pixel value difference of the corresponding area of the two frames is less than or equal to the difference threshold Dif1, it is determined as an overlapping area, and its coordinate range is recorded, such as the upper left corner (x1, y1) to the lower right corner (x5, y5).

[0077] The overlap degree D is calculated as the overlapping area / total area of the key frame, and the overlapping area is divided into three levels:

[0078] If D≥Dmax, it is divided into high overlap, and the area is mostly static background, and the complete features of the previous frame are directly reused without any recalculation.

[0079] The complete features refer to all basic visual features related to target detection in the overlapping area, which specifically include:

[0080] Edge features, which are represented as the contour lines of static backgrounds, such as the frame lines of fixed equipment, and the pixel gray value change trend obtained by gradient calculation.

[0081] Texture feature, embodied in the repeating pattern of the surface, such as the floor tile lines, the microstructure described by local binary pattern or gray level co-occurrence matrix;

[0082] Color feature, manifested as the pixel value distribution of the region, such as the RGB mean of the white wall, the macro attribute quantified by histogram or color moment;

[0083] If Dmin≤D<Dmax, it is classified as medium overlap, there is partial slight change in the region, such as the slight movement of tree leaves, the slow change of light and shadow, etc., the core feature of the previous frame needs to be reused, and only the detail feature of the changed region needs to be recalculated;

[0084] The core feature refers to the feature that remains stable in slight changes and plays a key role in target detection, specifically including:

[0085] Contour feature, the overall shape of the target or background, such as the trunk contour of the tree, the roof contour of the building, the closed or semi-closed curve formed by edge connection;

[0086] Dominant color feature, the dominant color of the region, such as the green dominant color of tree leaves, the gray dominant color of buildings, ignoring the color mean of local light and shade fluctuations;

[0087] Spatial position feature, the relative position of the region in the picture, such as the tree located on the left side of the picture, the coordinate range not affected by slight displacement;

[0088] The detail feature refers to the local micro feature that changes with slight changes in the scene, specifically including:

[0089] Dynamic texture feature, such as local texture changes caused by tree leaf shaking;

[0090] Light and shadow detail feature, such as the change of wall light spot position caused by sunlight movement;

[0091] Local edge feature, such as the edge of the swaying branch, the position of these edges slightly shifts over time;

[0092] If D<Dmin, it is classified as low overlap, mostly scene mutation, such as lens switching, new target entering in large quantities, no feature is reused, and the whole region is recalculated;

[0093] Wherein, Dmin and Dmax are the lower limit and upper limit of the overlap degree, which need to be adjusted according to the actual situation;

[0094] The region with pixel change ≥ difference threshold Dif2 in the inter-frame difference is divided into a motion region, and the motion vector (direction and speed) of each pixel is calculated by combining the optical flow method. For the region with continuous and consistent motion vector, it is marked as a real dynamic target, such as the continuous moving track of a pedestrian.

[0095] The real dynamic target is divided into high, medium and low categories according to the motion speed, and is marked as different priorities: high > medium > low.

[0096] The specific steps for obtaining the overlap stable region according to the high overlap and medium overlap regions are as follows:

[0097] A fixed time window T is selected, which is consistent with the interval of the above two key frames.

[0098] For the i-th frame in the time window, i∈[adjacent key frame, key frame], the overlap region between the i-th frame and the i-1-th frame is calculated, and the high overlap region and the medium overlap region are selected therefrom, obtaining T-1 frame pair overlap regions, denoted as R 1,2 , R 2,3 ,..., R T-1,T .

[0099] The region that continuously maintains high and medium overlap with the previous frame in the T frames is calculated, and is denoted as the preliminary stable region, i.e. R 1,2 ∩R 2,3 ∩...∩R T-1,T .

[0100] The real dynamic target in the preliminary stable region is removed to obtain the overlap stable region.

[0101] The motion vector of each pixel in the key frame and the previous frame image of the overlap stable region is calculated by using the optical flow method, and a speed threshold Vth is set. The pixels with speed < Vth are marked as potential static pixels.

[0102] For the potential static pixels, the previous N frames are continuously observed: if the speed of each frame is < Vth and the direction is irregular, then the connected region composed of these pixels is marked as a real static target.

[0103] The feature enhancement module generates a binary mask image based on the coordinates of the motion region and the overlap stable region, wherein the mask value of the motion region is 1, the mask value of the overlap stable region is 0, and the morphological erosion-dilation operation is used to eliminate the pixel noise of the region boundary, and the motion-overlap boundary region is marked as a transition region, and the mask value can be set to 0.5 to ensure that the region division has no overlap and no omission.

[0104] For the motion region, a spatio-temporal Transformer model is selected to extract dynamic basic features, which focuses on capturing the temporal correlation and motion pattern of the target. The specific operation is as follows:

[0105] The motion region cropped image of the input continuous 3 frames of images is normalized and input into the model, and the continuous 3 frames of images specifically include the current key frame and the previous two frames;

[0106] Each frame of image is split into 16x16 image blocks, converted into 512-dimensional vectors through linear embedding, and frame position coding is added to distinguish the 1st / 2nd / 3rd frame;

[0107] A space-time attention layer (4 layers stacked) is constructed, each layer containing spatial attention and temporal attention, for capturing the spatial correlation of targets within the same frame and the temporal correlation of targets across frames, outputting a 1024-dimensional feature vector;

[0108] The 1024-dimensional vector is restored to a 256-dimensional feature map consistent with the size of the motion region through convolution projection, containing dynamic features such as motion direction, speed, and local deformation;

[0109] For the overlapping stable region, a ResNet50 model is selected to extract static basic features, focusing on extracting high-level static semantics, and the specific operation is as follows:

[0110] The overlapping stable region cropped image of the current key frame is input into ResNet50 after standardization, and 1024-dimensional deep features are extracted through the first 50 layers of convolution (including 4 residual blocks);

[0111] The 1024-dimensional features are reduced to 256-dimensional features through 1x1 convolution, retaining core static features such as outline, dominant color, and texture distribution;

[0112] For the motion region, the SE attention module is used to calculate the channel weight of the 256-dimensional dynamic basic features, and the weight of the motion trajectory (such as channels 100-150) and edge change (such as channels 200-250) is increased to 1.2-1.5 times, while the weight of other channels remains at 0.8-1.0, highlighting the core dynamic features;

[0113] SE attention can automatically identify the most discriminative channels in dynamic features, such as vehicle displacement trajectory channels being more important than color channels, and adjusting the weight is mainly to enhance useful information and suppress noise;

[0114] The feature map is processed through a 3x3 convolution layer for spatial refinement, with a LeakyReLU activation function to retain local motion details of small targets, and the output is still a 256-dimensional feature map;

[0115] 3x3 convolution strengthens spatial continuity through local correlation calculation without changing the size, while LeakyReLU avoids the loss of negative feature values, making it suitable for describing negative motion such as deceleration;

[0116] For the overlapping stable region, 140-dimensional features were obtained by using principal component analysis (PCA) to filter 5% noise redundancy from the 256-dimensional static base features, retaining the top 140 principal components with cumulative variance contribution rate ≥95%;

[0117] PCA removes highly correlated redundant channels in static features through linear transformation, while retaining more than 95% of the core information in compressed dimensions;

[0118] The 1x1 convolution layer further reduces the dimension, and the global maximum pooling extracts the semantic peak value of each channel, outputting a 128-dimensional feature map;

[0119] The 1x1 convolution layer further reduces the dimension, and the global maximum pooling extracts the semantic peak value of each channel, outputting a 128-dimensional feature map;

[0120] The essence of the first convolution layer is to remove redundant information in static features to retain core structures and reduce computational load. The purpose of constructing the second convolution layer is to expand features based on core features to generate new features matching the dynamic feature dimension;

[0121] For the transition region, the specific steps for calculating the feature map are as follows:

[0122] Get the coordinate range of the mask value 0.5 in the original image;

[0123] Crop the corresponding position features from the motion region feature map (256 dimensions), denoted as Fd_transition, with a size of transition width x transition height x 256;

[0124] Crop the features with the same coordinate range from the overlapping stable region feature map, denoted as Fs_transition, with the same size as Fd_transition;

[0125] Calculate the Euclidean distance d1 of each pixel in the transition region to the boundary of the motion region (mask value = 1), and the Euclidean distance d2 to the boundary of the overlapping stable region (mask value = 0);

[0126] Calculate the dynamic weight w1 = d2 / (d1+d2) and the static weight w2 = d1 / (d1+d2);

[0127] According to the formula Ft = Fd_transition x w1 + Fs_transition x w2, the transition region feature map Ft is calculated, with an output size of transition width x transition height x 256, consistent with the feature map dimension of the motion / overlapping stable region;

[0128] The feature maps of the motion region, overlapping stable region, and transition region are spliced according to the original image coordinates, and a 256-dimensional mixed feature map of the whole image is output.

[0129] The window generation module calculates a feature response mean value of the motion region, a contour matching degree of the overlap stable region, and a comprehensive response value of the transition region based on the mixed feature map, and marks out a potential dynamic target area, a potential static target area, and a potential transition target area;

[0130] The feature response mean value of the motion region is calculated:

[0131] An edge channel (depicting target contour change) and a motion trajectory channel (depicting target displacement vector) are selected from the 256-dimensional enhanced features, and weighted average is performed on the feature values of the two types of channels, with the edge channel weight being 0.6 and the motion trajectory channel weight being 0.4, to obtain the feature response mean value, and if the mean value is greater than or equal to R1, the potential dynamic target area is marked;

[0132] The contour matching degree of the overlap stable region is calculated:

[0133] A contour feature sub-vector (depicting the shape and texture distribution of the static contour of the target) is extracted from the 256-dimensional simplified features after convolution, and a cosine similarity is calculated with the static background contour feature vector trained offline to obtain the contour matching degree, and if the matching degree is less than or equal to R2 (significantly different from the background contour), the potential static target area is marked;

[0134] The comprehensive response value of the transition region is calculated:

[0135] An arithmetic mean value is taken of the 256-dimensional transition region feature values, because the transition region target has both dynamic and static characteristics, and has no single core feature, so general features need to be integrated to obtain the comprehensive response value, and if the value is greater than or equal to R3, the potential transition target area is marked;

[0136] R1, R2, and R3 are threshold values, which are set according to actual conditions;

[0137] The potential dynamic target area, the potential static target area, and the potential transition target area are divided into MxM pixel statistical units, and the number of candidate targets in the statistical unit is counted:

[0138] If it is a potential dynamic target area, the number of candidate targets is the number of real dynamic targets;

[0139] If it is a potential static target area, the number of candidate targets is the number of real static targets;

[0140] If it is a potential transition target area, the number of candidate targets is the number of real dynamic targets and real static targets;

[0141] The target density p = the number of candidate targets in the statistical unit / the unit area is calculated;

[0142] If p >= p max , it is divided into high density, and if p min < p < p max, then it is classified as medium density, if p < p min , then it is classified as low density;

[0143] According to the target density level, dynamically adjust the size and step of the detection window:

[0144] If it is high density, the window size is set to 32x32, and the step is 8 pixels;

[0145] In the dense area, the number of targets per unit area is large, and the spatial distance between targets is small. The 32x32 small window can reduce the mutual shielding interference between adjacent targets, avoid the detection confusion caused by a single large window covering multiple targets, and ensure that the overlap rate of adjacent windows reaches 75% by setting the step to 8 pixels. High overlap rate can maximize the coverage of densely distributed targets and reduce the risk of missed detection.

[0146] If it is medium density, the window size is set to 48x48, and the step is 12 pixels;

[0147] In the medium area, the number of targets is moderate, and the distribution is neither compact nor sparse. The 48x48 medium window can adapt to most possible target size ranges, and the step of 12 pixels corresponds to an overlap rate of 75%. Compared with the small window + small step in the dense area, it reduces the number of scans while ensuring coverage accuracy.

[0148] If it is low density, the window size is set to 64x64, and the step is 16 pixels;

[0149] In the sparse area, the number of targets is small, and the distance between targets per unit area is large. The 64x64 large window can reduce the number of scans and improve efficiency. The step of 16 pixels corresponds to an overlap rate of 75%. On the premise of efficiency priority, it still maintains a certain overlap to avoid missed detection.

[0150] If the potential static target area is low density, a larger 80x80 window and a 20-pixel step can be selected to further reduce the number of scans, and there will be no missed detection due to motion. The efficiency improvement is more obvious.

[0151] As Figure 2 A real-time multi-target detection method based on deep learning, comprising:

[0152] S1, for the collected video stream, set a key frame every T frame, and through inter-frame feature point matching and difference calculation, obtain the overlap area, motion area, and combine the two to obtain the overlap stable area.

[0153] S2, generating a binary mask map based on the motion region and the overlap stable region, dividing a transition region, extracting and enhancing dynamic basic features of the motion region, extracting and simplifying static basic features of the overlap stable region, obtaining a transition region feature map, and splicing the transition region feature map into a hybrid feature map, wherein the mask value of the motion region is 1, the mask value of the overlap stable region is 0, and the mask value of the transition region is 0.5;

[0154] S3, calculating the feature response mean of the motion region, the contour matching degree of the overlap stable region, and the comprehensive response value of the transition region based on the hybrid feature map, marking out a potential dynamic target area, a potential static target area, and a potential transition target area, counting the target density of different regions, and dynamically adjusting the size and step length of the detection window to realize multi-target detection.

[0155] Some data in the above formula are dimensionless numerical calculations, and the contents not described in detail in the specification all belong to the prior art known to those skilled in the art.

[0156] The above examples are only used to illustrate the technical method of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A real-time multi-target detection system based on deep learning, characterized in that, The application relates to a multi-target detection method based on video stream. The method comprises the following steps: A space-time multiplexing module is used to set a key frame every T frames for the collected video stream, and the key frame and the previous frame image are matched and calculated by using the frame difference to obtain an overlapping region, a motion region and an overlapping stable region; A feature enhancement module is used to generate a binary mask based on the motion region and the overlapping stable region, to divide a transition region, to extract and enhance a dynamic basic feature of the motion region, to extract and simplify a static basic feature of the overlapping stable region, to obtain a transition region feature map, and to splice the transition region feature map into a mixed feature map, wherein the mask value of the motion region is 1, the mask value of the overlapping stable region is 0, and the mask value of the transition region is 0.5; 2. The real-time multi-target detection system based on deep learning according to claim 1, wherein, A window generation module is used to calculate the feature response mean of the motion region, the contour matching degree of the overlapping stable region and the comprehensive response value of the transition region based on the mixed feature map, to mark a potential dynamic target area, a potential static target area and a potential transition target area, to count the target density of different areas and to dynamically adjust the size and step of the detection window, and to realize multi-target detection. The specific steps for calculating the overlapping region are as follows: ORB feature point matching is used to realize pixel-level alignment of the key frame and the previous frame image, and the difference is calculated pixel by pixel; When the pixel value difference of the corresponding region of the two frames is less than or equal to a difference threshold Dif1, the region is determined as an overlapping region, and the coordinate range is recorded; The overlapping degree D is calculated as overlapping region area / key frame total area, and the overlapping region is divided into three levels: If D is greater than or equal to Dmax, the region is divided into a high overlapping region, and the complete feature of the previous frame is directly reused, wherein the complete feature refers to all basic visual features related to target detection in the overlapping region, including edge features, texture features and color features; If Dmin is less than D and D is less than Dmax, the region is divided into a medium overlapping region, and the core feature of the previous frame needs to be reused, and only the detail feature of the changed region is recalculated, wherein the core feature refers to the feature that remains stable in slight changes and plays a key role in target detection, including contour features, main tone features and spatial position features, and the detail feature refers to the local microscopic feature that changes with slight changes in the scene, including dynamic texture features, light and shadow detail features and local edge features; If D is less than Dmin, the region is divided into a low overlapping region, and no feature is reused, and the whole region is recalculated; 3. The real-time multi-target detection system based on deep learning according to claim 1, wherein, Wherein, Dmin and Dmax are the lower limit and the upper limit of the overlapping degree.

4. The real-time multi-target detection system based on deep learning according to claim 1, wherein, The region with a pixel change greater than a difference threshold Dif2 in the frame difference is divided into a motion region, and the motion vector of each pixel is calculated by using an optical flow method, and the region with a continuous and consistent motion vector is marked as a real dynamic target. The specific steps for obtaining the overlapping stable region are as follows: For the i-th frame in the time window, i∈[adjacent key frame, key frame], the overlapping area with the i-1-th frame is calculated, and the high overlapping area and the medium overlapping area are screened out, to obtain T-1 frame pair overlapping areas, denoted as R 1,2 , R 2,3 ,..., R T-1,T ; Calculate the T intra-frame duration and the high, medium overlap with the previous frame, and record it as the preliminary stable area, that is, R 1,2 ∩R 2,3 ∩...∩R T-1,T ; A fixed time window T is selected; The real dynamic target in the preliminary stable region is removed to obtain the overlapping stable region; The motion vector of each pixel in the key frame and the previous frame image of the overlapping stable region is calculated by using an optical flow method, and the pixels with a speed less than a speed threshold Vth are marked as potential static pixels; The potential static pixels are continuously observed for N frames: if the speed of each frame is less than Vth and the direction is irregular, the connected region formed by the pixels is marked as a real static target.

5. The real-time multi-target detection system based on deep learning according to claim 1, wherein, The specific operation of extracting dynamic basic features of the motion region and enhancing them is as follows: A spatiotemporal Transformer model is selected to extract 256-dimensional dynamic basic features; An SE attention module is used to calculate the channel weight of the 256-dimensional dynamic basic features, so that the weight of the motion trajectory and edge change is increased to 1.2-1.5 times, and the weight of other channels is kept at 0.8-1.0; A 3*3 convolution layer is used to perform spatial fine processing on the feature map, and a LeakyReLU activation function is used, and the output is a 256-dimensional feature map.

6. The real-time multi-target detection system based on deep learning according to claim 1, wherein, The specific operation of extracting static basic features of the overlapping stable region and simplifying them is as follows: A ResNet50 model is selected to extract 256-dimensional static basic features; Principal component analysis (PCA) is used to process the 256-dimensional static basic features, and the first 140 principal components with a cumulative variance contribution rate of ≥95% are retained to obtain 140-dimensional features; A 1*1 convolution layer is further used for dimension reduction, and a global maximum pooling is used to extract the semantic peak value of each channel, and an output of 128-dimensional feature map is obtained; Then, a 1*1 convolution layer is used to upgrade the 128-channel feature map to a 256-channel feature map.

7. The real-time multi-target detection system based on deep learning according to claim 1, wherein, The specific steps of obtaining the transition region feature map are as follows: Obtain the coordinate range of the mask value 0.5 in the original image; Crop the features of the corresponding position from the motion region feature map, denoted as Fd_transition, with a size of transition region width*transition region height*256; Crop the features of the same coordinate range from the overlapping stable region feature map, denoted as Fs_transition, with a size identical to Fd_transition; Calculate the Euclidean distance d1 of each pixel in the transition region to the boundary of the motion region, and the Euclidean distance d2 to the boundary of the overlapping stable region; Calculate the dynamic weight w1=d2 / (d1+d2) and the static weight w2=d1 / (d1+d2); According to the formula Ft=Fd_transition*w1+Fs_transition*w2, the transition region feature map Ft is calculated, and the output size is: transition region width*transition region height*256, which is consistent with the dimension of the motion / overlapping stable region feature map.

8. The real-time multi-target detection system based on deep learning according to claim 1, wherein, The specific operation of marking potential dynamic target regions, potential static target regions, and potential transition target regions is as follows: From the motion region feature map, the edge channel and the motion trajectory channel are selected, and the weighted average of the feature values of these two types of channels is calculated, with the edge channel weight being 0.6 and the motion trajectory channel weight being 0.4, to obtain the feature response mean. If the mean is ≥R1, it is marked as a potential dynamic target region. From the overlapping stable region feature map, the contour feature sub-vector is extracted, and the cosine similarity with the offline trained static background contour feature vector is calculated to obtain the contour matching degree. If the matching degree is ≤R2, it is marked as a potential static target region. From the transition region feature map, the arithmetic mean of the feature values is calculated as the comprehensive response value. If the comprehensive response value is ≥R3, it is marked as a potential transition target region. Wherein, R1, R2, R3 are threshold values.

9. The real-time multi-target detection system based on deep learning according to claim 1, wherein, The specific operation of dynamically adjusting the size and step length of the detection window is as follows: Divide the potential dynamic target region, potential static target region, and potential transition target region into M*M pixel statistical units, and count the number of candidate targets in the statistical unit; Calculate the target density p=number of candidate targets in the statistical unit / unit area; If ρ ≥ ρ max , then it is classified as high density, if ρ min ≤ ρ < ρ max , then it is classified as medium density, if ρ < ρ min , then it is classified as low density, wherein ρ min , ρ max are the lower and upper limits of the density. According to the target density level, the size and step of the detection window are dynamically adjusted: If the density is high, the window size is set to 32x32 and the step is 8 pixels; If the density is medium, the window size is set to 48x48 and the step is 12 pixels; If the density is low, the window size is set to 64x64 and the step is 16 pixels. The candidate target number judgment method is: if it is a potential dynamic target area, the number of candidate targets is the number of real dynamic targets; if it is a potential static target area, the number of candidate targets is the number of real static targets; if it is a potential transition target area, the number of candidate targets is the number of real dynamic targets and real static targets.

10. A deep learning based real-time multi-target detection method for implementing the deep learning based real-time multi-target detection system of any one of claims 1-9, characterized in that, It includes: S1, for the collected video stream, set a key frame every T frame, obtain the overlap area, motion area by inter-frame feature point matching and difference calculation, and obtain the overlap stable area by combining the two; S2, generate a binary mask map based on the motion area and the overlap stable area, divide the transition area, extract the dynamic basic features of the motion area and enhance them, extract the static basic features of the overlap stable area and simplify them, to obtain the transition area feature map, and splice it into a mixed feature map, the motion area mask value is 1, the overlap stable area mask value is 0, and the transition area mask value is 0.5; S3, based on the mixed feature map, calculate the feature response mean of the motion area, the contour matching degree of the overlap stable area, and the comprehensive response value of the transition area, and mark out the potential dynamic target area, the potential static target area, and the potential transition target area according to the above, and dynamically adjust the size and step of the detection window to realize multi-target detection.

Images