Shadow detection method and system based on high-bit video monitoring

Abstract

The application discloses a shadow detection method and system based on high-bit video monitoring. The method comprises the following steps: inputting a target-level shadow image data set into a shadow detection extraction network for feature extraction, and outputting a target-level shadow image feature set; inputting the target-level shadow image feature set into a shadow detection classification regression network for learning, and outputting a predicted target category, a predicted shadow category, a predicted background category, a predicted target position and a predicted shadow position of each detection frame; inputting the target-level shadow image feature set into a target shadow mask learning network for learning, and outputting a predicted binary mask of each candidate region frame; constructing a model loss function according to the predicted target category, the predicted shadow category, the predicted background category, the predicted target position, the predicted shadow position of the detection frame and the predicted binary mask of the candidate region frame, and performing model optimization and parameter updating to obtain a shadow detection model, and the shadow detection model is used for shadow detection on a to-be-detected roadside monitoring scene image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a shadow detection method and system based on high-position video monitoring. BACKGROUND

[0002] In recent years, high-position video technology has developed rapidly. By installing high-position video cameras on the roadside for data collection, using visual algorithms on the collected data, vehicle detection, license plate recognition, vehicle tracking, pedestrian detection, parking line detection and other tasks can be achieved, which is beneficial to more accurate management of roadside parking, thereby providing data for the deployment and development of vehicle-road cooperation technology, and thus positively promoting various aspects of urban traffic management and driving safety. In different seasons of the year and different light time periods of the day, the shadows of targets such as pedestrians and vehicles in the video images recorded by the monitoring video lens also change, and the shadow detection of the targets is also of great significance.

[0003] The traditional shadow detection method uses an edge information detection algorithm to divide the low-order edge part of the image, obtain the edge of the shadow part, realize the classification of the shadow part from the image, and obtain the shadow area. However, the edge positioning of the traditional shadow detection method is not accurate, resulting in low detection accuracy of the shadow area, and the shadow area cannot be accurately detected. SUMMARY

[0004] The purpose of the present application is to solve the technical problem of low detection accuracy of the traditional shadow detection method. To achieve the above purpose, the present application provides a shadow detection method and system based on high-position video monitoring.

[0005] The present application provides a shadow detection method based on high-position video monitoring, comprising:

[0006] Constructing a target-level shadow image data set;

[0007] Inputting the target-level shadow image data set into a shadow detection extraction network of an initial shadow detection model for feature extraction, and outputting a target-level shadow image feature set;

[0008] Inputting the target-level shadow image feature set into a shadow detection classification regression network of the initial shadow detection model for learning, and outputting a predicted target class, a predicted shadow class, a predicted background class, a predicted target position and a predicted shadow position of each detection frame;

[0009] Inputting the target-level shadow image feature set into a target shadow mask learning network of the initial shadow detection model for learning, and outputting a predicted binary mask of each candidate region frame;

[0010] According to the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position of each detection frame and the predicted binary mask of each candidate area frame, a model loss function is constructed to optimize the initial shadow detection model and update the parameters to obtain a shadow detection model;

[0011] The roadside monitoring scene image to be measured is detected according to the shadow detection model to obtain a shadow area and a target area corresponding to the shadow area.

[0012] In one embodiment, inputting the target-level shadow image dataset into a shadow detection extraction network of an initial shadow detection model for feature extraction and outputting a target-level shadow image feature set includes:

[0013] Inputting the target-level shadow image dataset into the feature compression structure of the shadow detection and extraction network for feature compression, and outputting a global compressed feature set;

[0014] The global compressed feature set is input into the remodeling structure of the shadow detection and extraction network for reconstruction, and the target-level shadow image feature set is output.

[0015] In one embodiment, the object-level shadow image feature set is input into the shadow detection classification regression network of the initial shadow detection model for learning, and the predicted object category, predicted shadow category, predicted background category, predicted object position, and predicted shadow position of each detection frame are output, including:

[0016] Inputting the target-level shadow image feature set into a multi-layer first convolutional module and a first fully connected layer for category classification learning, and outputting the predicted target category, the predicted shadow category, and the predicted background category of each detection frame;

[0017] The target-level shadow image feature set is input into the multi-layer second convolution module and the second fully connected layer to perform two-dimensional detection frame coordinate position learning, and the predicted target position and the predicted shadow position of each detection frame are output.

[0018] In one embodiment, inputting the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning and outputting the predicted binary mask of each candidate region box includes:

[0019] Inputting the target-level shadow image feature set into a region extraction network of a two-stage mask branch network to perform region extraction, and obtaining multiple candidate region frames;

[0020] Inputting the multiple candidate region frames into the candidate frame alignment layer of the two-stage mask branch network for resizing to obtain multiple candidate region frames of the same size;

[0021] The multiple candidate region frames of the same size are input into the mask prediction layer of the two-stage mask branch network to obtain the predicted binary mask of each candidate region frame.

[0022] In one embodiment, the object-level shadow image dataset includes object annotation data, and the object annotation data includes the true object category, true shadow category, true background category, true object position, true shadow position and true binary mask of each detection frame;

[0023] The method comprises: constructing a model loss function based on the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position, and predicted binary mask of each candidate region frame to optimize the initial shadow detection model and update the parameters to obtain a shadow detection model, including:

[0024] Forming a classification loss function according to the true target category, the true shadow category, the true background category, the predicted target category, the predicted shadow category, and the predicted background category of each detection frame;

[0025] forming a position regression loss function based on the true target position, the true shadow position, the predicted target position, and the predicted shadow position;

[0026] Obtaining a true shadow offset of a center point of a shadow detection frame relative to a center point of a target detection frame based on the true target position and the true shadow position; obtaining a predicted shadow offset of a center point of a shadow detection frame relative to a center point of a target detection frame based on the predicted target position and the predicted shadow position; and forming a shadow offset loss function based on the true shadow offset and the predicted shadow offset;

[0027] Forming a mask loss function according to the true binary mask and the predicted binary mask;

[0028] The model loss function is constructed according to the classification loss function, the position regression loss function, the shadow offset loss function and the mask loss function.

[0029] In one embodiment, a shadow detection system based on high-position video surveillance includes:

[0030] Dataset construction module, used to construct object-level shadow image dataset;

[0031] A shadow detection and extraction network module is used to input the target-level shadow image dataset into the shadow detection and extraction network of the initial shadow detection model to perform feature extraction and output a target-level shadow image feature set;

[0032] a shadow detection classification regression network module, configured to input the target-level shadow image feature set into the shadow detection classification regression network of the initial shadow detection model for learning, and output the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection frame;

[0033] A target shadow mask learning network module is used to input the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning, and output a predicted binary mask of each candidate region box;

[0034] A shadow detection model generation module is configured to construct a model loss function based on the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position, and predicted binary mask of each candidate area frame, to optimize and update the initial shadow detection model to obtain a shadow detection model;

[0035] The shadow area detection module is used to detect the roadside monitoring scene image to be measured according to the shadow detection model to obtain the shadow area and the target area corresponding to the shadow area.

[0036] In one embodiment, the shadow detection and extraction network module includes:

[0037] A feature compression module, configured to input the target-level shadow image dataset into the feature compression structure of the shadow detection and extraction network for feature compression, and output a global compressed feature set;

[0038] A remodeling module is used to input the global compressed feature set into the remodeling structure of the shadow detection and extraction network for reconstruction, and output the target-level shadow image feature set.

[0039] In one embodiment, the shadow detection classification regression network module includes:

[0040] a classification learning module, configured to input the target-level shadow image feature set into a multi-layer first convolutional module and a first fully connected layer to perform category classification learning, and output the predicted target category, the predicted shadow category, and the predicted background category of each detection frame;

[0041] A position regression module is used to input the target-level shadow image feature set into the multi-layer second convolution module and the second fully connected layer to perform two-dimensional detection frame coordinate position learning, and output the predicted target position and the predicted shadow position of each detection frame.

[0042] In one embodiment, the target shadow mask learning network module includes:

[0043] A region extraction network module is used to input the target-level shadow image feature set into the region extraction network of the two-stage mask branch network to perform region extraction and obtain multiple candidate region frames;

[0044] A candidate frame alignment layer module is used to input the multiple candidate region frames into the candidate frame alignment layer of the two-stage mask branch network for resizing to obtain multiple candidate region frames of the same size;

[0045] A mask prediction layer module is used to input the multiple candidate area frames of the same size into the mask prediction layer of the two-stage mask branch network to obtain the predicted binary mask of each candidate area frame.

[0046] In one embodiment, the object-level shadow image dataset includes object annotation data, and the object annotation data includes the true object category, true shadow category, true background category, true object position, true shadow position and true binary mask of each detection frame;

[0047] The shadow detection model generation module includes:

[0048] A classification loss function forming module, configured to form a classification loss function according to the true target category, the true shadow category, the true background category, the predicted target category, the predicted shadow category, and the predicted background category of each detection frame;

[0049] a position regression loss function forming module, configured to form a position regression loss function according to the true target position, the true shadow position, the predicted target position, and the predicted shadow position;

[0050] a shadow offset loss function forming module, configured to obtain a true shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the true target position and the true shadow position, obtain a predicted shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the predicted target position and the predicted shadow position, and form a shadow offset loss function based on the true shadow offset and the predicted shadow offset;

[0051] The mask loss function forming module is configured to form a mask loss function according to the real binary mask and the predicted binary mask.

[0052] The model loss function forming module is configured to construct the model loss function according to the classification loss function, the position regression loss function, the shadow offset loss function and the mask loss function.

[0053] In the shadow detection method and system based on high-position video monitoring, the shadow detection extraction network, the shadow detection classification regression network and the target shadow mask learning network of the initial shadow detection model are constructed, the target and the corresponding shadow labeled in the target-level shadow image data set are learned in pairs, and the target and the shadow are distinguished and detected in multiple dimensions of target category, shadow category, background category, target position, shadow position and binary mask, so that each target area and shadow area can be positioned more accurately, the target and the corresponding shadow in the monitoring video image are detected, which is helpful for subsequent targeted processing of the shadow of each target, roadside parking management and more accurate judgment basis. After the shadow area in the image is detected by the shadow detection method based on high-position video monitoring, the shadow position of irrelevant targets in the image can be protected, and the light direction can be judged according to the detected shadow position and shape, thereby providing more judgment basis for subsequent more comprehensive and intelligent traffic management. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a step flowchart of the shadow detection method based on high-position video monitoring provided by the application.

[0055] Figure 2 is a structure diagram of the shadow detection system based on high-position video monitoring provided by the application. DETAILED DESCRIPTION

[0056] The technical solutions of the application will be further described in detail below with reference to the drawings and examples.

[0057] Please refer to Figure 1 The application provides a shadow detection method based on high-position video monitoring, which comprises the following steps:

[0058] S10, a target-level shadow image data set is constructed;

[0059] S20, the target-level shadow image data set is input into a shadow detection extraction network of an initial shadow detection model for feature extraction, and a target-level shadow image feature set is output;

[0060] S30, inputting the target-level shadow image feature set into the shadow detection classification regression network of the initial shadow detection model for learning, and outputting the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection box;

[0061] S40, inputting the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning, and outputting the predicted binary mask of each candidate region box;

[0062] S50, constructing a model loss function to optimize and update the parameters of the initial shadow detection model based on the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position, and predicted binary mask of each candidate region frame, to obtain a shadow detection model;

[0063] S60 , detecting the roadside monitoring scene image to be measured according to the shadow detection model to obtain a shadow area and a target area corresponding to the shadow area.

[0064] In this embodiment, the target-level shadow image dataset includes multiple traffic scene video images and annotation information for each target and target shadow in the image. Each target includes pedestrians, vehicles, non-motor vehicles, etc. For example, in an image, the targets are vehicle A, vehicle B, vehicle C, pedestrian D, and pedestrian F. The vehicle category can be defined as 1, and the pedestrian category can be defined as 2. For vehicle A, the annotation information includes target category 1, target instance ID 1, the polygonal outline annotation coordinate information of vehicle A, and the polygonal outline annotation coordinate information of vehicle A's shadow. For vehicle B, the annotation information includes target category 1, target instance ID 2, the polygonal outline annotation coordinate information of vehicle B, and the polygonal outline annotation coordinate information of vehicle B's shadow. For pedestrian D, the annotation information includes target category 2, target instance ID 3, the polygonal outline annotation coordinate information of pedestrian D, and the polygonal outline annotation coordinate information of pedestrian D's shadow. The binary mask of the target shadow area is obtained by subtracting the binary mask formed by the target polygon area from the binary mask formed by the target and shadow polygon areas.

[0065] The shadow detection and extraction network can be a convolutional neural network structure such as MobileNet, ResNet (Residual Network), etc., which performs feature extraction on each target-level shadow image in the target-level shadow image dataset and outputs a target-level shadow image feature set.

[0066] The shadow detection classification and regression network consists of a classification branch and a regression branch. The classification branch includes multi-layer convolutional modules. Each convolutional module consists of a convolutional layer, a normalization layer, and an activation function layer. It is used to classify and learn the categories of each target in the target-level shadow image feature set, enabling detection of multiple target categories, shadow categories, and background categories. The regression branch includes multi-layer convolutional modules and fully connected layers, which are used to regress and learn the positions of each target and each shadow. The learning parameters of the position include the center point, width, and height of each target, and the center point, width, and height of each shadow, enabling detection of the target and shadow positions.

[0067] The target shadow mask learning network can use an instance segmentation network, including but not limited to a single-stage or two-stage instance segmentation network, such as Cascade Mask R-CNN, YOLACT, BlendMask, etc., to realize the binary mask learning of each target and shadow.

[0068] The predicted target category, shadow category, background category, target position, and shadow position output by the shadow detection classification and regression network, along with the predicted binary mask output by the target shadow mask learning network, are used to construct a loss function to optimize and train the initial shadow detection model, ultimately obtaining a trained shadow detection model. The shadow detection model detects images of roadside surveillance scenes to determine shadow areas within the image and the corresponding target areas.

[0069] The shadow detection method based on high-position video surveillance provided by the present invention constructs a shadow detection extraction network, a shadow detection classification regression network and a target shadow mask learning network of the initial shadow detection model, performs paired learning on the targets and corresponding shadows marked in the target-level shadow image data set, and distinguishes, detects and identifies targets and shadows from multiple dimensions of target category, shadow category, background category, target position, shadow position and binary mask. It can more accurately locate each target area and shadow area, detect the targets and their corresponding shadows in the monitoring video image, and help to carry out targeted processing of the shadow of each target in the future, carry out roadside parking management and provide more accurate judgment basis. Through the shadow detection method based on high-position video surveillance provided by the present invention, after the shadow area in the image is detected, the privacy of the shadow position of irrelevant targets in the image can be protected, and the light direction can be judged according to the detected shadow position and shape, providing more judgment basis for the subsequent realization of more comprehensive and intelligent traffic management.

[0070] In one embodiment, S20, the target-level shadow image dataset is input into the shadow detection extraction network of the initial shadow detection model for feature extraction, and a target-level shadow image feature set is output, including:

[0071] S210, inputting the target-level shadow image dataset into the feature compression structure of the shadow detection and extraction network for feature compression, and outputting a global compressed feature set;

[0072] S220: Input the global compressed feature set into the remodeling structure of the shadow detection and extraction network for reconstruction, and output the target-level shadow image feature set.

[0073] In this embodiment, the shadow detection and extraction network can be a SENet feature extraction network, which includes two steps, Squeeze and Excitation, to achieve feature weighting in the channel dimension, improving the network's feature extraction capabilities. The input features are compressed along the spatial dimension through a feature compression structure. This feature compression structure includes, but is not limited to, using a global average pooling operation to represent the global distribution of features in the channel dimension C.

[0074] The remodeling structure remodels the feature channel dimension, updating the model parameters of each feature in the channel dimension, then weighting them to the original input features. The final output is the remodeled features, obtaining the target-level shadow image feature set. The remodeling structure includes but is not limited to the use of multiple fully connected layers and nonlinear activation function layers.

[0075] In one embodiment, S30, the object-level shadow image feature set is input into the shadow detection classification regression network of the initial shadow detection model for learning, and the predicted object category, predicted shadow category, predicted background category, predicted object position, and predicted shadow position of each detection frame are output, including:

[0076] S310, inputting the target-level shadow image feature set into the multi-layer first convolution module and the first fully connected layer for category classification learning, and outputting the predicted target category, predicted shadow category, and predicted background category of each detection box;

[0077] S320: Input the target-level shadow image feature set into the multi-layer second convolution module and the second fully connected layer to learn the two-dimensional detection frame coordinate position, and output the predicted target position and predicted shadow position of each detection frame.

[0078] In this embodiment, each layer of the first convolutional module is composed of a convolutional layer, a normalization layer and an activation function layer. The first fully connected layer is used for classification, which maps the feature representation learned by the first convolutional module to a sample label space, and realizes classification of specific categories. Each layer of the second convolutional module is composed of a convolutional layer, a normalization layer and an activation function layer. The second fully connected layer is used for classification, which further learns more accurate position information of the target and the shadow through the second convolutional module and the second fully connected layer, and obtains the position regression information of the target and the position regression information of the shadow. The predicted target position includes the center point and the width and height information of the predicted target detection box, and the predicted shadow position includes the center point and the width and height information of the predicted shadow detection box.

[0079] In one embodiment, S40, the target-level shadow image feature set is input into a target shadow mask learning network of an initial shadow detection model for learning, and a predicted binary mask of each candidate region box is output, including:

[0080] S410, the target-level shadow image feature set is input into a region extraction network of a two-stage mask branch network for region extraction, and a plurality of candidate region boxes are obtained;

[0081] S420, the plurality of candidate region boxes are input into a candidate box alignment layer of the two-stage mask branch network for size adjustment, and a plurality of same-size candidate region boxes are obtained;

[0082] S430, the plurality of same-size candidate region boxes are input into a mask prediction layer of the two-stage mask branch network, and a predicted binary mask of each candidate region box is obtained.

[0083] In this embodiment, the two-stage mask branch network can reduce the computational complexity of the model, improve the running speed of the model, and realize more accurate segmentation. First, the target-level shadow image feature set is input into an RPN (Region Proposal Network) region extraction network to generate a plurality of ROI (Region Of Interest) candidate region boxes. Then, the generated plurality of target and shadow candidate boxes are adjusted to the same size through an ROIAlign (Region Of Interest Align) candidate box alignment layer. Finally, the obtained plurality of same-size candidate region boxes are input into a mask prediction layer to generate the class information and position information of each candidate region box and the binary mask of each candidate region box.

[0084] The region extraction network is primarily responsible for generating multiple candidate bounding boxes for objects and shadows. After feature extraction from the object-level shadow image through the shadow detection and extraction network, a feature map of the object-level shadow image is generated. This map is then divided into W x H regions. The center point of each region is represented by a pixel on the resulting feature map after a series of convolution operations. In each region, k candidate bounding boxes of objects and shadows of varying lengths and widths are generated, and a selection is made to identify the candidate bounding boxes containing both the object and the shadow.

[0085] The role of the candidate box alignment layer is to unify the sizes of the candidate region boxes obtained above. Boxes of different sizes represent regions of different sizes on the feature map. The candidate boxes are unified into the same size for mask prediction.

[0086] Multiple candidate region boxes of the same size are fed into the mask prediction layer to generate category and location information for each candidate box, as well as a binary mask for each candidate box. The mask prediction layer consists of multiple convolutional layers. Through various training parameter combinations and experimental comparisons, a smaller number of convolutional layers and channels is used to achieve accurate and relatively fast model execution, predicting the binary mask for each candidate region box.

[0087] In one embodiment, the object-level shadow image dataset includes object annotation data, where the object annotation data includes a true object category, a true shadow category, a true background category, a true object position, a true shadow position, and a true binary mask of each detection box;

[0088] S50: Based on the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position, and predicted binary mask of each candidate region frame, a model loss function is constructed to optimize and update the parameters of the initial shadow detection model to obtain a shadow detection model, including:

[0089] S510, forming a classification loss function based on the true target category, true shadow category, true background category, predicted target category, predicted shadow category, and predicted background category of each detection frame;

[0090] S520, forming a position regression loss function based on the true target position, the true shadow position, the predicted target position, and the predicted shadow position;

[0091] S530: Obtain a true shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the true target position and the true shadow position; obtain a predicted shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the predicted target position and the predicted shadow position; and form a shadow offset loss function based on the true shadow offset and the predicted shadow offset.

[0092] S540, forming a mask loss function according to the true binary mask and the predicted binary mask;

[0093] S550: Construct a model loss function according to the classification loss function, the position regression loss function, the shadow offset loss function, and the mask loss function.

[0094] In this embodiment, the classification loss function is used to distinguish the categories of targets, shadows, and backgrounds. Classification loss functions include but are not limited to the use of Cross Entropy loss function, Focal Loss loss function and other classification loss functions. Position regression loss functions include but are not limited to the use of L1 Loss mean absolute error loss function, L2 Loss mean square error loss function, IoU Loss intersection-over-union loss function and other position regression loss functions. The shadow offset loss function represents the loss function for learning the offset of the detection frame center point of the target and the corresponding shadow, which can enhance the constraints between the target and the corresponding shadow. The shadow offset loss function includes but is not limited to the use of L1 Loss mean absolute error loss function, L2 Loss mean square error loss function, IoU Loss intersection-over-union loss function and other regression loss functions. The mask loss function can adopt the segmentation loss function and be calculated using the cross entropy loss function.

[0095] See Figure 2 The present invention provides a shadow detection system 100 based on high-position video surveillance. The shadow detection system 100 based on high-position video surveillance includes a data set construction module 10, a shadow detection extraction network module 20, a shadow detection classification regression network module 30, a target shadow mask learning network module 40, a shadow detection model generation module 50, and a shadow area detection module 60. The data set construction module 10 is used to construct a target-level shadow image data set. The shadow detection extraction network module 20 is used to input the target-level shadow image data set into the shadow detection extraction network of the initial shadow detection model for feature extraction, and output a target-level shadow image feature set. The shadow detection classification regression network module 30 is used to input the target-level shadow image feature set into the shadow detection classification regression network of the initial shadow detection model for learning, and output the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection frame.

[0096] The target shadow mask learning network module 40 is used to input the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning, and output the predicted binary mask of each candidate region frame. The shadow detection model generation module 50 is used to construct a model loss function based on the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection frame and the predicted binary mask of each candidate region frame, and perform model optimization and parameter update on the initial shadow detection model to obtain a shadow detection model. The shadow area detection module 60 is used to detect the roadside monitoring scene image to be tested based on the shadow detection model, and obtain the shadow area and the target area corresponding to the shadow area.

[0097] In this embodiment, the description of the dataset construction module 10 may refer to the description of S10 in the above embodiment. The description of the shadow detection and extraction network module 20 may refer to the description of S20 in the above embodiment. The description of the shadow detection classification and regression network module 30 may refer to the description of S30 in the above embodiment. The description of the target shadow mask learning network module 40 may refer to the description of S40 in the above embodiment. The description of the shadow detection model generation module 50 may refer to the description of S50 in the above embodiment. The description of the shadow area detection module 60 may refer to the description of S60 in the above embodiment.

[0098] In one embodiment, the shadow detection and extraction network module 20 includes a feature compression module and a remodeling module. The feature compression module is configured to input the target-level shadow image dataset into the feature compression structure of the shadow detection and extraction network for feature compression, and output a global compressed feature set. The remodeling module is configured to input the global compressed feature set into the remodeling structure of the shadow detection and extraction network for reconstruction, and output a target-level shadow image feature set.

[0099] In this embodiment, the description of the feature compression module can refer to the description of S210 in the above embodiment. The description of the remodeling module can refer to the description of S220 in the above embodiment.

[0100] In one embodiment, the shadow detection classification and regression network module 30 includes a classification learning module and a position regression module. The classification learning module is used to input the target-level shadow image feature set into the multi-layer first convolution module and the first fully connected layer for category classification learning, and output the predicted target category, predicted shadow category, and predicted background category for each detection frame. The position regression module is used to input the target-level shadow image feature set into the multi-layer second convolution module and the second fully connected layer for two-dimensional detection frame coordinate position learning, and output the predicted target position and predicted shadow position for each detection frame.

[0101] In this embodiment, the description of the classification learning module can refer to the description of S310 in the above embodiment. The description of the position regression module can refer to the description of S320 in the above embodiment.

[0102] In one embodiment, the target shadow mask learning network module 40 includes a region extraction network module, a candidate frame alignment layer module, and a mask prediction layer module. The region extraction network module is used to input the target-level shadow image feature set into the region extraction network of the two-stage mask branch network for region extraction to obtain multiple candidate region frames. The candidate frame alignment layer module is used to input multiple candidate region frames into the candidate frame alignment layer of the two-stage mask branch network for size adjustment to obtain multiple candidate region frames of the same size. The mask prediction layer module is used to input multiple candidate region frames of the same size into the mask prediction layer of the two-stage mask branch network to obtain a predicted binary mask for each candidate region frame.

[0103] In this embodiment, the description of the region extraction network module can refer to the description of S410 in the above embodiment. The description of the candidate frame alignment layer module can refer to the description of S420 in the above embodiment. The description of the mask prediction layer module can refer to the description of S430 in the above embodiment.

[0104] In one embodiment, the object-level shadow image dataset includes object annotation data, which includes the true object category, true shadow category, true background category, true object position, true shadow position, and true binary mask of each detection frame. The shadow detection model generation module 50 includes a classification loss function formation module, a position regression loss function formation module, a shadow offset loss function formation module, a mask loss function formation module, and a model loss function formation module. The classification loss function formation module is used to form a classification loss function based on the true object category, true shadow category, true background category, predicted object category, predicted shadow category, and predicted background category of each detection frame.

[0105] The position regression loss function formation module is used to form a position regression loss function based on the true target position, true shadow position, predicted target position, and predicted shadow position. The shadow offset loss function formation module is used to obtain the true shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the true target position and the true shadow position, obtain the predicted shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the predicted target position and the predicted shadow position, and form a shadow offset loss function based on the true shadow offset and the predicted shadow offset. The mask loss function formation module is used to form a mask loss function based on the true binary mask and the predicted binary mask. The model loss function formation module is used to construct a model loss function based on the classification loss function, the position regression loss function, the shadow offset loss function, and the mask loss function.

[0106] In this embodiment, the description of the classification loss function forming module can refer to the description of S510 in the above embodiment. The description of the position regression loss function forming module can refer to the description of S520 in the above embodiment. The description of the shadow offset loss function forming module can refer to the description of S530 in the above embodiment. The description of the mask loss function forming module can refer to the description of S540 in the above embodiment. The description of the model loss function forming module can refer to the description of S550 in the above embodiment.

[0107] In the various embodiments described above, the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of this disclosure. The accompanying method claims provide elements of the various steps in an exemplary order and are not intended to be limited to a specific order or hierarchy.

[0108] Those skilled in the art will also appreciate that the various illustrative logical blocks, modules, and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, modules, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0109] The various illustrative logic blocks described in the embodiments of the present application, or modules can be implemented or operated by the design of a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination thereof. The general-purpose processor can be a microprocessor, alternatively, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration to implement.

[0110] The steps of the methods or algorithms described in the embodiments of the present application can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be provided in an ASIC, which can be provided in a user terminal. Alternatively, the processor and the storage medium can also be provided in different components in the user terminal.

[0111] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A shadow detection method based on high-position video surveillance, characterized in that: include: Construct an object-level shadow image dataset; Inputting the target-level shadow image dataset into the shadow detection extraction network of the initial shadow detection model for feature extraction, and outputting a target-level shadow image feature set; Inputting the target-level shadow image feature set into the shadow detection classification regression network of the initial shadow detection model for learning, and outputting the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection frame; Inputting the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning, and outputting the predicted binary mask of each candidate region box; According to the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position of each detection frame and the predicted binary mask of each candidate area frame, a model loss function is constructed to optimize the initial shadow detection model and update the parameters to obtain a shadow detection model; The roadside monitoring scene image to be measured is detected according to the shadow detection model to obtain a shadow area and a target area corresponding to the shadow area.

2. The shadow detection method based on high-position video surveillance according to claim 1 is characterized in that: The step of inputting the target-level shadow image dataset into a shadow detection extraction network of an initial shadow detection model for feature extraction and outputting a target-level shadow image feature set comprises: Inputting the target-level shadow image dataset into the feature compression structure of the shadow detection and extraction network for feature compression, and outputting a global compressed feature set; The global compressed feature set is input into the remodeling structure of the shadow detection and extraction network for reconstruction, and the target-level shadow image feature set is output.

3. The shadow detection method based on high-position video surveillance according to claim 2, characterized in that: The target-level shadow image feature set is input into the shadow detection classification regression network of the initial shadow detection model for learning, and the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection frame are output, including: Inputting the target-level shadow image feature set into a multi-layer first convolutional module and a first fully connected layer for category classification learning, and outputting the predicted target category, the predicted shadow category, and the predicted background category of each detection frame; The target-level shadow image feature set is input into the multi-layer second convolution module and the second fully connected layer to perform two-dimensional detection frame coordinate position learning, and the predicted target position and the predicted shadow position of each detection frame are output.

4. The shadow detection method based on high-position video surveillance according to claim 3 is characterized in that: The step of inputting the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning and outputting a predicted binary mask of each candidate region box comprises: Inputting the target-level shadow image feature set into a region extraction network of a two-stage mask branch network to perform region extraction, and obtaining multiple candidate region frames; Inputting the multiple candidate region frames into the candidate frame alignment layer of the two-stage mask branch network for resizing to obtain multiple candidate region frames of the same size; The multiple candidate region frames of the same size are input into the mask prediction layer of the two-stage mask branch network to obtain the predicted binary mask of each candidate region frame.

5. The shadow detection method based on high-position video surveillance according to claim 4 is characterized in that: The object-level shadow image dataset includes object annotation data, and the object annotation data includes the true object category, true shadow category, true background category, true object position, true shadow position and true binary mask of each detection frame; The method comprises: constructing a model loss function based on the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position, and predicted binary mask of each candidate region frame to optimize the initial shadow detection model and update the parameters to obtain a shadow detection model, including: Forming a classification loss function according to the true target category, the true shadow category, the true background category, the predicted target category, the predicted shadow category, and the predicted background category of each detection frame; forming a position regression loss function based on the true target position, the true shadow position, the predicted target position, and the predicted shadow position; Obtaining a true shadow offset of a center point of a shadow detection frame relative to a center point of a target detection frame based on the true target position and the true shadow position; obtaining a predicted shadow offset of a center point of a shadow detection frame relative to a center point of a target detection frame based on the predicted target position and the predicted shadow position; and forming a shadow offset loss function based on the true shadow offset and the predicted shadow offset; Forming a mask loss function according to the true binary mask and the predicted binary mask; The model loss function is constructed according to the classification loss function, the position regression loss function, the shadow offset loss function and the mask loss function.

6. A shadow detection system based on high-position video surveillance, characterized in that: include: Dataset construction module, used to construct object-level shadow image dataset; A shadow detection and extraction network module is used to input the target-level shadow image dataset into the shadow detection and extraction network of the initial shadow detection model to perform feature extraction and output a target-level shadow image feature set; a shadow detection classification regression network module, configured to input the target-level shadow image feature set into the shadow detection classification regression network of the initial shadow detection model for learning, and output the predicted target category, predicted shadow category, predicted background category, predicted target position, and predicted shadow position of each detection frame; A target shadow mask learning network module is used to input the target-level shadow image feature set into the target shadow mask learning network of the initial shadow detection model for learning, and output a predicted binary mask of each candidate region box; A shadow detection model generation module is configured to construct a model loss function based on the predicted target category, predicted shadow category, predicted background category, predicted target position, predicted shadow position, and predicted binary mask of each candidate area frame, to optimize and update the initial shadow detection model to obtain a shadow detection model; The shadow area detection module is used to detect the roadside monitoring scene image to be measured according to the shadow detection model to obtain the shadow area and the target area corresponding to the shadow area.

7. The shadow detection system based on high-position video surveillance according to claim 6, characterized in that: The shadow detection and extraction network module includes: A feature compression module, configured to input the target-level shadow image dataset into the feature compression structure of the shadow detection and extraction network for feature compression, and output a global compressed feature set; A remodeling module is used to input the global compressed feature set into the remodeling structure of the shadow detection and extraction network for reconstruction, and output the target-level shadow image feature set.

8. The shadow detection system based on high-position video surveillance according to claim 7, characterized in that: The shadow detection classification regression network module includes: a classification learning module, configured to input the target-level shadow image feature set into a multi-layer first convolutional module and a first fully connected layer to perform category classification learning, and output the predicted target category, the predicted shadow category, and the predicted background category of each detection frame; A position regression module is used to input the target-level shadow image feature set into the multi-layer second convolution module and the second fully connected layer to perform two-dimensional detection frame coordinate position learning, and output the predicted target position and the predicted shadow position of each detection frame.

9. The shadow detection system based on high-position video surveillance according to claim 8, characterized in that: The target shadow mask learning network module includes: A region extraction network module is used to input the target-level shadow image feature set into the region extraction network of the two-stage mask branch network to perform region extraction and obtain multiple candidate region frames; A candidate frame alignment layer module is used to input the multiple candidate region frames into the candidate frame alignment layer of the two-stage mask branch network for resizing to obtain multiple candidate region frames of the same size; A mask prediction layer module is used to input the multiple candidate area frames of the same size into the mask prediction layer of the two-stage mask branch network to obtain the predicted binary mask of each candidate area frame.

10. The shadow detection system based on high-position video surveillance according to claim 9, characterized in that: The object-level shadow image dataset includes object annotation data, and the object annotation data includes the true object category, true shadow category, true background category, true object position, true shadow position and true binary mask of each detection frame; The shadow detection model generation module includes: A classification loss function forming module, configured to form a classification loss function according to the true target category, the true shadow category, the true background category, the predicted target category, the predicted shadow category, and the predicted background category of each detection frame; a position regression loss function forming module, configured to form a position regression loss function according to the true target position, the true shadow position, the predicted target position, and the predicted shadow position; a shadow offset loss function forming module, configured to obtain a true shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the true target position and the true shadow position, obtain a predicted shadow offset of the center point of the shadow detection frame relative to the center point of the target detection frame based on the predicted target position and the predicted shadow position, and form a shadow offset loss function based on the true shadow offset and the predicted shadow offset; A mask loss function forming module, configured to form a mask loss function according to the true binarized mask and the predicted binarized mask; A model loss function forming module is used to construct the model loss function based on the classification loss function, the position regression loss function, the shadow offset loss function and the mask loss function.

Images