A virtual reality security monitoring platform based on deep learning target detection

Abstract

A virtual reality security monitoring platform based on deep learning target detection belongs to the field of intelligent security detection. The weather classification network of the present invention is used to classify the weather of the input image, and different image enhancement models are called according to the output classification results. The image enhancement model includes dark light enhancement, rain removal model and defogging model. This part of the model is improved to have better processing effect and meet the real-time application requirements. The output of the image enhancement model will be used as the input of the YOLOV5 network for target detection. The feature extraction network of the Yolov5 target detection network is improved to make it more lightweight to meet the real-time requirements of the system. The system terminal is developed based on the UE4 engine, provides a three-dimensional virtual interface, deploys the algorithm model on the Web server, and returns the algorithm detection results to the terminal display through the http protocol. The intuitive interface display effect allows security personnel to view and locate campus real-time monitoring information more quickly.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent security detection, and in particular to a virtual reality security monitoring platform based on deep learning target detection. Background Art

[0002] Campus safety has increasingly become a hot topic of social concern. Traditional campus security measures are mainly divided into human defense and physical defense, and security management is carried out through personnel patrols and physical defense. With the continuous advancement of science and technology, technical prevention has become a supplement and extension of traditional security measures. The efficient combination of information technology and campus security is an important part of improving campus security.

[0003] The traditional campus security monitoring and management system only displays the real-time monitoring screen in a unified two-dimensional interface. This type of system is more suitable for reviewing the monitoring screen, which is not conducive to the real-time inspection of security personnel and cannot effectively improve the efficiency of campus security management. Although the YOLO series of target detection algorithms have high detection accuracy and detection speed, they are not well adapted to target detection tasks under severe weather conditions and low light conditions. Based on the UE4 engine to build a virtual scene of a real campus, and then use the improved target detection algorithm to detect targets in real-time monitoring videos, it can adapt to target detection tasks under a variety of severe weather and low light conditions, and transmit real-time video data back to the virtual campus platform, so that real-time monitoring can be viewed in the virtual scene. The system has high real-time performance, intuitive interface display and friendly human-computer interaction, which is conducive to large-scale application.

[0004] The real-time video surveillance system based on BS architecture uses network IP cameras to collect real-time video data, uses RTMP / RTSP protocol to transmit video streams to the background video server, and uses CGI and JavaScript technology to enable multiple smart terminals to log in to the browser to display the monitoring screen. However, this type of system only performs simple video data transmission, is not integrated with deep learning algorithms, and has a single interface display effect. The RCNN target detection algorithm uses a selective search algorithm to generate several areas that may contain objects, uses a convolutional neural network for feature extraction, uses a support vector machine classifier for classification, and finally uses a bounding box regression method for accurate positioning. The FasterR-CNN algorithm proposed by Ren et al. adds a region proposal network after the convolution layer instead of the selective search algorithm, so it has a significant improvement in speed. However, although this series of target detection algorithms has high detection accuracy, there is still computational redundancy, which is not conducive to the application of systems with high real-time requirements. The Yolo series of target detection algorithms are single-stage target detection algorithms. Based on the regression of bounding boxes, classification and regression are performed while generating bounding boxes, which can meet real-time requirements. However, this series of target detection algorithms has poor detection effects on low-resolution images and cannot meet the target detection needs in severe weather conditions. High-speed rail perimeter intrusion target detection under typical severe weather conditions - image processing method based on deep learning [Liu Qinghong. High-speed rail perimeter intrusion target detection under typical severe weather conditions [D]. Beijing Jiaotong University, 2021. DOI: 10.26944 / d.cnki.gbfju.2021.000202.] It is proposed to pre-process the image by removing fog and rain, and then use the processed image as the input of the target detection network, which can effectively improve the target detection effect under severe weather conditions. However, this method will sacrifice the target detection performance in normal weather and is only suitable for target detection in a single severe weather. Summary of the invention

[0005] In view of the defects of the prior art, the present invention provides a virtual reality security monitoring platform based on deep learning target detection. The powerful three-dimensional graphics rendering capability of the UE4 engine is utilized, and a three-dimensional virtual campus scene is built in combination with the actual campus scene. Real-time monitoring data is collected through a network IP camera, and the video data is transmitted to the background server through the RTMP protocol. The improved target detection algorithm is called in the background to perform target detection, and the detection results are responded to the virtual campus terminal using the HTTP protocol. The Yolo target detection algorithm is combined with the image classification and image enhancement algorithm, which can effectively adapt to target detection in various severe weather and low-light conditions and has strong real-time performance. The web service technology is combined with the virtual terminal, and the more intuitive interface display effect allows security personnel to more quickly view and locate campus real-time monitoring information, thereby improving the utilization efficiency of the campus security monitoring platform and the work efficiency of security personnel.

[0006] In order to solve the above technical problems, the technical solution of the present invention is to provide a virtual reality security monitoring platform based on deep learning target detection, which is characterized by comprising the following steps:

[0007] Step 1: Build a backend audio and video server based on Nginx, use the RTMP protocol to obtain remote real-time video data, and respond to the virtual terminal with real-time video data; build a Web backend server based on the Flask framework, and pull video stream data from the audio and video server routing address.

[0008] Step 2: Construct a dataset of images with different weather conditions; collect multiple intact weather condition images and annotate them with weather categories as labels to drive deep neural network training; weather conditions are divided into foggy, rainy, and sunny.

[0009] Step 3: Train the weather state classification model based on the lightweight network model ShuffleNet, frame the real-time video data obtained in step 1, use the image of each frame as the input of the network, and the network will output three weather state classification results (rainy day, foggy day, sunny day); build a lightweight neural network model ShuffleNet, and use the weather state image dataset constructed in step 2 for training; the lightweight neural network model uses grouped convolution to extract features and rearranges the channels after convolution to reduce the amount of calculation while ensuring model efficiency; design a ShuffleNet unit, use 1*1 grouped point-by-point convolution to extract features, and then rearrange the channels; apply 3*3 depth convolution and grouped point-by-point convolution to match the number of channels connected to it.

[0010] Step 4: Design a parallel three-channel connector, which is divided into channel A, channel B and channel C to perform image enhancement operations; use the three classification results obtained in step 3 as the input of the connector respectively, channel A is the rain removal channel, which will integrate the rain removal model and output the rain removal image; channel B is the defogging channel, which will integrate the defogging model and output the defogging image; channel C is an empty channel and will output the original image; all three channels integrate the dark light enhancement algorithm to preprocess the dark light image.

[0011] Step 5: The image enhancement output completed in step 4 is used as the input of the YOLOv5 detection network; the YOLOv5 detection network is widely used in image editing, crowd detection and autonomous driving in practical applications. Compared with the previous version, YOLOv3 designed darknet-53 based on the idea of ​​Resnet, which consists of continuous 3×3 and 1×1 convolution kernels. The algorithm realizes multi-scale training by predicting multi-scale feature maps, thereby further improving the detection accuracy, especially for small targets. The backbone network of YOLOV5 is improved to consist of Shufflechannel and Shuffle block. At the same time, 4 slice operations are used in the upper structure of feature extraction to form the Focus layer, and the ordinary convolution in the original model is replaced by depth-separable convolution. The activation function uses the Swin activation function, thereby reducing network parameters and calculation amount, and improving the target detection speed.

[0012] Step 6: Develop a virtual terminal based on the UE4 engine; build a virtual campus scene according to the actual campus scene, and the virtual terminal obtains real-time video data and detection result data through the http protocol; develop a virtual scene roaming function, and security personnel can switch to the first-person perspective and global roaming perspective to browse the virtual campus after logging into the virtual terminal; set the virtual campus monitoring location icon according to the actual campus monitoring location, and render the chart in different states according to the http response data. Use the ray detection principle to enable users to control the virtual icon and pop up the real-time monitoring screen.

[0013] The present invention adopts the above technical solution and has the following advantages compared with the prior art:

[0014] 1. The present invention first performs image preprocessing on the input of the Yolo target detection algorithm, so as to combine it with the image classification algorithm and the image enhancement algorithm, which can effectively adapt to target detection in various severe weather and low-light conditions and has strong real-time performance; a three-dimensional virtual terminal is developed based on the UE4 engine, and web service technology is combined with the virtual terminal. Through a more intuitive interface display effect, security personnel can more quickly view and locate real-time monitoring information of the campus, thereby improving the utilization efficiency of the campus security monitoring platform and the work efficiency of security personnel.

[0015] 2. The image preprocessing method of the present invention proposes a new target detection architecture, and designs a parallel three-channel connector to combine the weather classification network with the defogging model, the rain removal model, and the dark light enhancement model, so that the Yolo network can adapt to the target detection tasks in foggy, rainy, and low-light conditions while taking into account real-time performance, and at the same time will not reduce the detection performance of the input image in good condition.

[0016] 3. The present invention develops a three-dimensional virtual terminal based on the UE4 engine, combines web service technology with the virtual terminal, and allows security personnel to more quickly view and locate campus real-time monitoring information through a more intuitive interface display effect, thereby improving the use efficiency of the campus security monitoring platform and the work efficiency of security personnel.

[0017] 4. The parallel three-channel connector of the present invention is divided into channel A, channel B and channel C to perform image enhancement operations; the three classification results obtained by the image classification network are respectively used as the input of the connector, channel A is the rain removal channel, which will integrate the rain removal model and output the rain removal image; channel B is the defogging channel, which will integrate the defogging model and output the defogging image; channel C is an empty channel and will output the original image; all three channels integrate the dark light enhancement algorithm to pre-process the dark light image. Through this design, a set of systems can optimize and process multiple images of severe weather conditions at the same time, and the parallel design ensures the real-time performance of the system.

[0018] 5. The defogging model of the present invention is an image defogging algorithm based on dark channel prior, which improves the atmospheric light estimation method and the low-complexity morphological reconstruction method. On the basis of the prior-based method, a new atmospheric light estimation method is proposed, which improves the robustness of the algorithm and solves the overexposure problem. The algorithm complexity is also lower than other algorithms with the same performance.

[0019] 6. The rain removal model of the present invention improves the classic rain removal algorithm based on sparse representation. In the process of the classic algorithm dividing the dictionary into a geometric dictionary and a rain dictionary, there will be clustering errors. When the dictionary is restored to an image, the error is too large, or the background information is filtered too severely and severe blurring occurs. Therefore, we use the high-frequency components of the image to subtract the low-frequency components, and add the new geometric dictionary components obtained to the original low-frequency components. Compared with the original algorithm that only uses the high-frequency components and separates the geometric dictionary obtained after dictionary learning to automatically and directly superimpose the low-frequency components, this algorithm solves the problem of residual errors.

[0020] 7. The dark light enhancement model of the present invention uses a hyperbolic tangent curve to map the image brightness to an ideal level, and then determines the weighting parameters according to the maximum entropy of the image, which can effectively suppress the enhancement of highlight areas and reduce the appearance of noise.

[0021] 8. The Yolo detection network of the present invention improves the backbone network of YOLOV5, making it consist of Shufflechannel and Shuffleblock. At the same time, four slice operations are used in the upper structure of feature extraction to form a Focus layer, and the ordinary convolution in the original model is replaced by a depth-separable convolution. The activation function uses the Swin activation function, thereby reducing network parameters and calculation amount, and improving the target detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the architecture of the virtual reality security monitoring platform of the present invention.

[0023] Figure 2 It is a flow chart of the rain removal algorithm of the present invention.

[0024] Figure 3 It is a flow chart of the defogging algorithm of the present invention.

[0025] Figure 4 It is a flow chart of the dark light enhancement algorithm of the present invention. DETAILED DESCRIPTION

[0026] Example

[0027] Figure 1 A flowchart of a virtual reality security monitoring platform based on deep learning target detection provided in this embodiment specifically includes the following steps:

[0028] Step 1: Build a backend audio and video server based on Nginx, use the RTMP protocol to obtain remote real-time video data, and respond to the virtual terminal with real-time video data; build a Web backend server based on the Flask framework, and pull video stream data from the audio and video server routing address.

[0029] Step 2: Construct a dataset of images in different weather conditions; obtain images through major search engines such as Baidu, Bing, and Google. The dataset includes three types of weather: foggy, rainy, and sunny, with a total of 9,000 images, 3,000 images for each type of weather. The dataset is multi-scene, and the background of each image is different and complex.

[0030] We use mirroring, flipping, cropping, and adding Gaussian noise to enrich the data set and improve the algorithm's generalization ability, thereby better optimizing the network model. We expand the three types of weather pictures to 10,000 each, and select 20% of each type of weather data as a test set without data enhancement to ensure that the algorithm's accuracy is consistent with the actual situation. We perform data enhancement on the remaining 80% of the data as training and validation sets to improve sample balance and diversity.

[0031] Step 3: Based on the lightweight network model ShuffleNet, train the weather status classification model, divide the real-time video data obtained in step 1 into frames, and use the image of each frame as the input of the network; ShuffleNet proposes point-by-point group convolution to effectively save hardware computing power and improve computing speed, and uses channel mixing to help information flow between groups to build a lightweight ShuffleNet network architecture. This classification model uses 16 ShuffleNet units and LeakyReLU activation function to build a weather status classification model.

[0032] The network is pre-trained using the ImageNet dataset to optimize the model parameters and reduce the number of network iterations. First, the training set data is used as the input of the network, and convolution operations are performed to extract image features to form feature maps. Then, the point-by-point grouped convolution operations and channel mixing operations unique to Softmax are propagated backward. After several convolutional layers and pooling layers, Softmax approximates the network to predict the results. The predicted value and actual value of the network are used as the input of the loss function for loss calculation, and the network parameters are updated using the gradient descent algorithm. The best weather classification model is trained through continuous iterations.

[0033] Step 4: Design a parallel three-channel connector, which is divided into channel A, channel B and channel C to perform image enhancement operations; the three classification results obtained in step 3 are used as inputs of the connector respectively.

[0034] Step 4.1: Figure 2 As shown in the figure, when the weather state classification model determines that the image is in a rainy state, the image will be used as the input of channel A. Channel A is the rain removal channel, which will integrate the rain removal model and output the rain removal image.

[0035] The original rain image I is decomposed into the low-frequency part ILf using the bilateral filtering algorithm, and the high-frequency part IHf of the image is obtained by using I-ILf. The principle of the bilateral filter is:

[0036]

[0037]

[0038]

[0039]

[0040] Where (i, j) is the center point coordinate, f(i, j) is the center point coordinate pixel value; point (k, l) is any point in the neighborhood S of the center point (i, j), σ d and σ r is the smoothing parameter; f(k,l) is the pixel value of point (k,l), the spatial distance from point (k,l) to point (i,j) is d(i,j,k,l), the grayscale difference domain value matrix is ​​r(i,j,k,l), ω(i,j,k,l) ​​is the weight matrix of bilateral filtering, and the pixel value of each pixel output after bilateral filtering is g(i,j);

[0041] The high-frequency component IHF contains background information and noise, so the image HOG features are extracted and the high-frequency image dictionary is obtained through sparse matrix encoding and dictionary learning. In the HOG feature descriptor, the distribution (histogram) of the oriented gradient direction is used as a feature. The gradient amplitude is large around edges and corners (areas with sudden changes in intensity), indicating that the edge of the object contains more object shape information than the plane area.

[0042] The IHF is block-taken and dictionary learning is performed to obtain the dictionary D. The dictionary is divided into two categories, rain dictionary and geometric dictionary, using the KMeans nearest neighbor algorithm. Dictionary learning is a branch of signal processing and machine learning. Its goal is to find a framework (called a dictionary) in which some training data are sparsely represented. The sparser the representation, the better the dictionary.

[0043] The rain dictionary is classified again according to the prior information of rain, and the misclassified parts are removed to obtain the final rain dictionary DR1. According to the rain dictionary DR1, the rain component IHFR of the high-frequency image is restored, and IHFG is obtained by using IHF-IHFR. IHFG is bilaterally filtered again and denoised using the weaker BM3D algorithm to obtain a new geometric component IHFG. BM3D relies on the non-local and local features of natural images, that is, a large number of similar spots and image data are highly correlated locally. If these features are verified, the group is correlated in all three dimensions, and a sparse representation of the real signal can be obtained by applying a decorative three-dimensional transformation to the group. Finally, IHFG is added to ILF to obtain the final deraining result.

[0044] Step 4.2: Channel B is the defogging channel, which will integrate the defogging model and output the defogging image;

[0045] The defogging model used in this embodiment is based on the improved DCP algorithm, such as Figure 3 As shown, in any local area of ​​the three-channel input image, the pixel value in at least one channel will approach zero, thus obtaining the dark channel expression:

[0046] Ω(x) is the local area block centered at pixel position x, I C represents a color channel of the input image I, and y represents the pixel value in the region Ω(x). Then the DCP is expressed as:

[0047] I dark (x)→0

[0048] Under bad weather conditions, the captured images will be fogged due to the scattering effect of particles in the air. This process can be expressed as:

[0049]

[0050] x is the pixel position, I(x) is the color intensity of the captured image, J(x) is the original scene brightness to be restored, A is the atmospheric light, and t(x) is the transmittance; the restoration problem of the haze-free image J(x) is transformed into the prediction of the unknown parameters of atmospheric light intensity A and transmittance t(x).

[0051] If the estimated value of the atmospheric light intensity A is too low, the result will be blurred, while if the estimated value is too high, too much noise will be generated. This embodiment proposes a new atmospheric light estimation method, A brightest Indicates the highest value of the original image in the dark channel, A 0.1％ is the average of the first 0.1% brightest pixels in the dark channel, a is the default value of the weight, which is set to 0.6. Using this method will significantly improve the robustness of the estimation.

[0052] A=aA brightest +(1-a)A 0.1％

[0053] The lower bound of transmittance is introduced to prevent color distortion in the sky area of ​​the reconstructed image. t0 is defined as the lower bound of transmittance, and its default value is 0.4.

[0054] Under normal circumstances, the transmittance of the target area will not be lower than 0.6, while the transmittance of the sky area is close to 0. Therefore, the dehazing process of the target area will not be affected by this lower bound. Since DCP dehazing is applied to the entire image, the pixel values ​​of the sky area are not correctly estimated due to the change in transmittance, resulting in a high color intensity in the sky area. However, since the colors of haze and sky are very similar, this incorrect estimation does not affect the overall quality of the result. In the case of heavy haze, the transmittance may be lower than 0.4, resulting in residual haze in the corresponding area. To solve this problem, a set of morphological operations are introduced to reconstruct the transmission map before applying the transmittance lower bound. First, an opening operation is performed. Then perform a closing operation γ; the closing operation is used to fill the holes in the image and enhance the structure of the object, while the opening operation removes small objects in the image while retaining larger objects, ε B is the expansion factor, δ B is the shrinkage factor;

[0055]

[0056] and

[0057] t CO =γ(t C )=δ B [ε B (t C )]

[0058] Finally, the sky and the object are separated and the transmittance t,t is re-estimated by the following method. reconstructed is the transmittance finally used to restore the haze-free image;

[0059]

[0060] This method avoids over-enhancement by estimating the lower limit of scene brightness more robustly; significantly reduces processing time through morphological reconstruction and provides smooth edges.

[0061] Step 4.3: Channel C is an empty channel and the original image will be output;

[0062] Step 4.4: If Figure 4 As shown in the figure, channel A, channel B, and channel C all integrate dark light enhancement algorithms; the hyperbolic tangent curve (HTC) is a monotonically increasing function in the range of (-1, +1). When the image intensity value is normalized to [0, 1], the hyperbolic tangent curve is used for brightness mapping; for RGB images, let the pixel value be I(x, y) = {R(x, y), G(x, y), B(x, y)}; the red, green, and blue components are represented as R, G, and B, respectively, and are normalized to [0, 1], and the pixel position coordinates are (x, y), and the hyperbolic tangent function is obtained:

[0063]

[0064] in k is a scaling factor, and its value is determined according to the brightness of the image.

[0065] In order to suppress the enhancement of highlight areas, the brightness of the output image is weighted as:

[0066] I w (x,y)=wI(x,y)+(1-w)tanh(kI(x,y))

[0067] w is the weighting coefficient, which is used as the average of the RGB channel values ​​to maintain the image intensity distribution;

[0068] Brightness enhancement should cover all pixel grayscales from 0 to 1 to ensure that the image is globally enhanced. To this end, the brightness weighted output image is stretched:

[0069]

[0070] I w (x,y) max and I w (x,y) min are the maximum and minimum values ​​of the image intensity respectively; Is(x,y) represents the pixel value of each point after global enhancement of the image.

[0071] The larger the image entropy, the more information the image carries and the clearer the image. Image entropy is defined as:

[0072]

[0073] p(i) is the ratio of the intensity of the i-th image; the above formula is used to calculate the entropy value of Is(x,y) at different k. When the entropy value is the largest, the corresponding proportional factor k is the optimal value, and vice versa; after completing the image brightness enhancement, the basic bilateral filtering algorithm is used for image denoising.

[0074] Step 5: Use the frame-by-frame image enhancement output completed in step 4 as the input of the YOLOV5 detection network, use Darknet-53 to train the backbone network for classification as a feature extractor, input the image to obtain features of three scales, and then perform multi-scale target detection. Because it is a full convolutional layer, it is compatible with input images of any scale, and the backbone network can use the GPU more efficiently. The network output is a feature map of three scales, each gridcell has 3 anchors, each anchor has 5 (box coordinates) + 80 (80 category probabilities) parameters, 3*85=255. The 13*13 scale corresponds to the 32*32 receptive field of the original image, that is, 1 gridcell is equivalent to the 32*32 area of ​​the original image, and the receptive fields of different scales are responsible for predicting objects of different scales. Through multi-scale fusion, the abstract features of the deep network semantics are fully utilized, and the fine-grained pixel-level underlying features of the shallow network are fully utilized. Therefore, multi-scale fusion and prediction of objects of different scales can be achieved through this structure. Then, the bounding box is screened and filtered by the non-maximum suppression method to filter out low confidence and repeated boxes to obtain the final target detection result. The backbone network of YOLOV5 is improved to consist of Shufflechannel and Shuffleblock. At the same time, four slice operations are used in the upper structure of feature extraction to form the Focus layer. The ordinary convolution in the original model is replaced by a depth-separable convolution, and the activation function uses the Swin activation function, which reduces the network parameters and calculation amount and improves the target detection speed.

[0075] Step 6.1: Develop a virtual terminal based on the UE4 engine. Instantiate the object of the Pawn class. The Pawn class object can be controlled by the user. When the user roams in the scene, the Pawn class object is actually controlled; then the camera component and spring arm component need to be added to the Pawn class object so that the scene can be displayed and scaled. Finally, bind physical devices, such as keyboard keys, so that the Pawn class object can be controlled through the keyboard keys to complete the roaming function of the character in the scene.

[0076] Step 6.2: Declare the camera icon as an Actor class. The Actor class is the basic type of object that can be placed in the virtual scene. Instances of the Actor class can interact with the user. Bind the pawn class object to a physical device such as the right mouse button. When you click the right mouse button, a ray will be emitted in the current direction. For example, if you click the camera icon, a ray will be emitted toward the camera icon. Determine whether the ray hits an object of the Actor class. If so, execute the corresponding logic such as opening the real-time monitoring screen. The user can view the real-time monitoring screen in the virtual scene.

[0077] Step 6.3: Write the http data request code in the Actor class blueprint, call the CallURL communication node provided by the VaRest plug-in to obtain the server address and obtain the detection data information through the Get method. Judge the obtained detection data information, and if the detection data is obtained, communicate between blueprints, call the monitoring site icon blueprint in the communication class blueprint, and use the setmaterial node to highlight the monitoring site icon.

[0078] The above is only a preferred embodiment of the present invention, and is not any formal or substantial limitation of the present invention. It should be pointed out that, for ordinary technicians in this technical field, without departing from the method of the present invention, several improvements and supplements can be made, and these improvements and supplements should also be regarded as the protection scope of the present invention. Any technician familiar with this profession, without departing from the spirit and scope of the present invention, can make some changes, modifications and evolutions of the equivalent changes made by using the technical content disclosed above, which are all equivalent embodiments of the present invention; at the same time, any changes, modifications and evolutions of any equivalent changes made to the above embodiments based on the essential technology of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A virtual reality security monitoring platform based on deep learning target detection, characterized by The steps include: Step 1: Build a backend audio and video server based on Nginx, use the RTMP protocol to obtain remote real-time video data, and respond to the virtual terminal with real-time video data; Build a Web backend server based on the Flask framework to pull video stream data from the routing address of the audio and video server; Step 2: Construct a dataset of images with different weather conditions; collect multiple intact weather condition images and annotate them with weather categories as labels to drive deep neural network training; weather conditions are divided into foggy, rainy, and sunny; Step 3: Train the weather state classification model based on the lightweight network model ShuffleNet, perform frame processing on the real-time video data obtained in step 1, use the image of each frame as the input of the network, and the network will output three weather state classification results; build a lightweight neural network model ShuffleNet, and use the weather state image dataset constructed in step 2 for training; the lightweight neural network model uses grouped convolution to extract features and rearranges the channels after convolution to reduce the amount of calculation while ensuring model efficiency; design a ShuffleNet unit, use 1*1 grouped point-by-point convolution to extract features, and then rearrange the channels; apply 3*3 depth convolution and grouped point-by-point convolution to match the number of channels connected to it; Step 4: Design a parallel three-channel connector, which is divided into channel A, channel B and channel C to perform image enhancement operations; the three classification results obtained in step 3 are used as the input of the connector respectively, channel A is the rain removal channel, which will integrate the rain removal model and output the rain removal image; channel B is the defogging channel, which will integrate the defogging model and output the defogging image; channel C is an empty channel and will output the original image; all three channels integrate the dark light enhancement algorithm to preprocess the dark light image; Step 5: The image enhancement output completed in step 4 is used as the input of the YOLO v5 detection network; the backbone network of the YOLO v5 is composed of Shuffle channel and Shuffle block, and four slice operations are used in the upper structure of feature extraction to form a Focus layer, and the ordinary convolution in the original model is replaced by a depth-separable convolution, and the activation function uses the Swin activation function; Step 6: Develop a virtual terminal based on the UE4 engine; build a virtual campus scene according to the actual campus scene, and the virtual terminal obtains real-time video data and detection result data through the http protocol; develop a virtual scene roaming function, and security personnel can switch to the first-person perspective and global roaming perspective to browse the virtual campus after logging into the virtual terminal; set the virtual campus monitoring site icon according to the actual campus monitoring site, render the chart in different states according to the http response data, use the ray detection principle to enable users to control the virtual icon, and pop up the real-time monitoring screen.

2. The virtual reality security monitoring platform according to claim 1 is characterized in that The specific process of step 2 above is as follows: construct a dataset of images of different weather conditions; obtain images through a search engine. The dataset includes three types of weather: foggy, rainy, and sunny, with a total of 9,000 images, 3,000 images for each type of weather, and the dataset is for multiple scenes; By using mirroring, flipping, cropping, and adding Gaussian noise to enrich the data set, the three types of weather pictures were expanded to 10,000 each. 20% of each type of weather data was selected as the test set without data enhancement, and the remaining 80% of the data was enhanced as training and validation sets to improve sample balance and diversity.

3. The virtual reality security monitoring platform according to claim 2 is characterized in that The specific process of step 3 is as follows: based on the lightweight network model ShuffleNet, the weather status classification model is trained, the real-time video data obtained in step 1 is processed by frame, and each frame image is used as the input of the network; at the same time, channel mixing is used to help the information flow between groups to construct a lightweight ShuffleNet network architecture; This classification model uses 16 ShuffleNet units and LeakyReLU activation function to build a weather status classification model; The network is pre-trained using the ImageNet dataset. First, the training set data is used as the input of the network, and convolution operations are performed to extract image features to form feature maps. Then, the network is propagated backward through Softmax's unique point-by-point grouped convolution operations and channel mixing operations. After several convolutional and pooling layers, Softmax approximates the network to predict the results. The network's predicted values ​​and actual values ​​are used as inputs to the loss function for loss calculation, and the gradient descent algorithm is used to update the network parameters. The optimal weather classification model is trained through continuous iterations.

4. The virtual reality security monitoring platform according to claim 3 is characterized in that The specific process of step 4 above is: Step 4.1: When the weather state classification model determines that the image is in a rainy state, the image will be used as the input of channel A. Channel A is the rain removal channel, which will integrate the rain removal model and output the rain removal image; The original rain image I is decomposed into the low-frequency part ILf using the bilateral filtering algorithm, and the high-frequency part IHf of the image is obtained by using I-ILf. The principle of the bilateral filter is: Where (i, j) is the center point coordinate, f(i, j) is the center point coordinate pixel value; point (k, l) is any point in the neighborhood S of the center point (i, j), σ d and σ r is the smoothing parameter; f(k,l) is the pixel value of point (k,l), the spatial distance from point (k,l) to point (i,j) is d(i,j,k,l), the grayscale difference domain value matrix is ​​r(i,j,k,l), ω(i,j,k,l) ​​is the weight matrix of bilateral filtering, and the pixel value of each pixel output after bilateral filtering is g(i,j); The high-frequency part IHF contains background information and noise. The image HOG features are extracted and the high-frequency image dictionary is obtained through sparse matrix encoding and dictionary learning. In the HOG feature descriptor, the distribution of oriented gradient directions is used as a feature. The gradient amplitude is large around the area where the intensity changes suddenly, indicating that the edge of the object contains more object shape information than the plane area. The IHF is taken into blocks and dictionary learning is performed to obtain the dictionary D. The KMeans nearest neighbor algorithm is used to divide the dictionary into two categories: rain dictionary and geometric dictionary. The rain dictionary is classified again according to the prior information of rain, and the misclassified parts are removed to obtain the final rain dictionary DR1; the rain component IHFR of the high-frequency image is restored according to the rain dictionary DR1, and IHFG is obtained by using IHF-IHFR. IHFG is bilaterally filtered again and denoised using the BM3D algorithm to obtain a new geometric component IHFG, and IHFG is added to ILF to obtain the final deraining result; Step 4.2: Channel B is the defogging channel, which will integrate the defogging model and output the defogging image; The defogging model used is based on the improved DCP algorithm. In any local area of ​​the three-channel input image, at least one pixel value in one channel will approach zero, and the dark channel expression is obtained: Ω(x) is the local area block centered at pixel position x, I C represents a color channel of the input image I, and y represents the pixel value in the Ω(x) area; then the DCP is expressed as: I dark (x)→0 Under bad weather conditions, the captured images will be fogged, and this process can be expressed as: x is the pixel position, I(x) is the color intensity of the captured image, J(x) is the original scene brightness to be restored, A is the atmospheric light, and t(x) is the transmittance. The restoration problem of the haze-free image J(x) is transformed into predicting the unknown parameters of atmospheric light intensity A and transmittance t(x). If the estimated value of atmospheric light intensity A is too low, the result will be blurred, while if the estimated value is too high, too much noise will be generated; A=aA brightest +(1-a)A 0.1％ Among them A brightest Indicates the highest value of the original image in the dark channel, A 0.1％ is the average value of the first 0.1% brightest pixels in the dark channel, and a is the default value of the weight, which is set to 0.6 to improve the robustness of the estimated value of the atmospheric light intensity A; Define t0 as the lower bound of transmittance, with a default value of 0.4, to prevent color distortion in the sky area of ​​the reconstructed image; Under normal circumstances, the transmittance of the target area will not be lower than 0.6, while the transmittance of the sky area is close to 0. Therefore, the dehazing process of the target area will not be affected by this lower bound. Since DCP dehazing is applied to the entire image, the pixel value of the sky area is not correctly estimated due to the change in transmittance, resulting in a high color intensity in the sky area. In the case of heavy haze, the transmittance may be lower than 0.4, resulting in residual haze in the corresponding area; Before applying the transmittance lower bound, a set of morphological operations are introduced to reconstruct the transmission map. First, the opening operation Then perform the closing operation γ; The closing operation is used to fill the holes in the image and enhance the structure of the object, while the opening operation removes small objects in the image while retaining larger objects. B is the expansion factor, δ B is the shrinkage factor; and t CO =γ(t C )=d B [e B (t C )] Finally, the sky and the object are separated and the transmittance t,t is re-estimated by the following method. reconstructed is the transmittance finally used to restore the haze-free image; Step 4.3: Channel C is an empty channel and the original image will be output; Step 4.4: Channel A, channel B, and channel C all integrate dark light enhancement algorithms; the hyperbolic tangent curve HTC is a monotonically increasing function in the range of (-1, +1). When the image intensity value is normalized to [0, 1], the hyperbolic tangent curve is used for brightness mapping; for RGB images, let the pixel value be I(x, y) = {R(x, y), G(x, y), B(x, y)}; the red, green, and blue components are represented as R, G, and B, respectively, and are normalized to [0, 1]. The pixel position coordinates are (x, y), and the hyperbolic tangent function is obtained: in k is the scaling factor, and its value is determined according to the brightness of the image; In order to suppress the enhancement of highlight areas, the brightness of the output image is weighted as: I w (x,y)=wI(x,y)+(1-w)tanh(kI(x,y)) w is the weighting coefficient, which is used as the average of the RGB channel values ​​to maintain the image intensity distribution; Brightness enhancement should cover all pixel grayscales from 0 to 1 to ensure that the image is globally enhanced. To this end, the brightness weighted output image is stretched: I w (x,y) max and I w (x,y) min are the maximum and minimum values ​​of the image intensity value respectively; Is(x,y) represents the pixel value of each point after global enhancement of the image; The larger the image entropy, the more information the image carries and the clearer the image. Image entropy is defined as: p(i) is the ratio of the intensity of the i-th image; the above formula is used to calculate the entropy value of Is(x,y) at different k. When the entropy value is the largest, the corresponding proportional factor k is the optimal value, and vice versa; After image brightness enhancement is completed, the basic bilateral filtering algorithm is used to perform image denoising.

5. The virtual reality security monitoring platform according to claim 4 is characterized in that The specific process of step 5 above is: The frame-by-frame image enhancement output completed in step 4 is used as the input of the YOLO V5 detection network. The backbone network trained for classification using Darknet-53 is used as the feature extractor. After the image is input, features of three scales are obtained, and multi-scale target detection is performed later. The network output is a feature map of three scales. Each grid cell has three anchors, and each anchor has 5+80 parameters, 3*85=255; 5 corresponds to the box coordinates, and 80 corresponds to the category probability; the 13*13 scale corresponds to the 32*32 receptive field of the original image, that is, 1 grid cell is equivalent to the 32*32 area of ​​the original image, and receptive fields of different scales are responsible for predicting objects of different scales; then the bounding box is screened and filtered by the non-maximum suppression method to filter out low-confidence and repeated boxes to obtain the final target detection result.

6. The virtual reality security monitoring platform according to claim 4, characterized in that The specific process of step 6 above is: Step 6.1: Develop a virtual terminal based on the UE4 engine. When the user roams in the scene, the object of the Pawn class is actually controlled. It is necessary to add a camera component and a spring arm component to the Pawn class object so that the scene can be displayed and scaled. Bind the physical device to control the Pawn class object and complete the roaming function of the character in the scene. Step 6.2: Declare the camera icon as an Actor class, bind the pawn class object to the physical device, and execute the corresponding logic; Step 6.3: Write the http data request code in the Actor class blueprint, call the Call URL communication node provided by the Va Rest plug-in to obtain the server address and obtain the detection data information through the Get method; judge the obtained detection data information, and if there is detection data, communicate between blueprints, call the monitoring site icon blueprint in the communication class blueprint, and use the set material node to highlight the monitoring site icon.

Images