Portable explosion-proof pan-tilt camera, monitoring system and method thereof

Abstract

The invention relates to a portable explosion-proof pan-tilt camera, a monitoring system and a monitoring method. The portable anti-explosion pan-tilt camera comprises a base, a mounting box is rotationally connected to the upper portion of the base, a connecting pipe is mounted at one end of the mounting box, a first rotating shaft is rotationally connected into the connecting pipe, a shell is mounted at the end, away from the mounting box, of the first rotating shaft, and a camera body is mounted in the shell; the mounting box and the shell are made of stainless steel materials or anti-explosion materials, so that the camera has the anti-explosion and anti-corrosion characteristics, a lens is mounted at the front end of the shell in front of the camera body, a wiping mechanism is mounted on the surface of the lens, and a pickup is mounted in the shell below the camera body; the camera body is used for continuously acquiring video data, forming a real-time video data stream and sending the real-time video data stream to the memory and the communication module; the communication module is used for sending the real-time video data stream acquired by the camera body to a server of a monitoring center and other portable explosion-proof pan-tilt cameras; and the video monitoring module is used for receiving the real-time video data stream acquired by the camera body, processing the data stream, and sending a processing result to a monitoring center through the communication module.

Description

Technical Field

[0001] The present invention relates to a camera and a monitoring system, and in particular to a portable explosion-proof pan / tilt camera, a monitoring system and a method. Background Art

[0002] Special explosion-proof equipment is required in flammable and explosive environments. For flammable and explosive environments, real-time monitoring of the surrounding environment and on-site monitoring when safety accidents occur are very important. Explosion-proof cameras are mainly used to monitor flammable and explosive environments on-site. The main function of explosion-proof cameras is to shoot video images, but they can only obtain video images of the monitored places. Existing explosion-proof cameras can shoot images hundreds of meters or even thousands of meters away by using large zoom cameras.

[0003] At present, in addition to video images, audio signals are also extremely important when conducting on-site inspections in flammable and explosive environments. It is difficult to obtain a comprehensive picture of the scene by only looking at the video images. In the event of a safety accident, the video signal is easily blocked by obstacles, smoke, etc., which reduces the effectiveness of the video images. At the same time, when in use, dust in the air gathers on the surface of the camera lens, affecting the camera's shooting picture and reducing the clarity of the shooting picture. At the same time, dust generated by an accident can easily enter the camera through the heat dissipation holes of the camera body, which may corrode the internal components of the camera and affect the normal operation of the camera.

[0004] The invention patent "A Convenient Explosion-proof PTZ Camera" with application number 202310497541.9 discloses an explosion-proof PTZ camera suitable for use in flammable and explosive environments. However, as my country's mine security requirements continue to improve, video surveillance-related systems have been widely used in command, production organization, transmission services, operation monitoring, emergency rescue, and public security prevention, playing an irreplaceable role. According to incomplete statistics, there are currently millions of video surveillance channels in various mining units, and the volume of video surveillance data is huge, but most video image data is not fully mined in real time, and there are great challenges in the intelligent analysis process based on video images. Due to the timeliness of the production process, the video acquisition and real-time analysis requirements and service quality requirements of underground mine explosion-proof PTZ cameras are higher, mainly reflected in that when the mine explosion-proof PTZ camera captures a sudden safety accident, the monitoring center and the security department need to obtain information as soon as possible and immediately arrange rescue or corresponding processing. Therefore, when encountering emergencies, being able to effectively, quickly, safely and efficiently evacuate, guide and rescue personnel becomes a key and difficult issue in the operation and management of mining monitoring systems.

[0005] Although mining units have taken a lot of measures to deal with the above emergencies, such as increasing safety inspection personnel and installing a large number of video surveillance equipment to improve supervision capabilities, it is difficult to make timely rescue operations in a short period of time by relying solely on the manual service of staff when facing emergencies. At present, the safety operation management process of most explosion-proof pan-tilt cameras and monitoring systems is still in a passive management mode, that is, video surveillance is mainly used for real-time viewing and inspection by mine comprehensive video surveillance management personnel, as well as video tracing and playback of emergencies. This operation mode has the following shortcomings: there are many video images displayed in the monitoring center, and patrolling the monitoring area through video cannot timely discover problems in the production monitoring area. When emergencies occur in the production monitoring operation area, such as rapid gathering of crowd density, group abnormal behavior, and individual abnormal behavior, it is difficult to capture the event process in time from the monitoring screen; the monitoring personnel have high work intensity and are prone to visual fatigue, which is not conducive to long-term video monitoring operations. Therefore, for the massive monitoring video data of the monitoring system, it is urgent to introduce new technologies to realize deep mining of video data, give full play to the deeper data value of video monitoring data, and provide strong support for the operation services of mining units.

[0006] Therefore, in view of the shortcomings of traditional video surveillance systems, in recent years, with the vigorous development of machine vision and artificial intelligence technology, intelligent video surveillance systems have emerged. Compared with traditional monitoring systems, intelligent monitoring systems do not require human intervention. They can use algorithms to process video sequences collected by front-end cameras in real time, automatically detect and track moving targets in the scene, understand their behavioral intentions and discover suspicious situations. This frees monitoring personnel from the heavy and boring duty observation tasks, greatly reduces the waste of human resources, and improves monitoring efficiency. The emergence of intelligent video surveillance systems has enabled video surveillance to develop from the early "acquisition storage + manual analysis" working mode to the "intelligent" working mode. In response to the needs of different scenarios under mine production video surveillance, a large number of cameras are installed in the mine. For the comprehensive video surveillance center responsible for all video surveillance operations and management, the workload of different monitoring tasks in many scenarios is very large, and it is difficult to perceive all video surveillance content in a timely and accurate manner. Taking the actual operation of production video surveillance in a 500,000-ton production capacity mine as an example, the mine has installed a total of 273 explosion-proof pan-tilt cameras of various types. The management of multiple video channels is currently only managed by the management personnel of the integrated video surveillance center. The integrated video surveillance center is equipped with a large screen for unified command and can access 40 channels of real-time video surveillance images at the same time.

[0007] In the current operation and management process, monitoring personnel manually divide the cameras into different scenes. On the one hand, this brings a large workload to daily maintenance and management. On the other hand, current video surveillance only provides functions such as video acquisition, preview, transmission and storage, while analysis and recognition of video content need to be done manually, which is labor-intensive and prone to errors.

[0008] Therefore, through years of practice and continuous improvement, based on the invention patent "A portable explosion-proof pan-tilt camera" with application number 202310497541.9, we have proposed a new portable explosion-proof pan-tilt camera, monitoring system and its application method that are suitable for the needs of modern artificial intelligence. Summary of the invention

[0009] The objectives of the present invention are achieved through the following technical solutions.

[0010] To this end, the present invention provides a portable explosion-proof PTZ camera, a monitoring system and a method to solve the above-mentioned problems in the prior art.

[0011] According to another aspect of the present invention, a portable explosion-proof pan / tilt camera is provided. Figure 1 As shown. It includes a base, a mounting box is rotatably connected above the base, a connecting pipe is installed at one end of the mounting box, a rotating shaft 1 is rotatably connected inside the connecting pipe, a shell is installed at one end of the rotating shaft 1 away from the mounting box, a camera body is installed inside the shell, the mounting box and the shell are both made of stainless steel or explosion-proof material, so that they have explosion-proof and corrosion-resistant properties, a lens is installed at the front end of the shell in front of the camera body, a wiping mechanism is installed on the surface of the lens, and a pickup is installed inside the shell below the camera body; The camera body is used to continuously collect video data, form a real-time video data stream and send the real-time video data stream to the storage and communication module; A communication module is used to send the real-time video data stream collected by the camera body to the server in the monitoring center; The video monitoring module is used to receive the real-time video data stream collected by the camera body, process the data stream, and then send the processing result to the monitoring center through the communication module; The video monitoring module is also used to achieve: annotating the real-time video images collected by the portable explosion-proof pan-tilt camera in real time to obtain the annotated key points; performing video target segmentation on the annotated video images, establishing the correspondence between the same targets across frames, and achieving a more detailed representation of the target object. It divides the pixels into the foreground target of interest and the background area in the subsequent entire video sequence according to the target object mask given in the first frame, and generates a segmentation mask for the foreground target; performing video target tracking on the video images after video target segmentation, predicting the exact position of the target in the video image in the subsequent video sequence according to the rectangular box of the target of interest given in the first frame, and generating a rectangular bounding box; establishing a visual correspondence based on the cycle consistency of the video images after video target tracking, and establishing a feature representation of the correspondence between targets across video frames; using a sliding window to slide between consecutive video frames, and comparing the current video frame in the sliding window with the video frame previously scanned by the window, outputting a normal or abnormal result according to the comparison result, and issuing an alarm message when an abnormal result is output.

[0012] According to one aspect of the present invention, the communication module may be a 5G wireless communication module for communicating with other explosion-proof PTZ cameras or monitoring centers through a 5G network. The communication module may also be a 6G wireless communication module, a 4G wireless communication module, a wifi wireless communication module, a near field communication module, a Bluetooth communication module, or a wired communication module.

[0013] According to one aspect of the present invention, the present invention proposes a monitoring system based on a portable explosion-proof pan-tilt camera, such as Figure 2 The monitoring system comprises a plurality of portable explosion-proof PTZ cameras arranged in various positions; a server connected to the plurality of portable explosion-proof PTZ cameras for real-time transmission of video data streams; a storage array connected to the server for storing the video data streams received by the server; a video analysis and processing module connected to the server and the storage array respectively for analyzing and processing the received video data streams; the server is also used to receive the video detection results sent by the portable explosion-proof PTZ cameras and respond to the alarm information.

[0014] The video analysis and processing module is also used to achieve: annotating the real-time video images collected by the portable explosion-proof pan-tilt camera in real time to obtain the annotated key points; performing video target segmentation on the annotated video images, establishing the correspondence between the same targets across frames, and realizing a more detailed representation of the target object. It divides the pixels into the foreground target of interest and the background area in the subsequent entire video sequence according to the target object mask given in the first frame, and generates a segmentation mask for the foreground target; performing video target tracking on the video images after video target segmentation, predicting the exact position of the target in the video image in the subsequent video sequence according to the rectangular box of the target of interest given in the first frame, and generating a rectangular bounding box; establishing a visual correspondence based on the cycle consistency for the video images after video target tracking, and establishing a feature representation of the correspondence between targets across video frames; using a sliding window to slide between consecutive frames of the video, and comparing the current video frame in the sliding window with the video frame previously scanned by the window, outputting a normal or abnormal result according to the comparison result, and issuing an alarm message when an abnormal result is output.

[0015] According to one aspect of the present invention, the present invention proposes a monitoring method based on video images collected by a portable explosion-proof pan-tilt camera, such as Figure 3 shown.

[0016] The method comprises the following steps: Step S100, the portable explosion-proof PTZ camera collects real-time video images from all directions in real time; Step S200, annotating the video image to obtain annotated key points; Step S300, performing video object segmentation on the annotated video image, establishing a correspondence between the same objects across frames, and achieving a more detailed representation of the target object. It divides the pixels into the foreground object of interest and the background area in the subsequent entire video sequence according to the target object mask given in the first frame, and generates a segmentation mask of the foreground object; Step S400, performing video target tracking on the video image after the video target segmentation, predicting the accurate position of the target in the video image in the subsequent video sequence according to the rectangular frame of the target of interest given in the first frame, and generating a rectangular bounding box; Step S500, for the video image after video target tracking, a visual correspondence is established based on cycle consistency, and a feature representation of the correspondence of the target across video frames is established; Step S600, slide a sliding window between consecutive video frames, and compare the current video frame in the sliding window with the video frame previously scanned by the window, output a normal or abnormal result based on the comparison result, and issue an alarm message when an abnormal result is output.

[0017] This portable explosion-proof PTZ camera has a microphone installed under the camera body. The microphone can collect the audio of the monitoring scene and transmit the audio signal. The audio signal of the monitoring scene can be understood from a long distance, effectively improving the effectiveness of the video image. The overall structure of the camera is small and easy to carry.

[0018] The present invention can achieve corresponding beneficial effects.

[0019] 1) Through a detailed investigation of the current intelligence level of mines, a detailed analysis of the needs for intelligent identification of mine production based on video surveillance data is conducted. On this basis, a mine intelligent analysis model is constructed, and the implementation plan of intelligent analysis of mine production is studied.

[0020] 2) Through semantic analysis of key scenes in mine production monitoring images, it provides explainable labeled data for massive video data, provides basic data for the mine's classified management of video surveillance, and provides a rich data set for further deep learning in different mine segmentation scenarios.

[0021] 3) By constructing estimation models for mine production in different scenarios, real-time perception of the mine production process can be achieved, providing accurate data basis for mine production management. Especially in the event of emergencies, intelligent estimation based on video analysis can quickly perceive the current situation on site and provide accurate and reliable information for emergency response.

[0022] 4) By constructing a small sample model based on meta-learning, accurate identification in the mine production process can be achieved with a small number of labeled samples, reducing false alarms caused by workers "crossing the line" and thus improving the efficiency of safety supervision.

[0023] 5) Through in-depth modeling and analysis of absentee workers' abnormal behaviors on the production floor, the behaviors of absentee workers in scenarios such as fatigue, falling, tripping, and abnormal group running on the production floor can be identified and judged, thereby improving managers' timely control of abnormal events and reducing the associated risks caused by special abnormal behavior events. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Attached Figure 1 A schematic diagram of the composition of a portable explosion-proof pan-tilt camera according to an embodiment of the present invention is shown.

[0025] Attached Figure 2A schematic diagram of the composition of a monitoring system based on a portable explosion-proof PTZ camera according to an embodiment of the present invention is shown.

[0026] Attached Figure 3 A schematic diagram of a monitoring method based on video images collected by a portable explosion-proof pan-tilt camera according to an embodiment of the present invention is shown.

[0027] Attached Figure 4 A schematic diagram of a neural network structure according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] A large amount of video surveillance data is generated every day for video surveillance data in different scenarios. For most video storage data, due to a large number of storage difficulties, the video storage time is about 15 to 180 days. Video surveillance is stored in an unstructured form from acquisition to storage. From the perspective of the video data itself, there is no label attribute. Therefore, most of the current video surveillance data is used for post-verification and tracing. The comprehensive video surveillance center lacks further effective mining and utilization of videos, and there are few studies and applications related to self-perception of video images. To this end, by constructing a deep convolutional neural network, key scenes such as tunnels, fully mechanized mining faces, tunnels, maintenance platforms, freight yards, and intersections have been classified, and a visualization study of scene classification has been conducted. The semantic segmentation of key scenes is increased to improve the semantic characteristics of video images and lay a solid foundation for subsequent related intelligent analysis.

[0031] Raw video data usually contains a lot of redundant information. If it is directly operated without preprocessing, the computational complexity will be greatly increased and the detection difficulty will be increased. Therefore, before detecting and processing the video, it is necessary to first analyze the event content contained in it and extract the video event feature expression from it. The feature is required to be highly descriptive and discriminative. Descriptiveness requires that the feature can accurately describe the target behavior and motion state in the video scene and characterize the characteristics of the event, while discriminativeness means that the extracted features can effectively distinguish normal events from abnormal events. The video event feature expression has a profound impact on the abnormal event detection results.

[0032] Depending on the type of features used, video event feature expression methods are divided into expression methods based on underlying visual features and expression methods based on high-level semantic features.

[0033] The underlying visual features refer to the original state of the objective existence of the target in the video scene event perceived by the human eye, which is mainly divided into two types: motion features and appearance features. Motion features focus on describing the motion state of the target, including speed and direction, while appearance features focus on describing the size, shape and posture information of the target. Appearance features are also an important attribute for describing the state of the target in the video scene event. They mainly provide appearance information such as the size, shape and color of the target, and are suitable for handling the description and detection of abnormal events such as target intrusion.

[0034] High-level semantic feature expression usually uses target detection and tracking technology to extract information about high-level semantic objects (workers, equipment, etc.) in video scenes, and has a higher level of interpretation of the real semantics contained in video events. Among them, the most representative is the target trajectory feature. The trajectory feature is usually directly represented by a sequence of spatial coordinates of the moving target in multiple frames of video images. In addition, there are many different representation methods such as curvature features, linear dynamic systems, and motion histograms. Derivative features based on trajectory information are also a common expression method. Although high-level features such as trajectories contain clear semantic information, they are often affected by the performance of target detection and tracking algorithms, and can only better describe the content of video events in simple scenes with clear and sparse targets. For crowded scenes with severe occlusion between moving targets, the expression method based on high-level semantic features will fail because it is impossible to accurately locate the position of a single target and extract its trajectory.

[0035] Taking into account the shortcomings of the above two event feature expression methods, in recent years, with the rise of convolutional neural networks (CNNs) and their excellent performance in feature extraction, we can use CNN to convert pixel-level underlying visual features into high-level semantic features to describe the target motion state at a macro level.

[0036] Semantic segmentation technology is a basic application research in image interpretability work. Through the application of semantic segmentation models, the semantic elements of the content in mine video surveillance images can be fully perceived, and the extraction and segmentation of target pixel-level features can be realized. The automatic understanding of mine video surveillance scenes can be achieved by performing refined reasoning and prediction on the label classification of each pixel of the input image. Each pixel is labeled according to the target entity category it is surrounded by. Based on this research, more detailed scene task research can be carried out, such as semantic segmentation-based target detection, target tracking and other specific tasks. At the same time, by extracting the semantics of the mine scene and integrating it with the target recognition task, the automatic generation of the mine scene target data set can be realized, providing rich training samples for other supervised learning.

[0037] Given a sequence of video frames and the initial mask label of the first frame, the trained feature extraction network and ensemble enhanced attention are used to calculate the similarity matching matrix between the reference frame with known labels and the different feature vectors of the subsequent unlabeled video sequence frames to complete pixel-level matching. When processing ensemble enhanced attention, it is expected that the label can adaptively follow the appearance of the target of interest over time. Since there may be multiple objects to be tracked, the entire reference feature is regarded as the key part to adaptively enhance its important part area. Then, the integer values ​​with the highest similarity score in each row of the attention matching matrix are selected as the basis for weight propagation pixels. To reduce data noise interference, the integer value can be set to 10. These attention matching matrices are then used to propagate the required pixels with their respective weight scores. The number of reference frames M can be seen as an important way to better utilize the rich contextual information to improve the prediction quality. When M = 1, the model can only use a single reference frame information, which will cause the tracking of the target pixel to drift over time due to error accumulation. As M increases, the model can effectively use historical information to limit the pixel drift problem. However, too much historical information may bring noise data and more calculations, so it is necessary to adjust the size of M according to the actual situation. In the system of the present invention, the final size of M is set to 6. Finally, all predictions are averaged, and the highest value of each pixel in each classification channel is used as the final label. Finally, the pixel-level segmentation of each video frame is completed by upsampling the feature labels to the original input image size.

[0038] Through the semantic segmentation technology of the deep learning network, the targets in different scenes in the mine site are classified by subject, and the semantic analysis of video images is realized. The semantic analysis of video images can help with many subsequent specific tasks of video images, such as the labeling of targets through semantic analysis, which promotes the construction of mine site image data sets; the classification of specific targets is realized through semantic analysis, such as the classification of workers and equipment and facilities in the mine, which provides preprocessing for the analysis of the working population in the mine, on-site safety management, and equipment and facility management.

[0039] Based on the above goals, we adopted a new convolutional neural network model (CNN). The convolutional neural network (CCN) model can adapt to image inputs of different sizes. To ensure that the image prediction result is the same size as the input image, the convolutional neural network uses a transposed convolution layer to gradually transform the intermediate layer feature map to the input image size. On the basis of ensuring the consistency of the spatial dimension, the output of the channel dimension represents the category prediction of the corresponding pixel at that position. The convolutional neural network model (CNN) has been used in scene segmentation tasks and has achieved good results, and can be used as the backbone network of the semantic segmentation model.

[0040] Through the convolutional neural network training algorithm of the present invention, any continuous nonlinear function can be approximated with arbitrary precision. The convolutional neural network structure used is as follows: Figure 4 As shown in Figure 1. It contains an input layer, a hidden layer, and an output layer. Neurons in the same layer are not connected to each other, while neurons between two adjacent layers are fully connected to each other. The neural network training algorithm consists of three stages: calculation stage, feedback stage, and iteration and termination stage.

[0041] In the convolutional neural network structure, the input layer has m input data, the hidden layer has q units, and the output layer has n output data.

[0042] In the forward phase, the input layer receives the input signal and passes it to each neuron in the hidden layer. The hidden layer then processes these signals and passes the results to the output layer. ,The input and output signals of each neuron in the hidden layer are marked as and , these two signals are calculated by formula (1) and formula (2) respectively.

[0043] Formula (1) Formula (2) is the weight between input layer neuron i and hidden layer neuron j, and θj is the bias.

[0044] After the output layer receives the signal from the hidden layer, it also needs to perform subsequent processing. and The output signal is calculated by formula (3) and formula (4) respectively.

[0045] Formula (3) Formula (4) in, is the weight between hidden layer neuron j and output layer neuron k, is bias.

[0046] At this point, the information processing flow of the forward process ends. In the forward process, the weights W, V and biases θ, γ of the neural network model do not change. If the final output signal of the neural network obtained by the forward processing is consistent with the true signal, then the next input vector will be input into the neural network and a new round of forward process will begin. Otherwise, the algorithm will enter the backward process. Here, the difference between the final output signal of the neural network and the true signal is called error.

[0047] In the backward process, we first use formula (5) to calculate the output layer neuron Then, we further use formula (6) to calculate the deviation of each hidden layer neuron deviation.

[0048] Formula (5) Formula (6) In the above formula, Y=(y1,y2,y3,…,yk)is the output vector of the neural network.

[0049] The bias is fed back from the output layer to the hidden layer. By sending this bias back propagation method, the connection weights between the output layer and the hidden layer are updated using formula (7).

[0050] Formula (7) In the above formula, α1 is the learning rate ranging from 0 to 1. N is the number of the current training round.

[0051] The backpropagation algorithm has two training modes: online training and batch training. For the online training mode, one sample is used in each round of training, and the training samples are processed one by one. For the batch training mode, a batch of samples are used in each round of training, and the ΔW generated by each sample in the same round (ΔW represents the deviation of W, V, K, θ, γ between two rounds) will be accumulated, and the accumulated ΔW will be used together to correct the model weights.

[0052] The entire training process will be carried out in iterations until the training termination condition is reached. The two commonly used termination conditions are that the mean square error of the model reaches the preset threshold and the number of training iterations reaches the maximum value set. In fact, in order to calculate the deviations generated by all samples, the entire training data set needs to be applied to the neural network to perform the above training process. When the scale of the training data set to be processed is large, traditional serial processing is time-consuming, so parallelization is needed to accelerate the processing.

[0053] According to one aspect of the present invention, a convolutional neural network model (CNN) includes the following operations: (1) Convolution operation: The input image is convolved. The feature extraction process uses a 3 × 3 convolution kernel and average pooling is used to downsample the feature map.

[0054] (2) Transposed convolution: The transposed convolution operation converts the high-dimensional feature map into a low-dimensional feature map through a fixed small-sized convolution kernel and matrix multiplication. For example, a 5 × 5 input feature, after a convolution kernel operation with a step size of 1 and a size of 3 × 3, the output feature map size is 3 × 3. Specifically, the convolution operation process can be written in the form of affine transformation. The convolution process can be written as:

[0055] Where C represents the convolution kernel parameter set, which is a sparse matrix, ωi is the value in the coefficient matrix, i=1,2,3...; x is the input data, z is the output data. However, in the transposed convolution process, the low-dimensional feature map is mapped from low-dimensional features to high-dimensional features through a similar principle. According to the affine transformation process of the above features, if the input size is 3 × 3 feature maps, the output features of 5 × 5 size are obtained through affine transformation. Then, .

[0056] This matrix operation is the inverse operation of the previous matrix operation. T represents the transpose of C. In this matrix operation, z is the input data and x becomes the output data.

[0057] (3) High-level feature and low-level feature layer fusion strategy. After continuous convolution and pooling, the input image obtains the corresponding pooling features, which are fused with the high-level feature map to obtain a feature map of the same size as the original, and soft classification is completed. The results show that after the high-level features are fused with the upsampling of the underlying features, they can better capture the details of the image during the semantic segmentation process.

[0058] Based on the convolutional neural network model, an adaptive density estimation model based on the crowd density distribution scenario of the station is constructed according to the characteristics of the crowd density distribution in the mine and the difficulties in video image analysis. The model uses the pre-trained model-visualized deep convolution model as the backbone basic network for feature extraction of the front-end input image. On this basis, two parallel branches are constructed to process the dense crowd scene and the sparse crowd scene respectively, and the corresponding weights are given. Then, the weighted linear combination of the first two branches is processed through the third branch to comprehensively calculate the perceived weight of the scene density, thereby improving the density statistical accuracy of the entire network. The attention mechanism and the hole convolution operation are used to focus on the target feature image, so as to quickly and accurately locate the crowd target in the station video image with complex and diverse backgrounds, and ensure the accuracy of the crowd density estimation.

[0059] After semantic segmentation of the video, the video is processed by multi-angle feature map projection fusion, which mainly includes two parts: multi-angle feature map extraction and multi-angle feature map projection fusion. When extracting the multi-angle feature map, an image feature extractor is designed to extract multi-scale features of multiple video images respectively; on the basis of obtaining image information of different sizes, the attention feature map is obtained through feature alignment, fusion and attention mechanism processing. When projecting and fusion of multi-angle feature maps, the attention feature map of multiple video images is first transformed from a two-dimensional plane feature map to a projection feature map in a three-dimensional plane coordinate system through spatial projection, and then the multi-channel projection feature maps are fused to obtain a projection fusion feature map and complete the video analysis estimation on this basis. Common multi-view image projection fusion processing methods can be divided into the following three types: ① No feature extraction, directly project the original image first and then fuse it; ② After feature extraction of the original image, the feature map is first projected and then fused; ③ After feature extraction of the original image, the feature map is first fused and then projected. After multi-view feature map extraction, the obtained feature map source (attention density feature map) is projected and fused. Select the appropriate processing method based on your needs.

[0060] The projection process of the feature map is to project the two-dimensional video image into the ground coordinate system of the three-dimensional space. Since the reference coordinate system in the visual projection changes, the internal and external parameters of each camera must be calibrated before the projection changes to determine the mapping relationship between the two-dimensional image coordinates and the three-dimensional ground coordinate system. The visual projection change process is based on the affine transformation between the three-dimensional coordinates (x, y, z) of the pixel point and the pixel coordinate point (u, v) of the two-dimensional image. The coordinate conversion in different dimensions is realized through the matrix augmentation operation, that is, 1 row of row vectors is added to both the two-dimensional coordinate vector and the three-dimensional coordinate vector, and all are filled with 1.

[0061]

[0062] Where: ε is a real number proportional factor; is a 3×4 perspective change parameter matrix; A is a 3×3 internal parameter matrix; R is a rotation transformation matrix parameter; t is a translation transformation matrix parameter; [R︱t] is a 3×4 rotation and translation transformation matrix (external parameter matrix).

[0063] In the process of projecting the density feature map of the surveillance video from the two-dimensional coordinate system to the three-dimensional ground coordinate system, in order to minimize the density estimation error caused by occlusion, consider setting the z value of all coordinate points in the coordinate system to 0, that is, changing all pixel projection coordinate points to (x, y, 0), and obtaining the projection feature map of the feature map. The specific change process is given by the matrix in the previous formula It is obtained by matrix multiplication with the space vector (x, y, 0, 1), that is

[0064] Where: It is the visual transformation matrix of size 3×3 that projects the two-dimensional image to the ground coordinate system.

[0065] The projection fusion process is relatively simple, and different projection feature maps can be obtained through matrix addition operations. Since some feature maps under two-dimensional coordinates will be deformed during feature map projection, a certain error loss will be caused when estimating the density map after feature map projection fusion. Therefore, the fused feature map is reduced by the dilated convolution of the large convolution kernel to make the final feature map as close to the labeled value as possible. At the same time, the large convolution kernel processing can also maximize the preservation of spatial information in the multi-view fusion process.

[0066] During actual model training, the sample image is first converted into a single-angle labeled density map according to the annotation file information in the training data set before it can participate in the model learning and training process. Then, after a series of operations such as feature extraction, attention density map generation, multi-angle feature map projection, and projection fusion, the final projection fusion density estimation map is generated.

[0067] According to one aspect of the present invention, event detection is performed based on the projection fusion density estimation result. The event detection method uses boundary and metric learning strategies to detect unknown types of events under open set training settings. First, a video clip containing the temporal information and image spatial features of the video itself is used as input, and the spatial features of each frame of the image are extracted through a convolutional neural network, and then the temporal information between multiple frames of images is further extracted to obtain the final spatiotemporal features of the video clip. Subsequently, boundary metric learning is performed based on the spatiotemporal features through a loss function to narrow the distance between members in the spatiotemporal feature distribution of normal video clips and the spatiotemporal feature distribution of abnormal video clips (intra-class distance), and expand the distance between members of the two distributions (inter-class distance). Finally, using the metric learning results, the distance between the spatiotemporal features of the video clip to be tested and the metric center features is calculated as the basis for event detection.

[0068] The event detection method further includes two steps: 1. Video event expression: extract descriptive and discriminative features to characterize and express the content of video events; 2. Normal video event modeling: based on feature expression, model the normal event samples in the training data, obtain the normal event model, and identify the events in the video to be tested that have a low match with the model as abnormal.

[0069] The targets in the continuous video frame sequence are processed simultaneously, the dynamic information between the same targets is mined, and the motion model of the target is established. Video target tracking is to predict the exact position of the target in the video image in the subsequent video sequence based on the rectangular box of the target of interest given in the first frame, and generate a rectangular bounding box.

[0070] Video object segmentation establishes the correspondence between the same objects across frames and achieves a more detailed representation of the target object (pixel-level mask instead of a rectangular bounding box). It divides the pixels into the foreground object and background area of ​​interest in the entire subsequent video sequence based on the target object mask given in the first frame, and generates a segmentation mask for the foreground object.

[0071] A tracker based on convolutional neural networks is used to extract the features of the target template and the search area through two twin structures containing the same convolutional layer. Then, the correlation function is used for feature matching, that is, to find similarities between the search area features and the target features to achieve target positioning.

[0072] In dynamically changing video sequences, since the appearance of the target changes as the target moves in space, it becomes very difficult to find the visual correspondence of the target across video frames without using labeled data. An effective approach is to establish visual correspondence based on cycle consistency. The deep features learned from the target image patches are used to achieve forward-backward tracking through matching between the template branch and the search branch. In order to maximize cycle consistency, the model is required to learn feature representations that support target correspondence across video frames. As these features are continuously improved and optimized, the tracking ability will gradually improve, making the whole process gradually tend to cycle consistency. The correspondence learned in such a feature space can establish a visual similarity measure, which can be used as an independent pixel similarity measure during model testing. However, the image patch matching tracking method does not distinguish the target of interest from the meaningless background or even occlusion, which will deteriorate the extracted features and hinder the robustness of tracking, thereby affecting the calculation of cycle consistency and similarity. To overcome this problem, a feature space enhancement module and a two-stream channel module are introduced to integrate the spatial and channel dimensions. Combining these two modules, the network can focus on exploring the discriminative target area and suppress irrelevant parts to improve the robustness of tracking. In addition, by jointly calculating the spatial and channel dimensional features of the search branch and the template branch, the interdependence of the two twin branches is enhanced, and the visual correspondence in consecutive frames is fully utilized, which improves the accuracy and generalization of the model.

[0073] Cycle consistency is used as a supervisory signal for learning feature representations, which is completely generated by the video frame itself by finding the region in the search image that is most similar to the template image patch. Since the appearance of the rectangular region is random, one problem is the lack of ability to reason about the possible objects of interest. Previous work only works for the ideal case that the template image specified in the first frame of the tracking subsequence only contains the entire object of interest. If the template image contains both foreground and background, it will provide misleading information. Another problem is that learning features from image pairs cannot effectively handle partial occlusion. For example, if the target is complete in the template image patch, or even larger than the selected template image patch, but is partially occluded in the search image, then forward-backward tracking often fails because it is difficult to find the target part corresponding to the template image patch in the search image.

[0074] During the model training process, it is expected that the target of interest always appears in the predicted template image patch over time. When the target is smaller than the template image patch, it is hoped that the target part is emphasized and the irrelevant background part is suppressed. For the case where the target exceeds or is equal to the template image patch, it is hoped that the most important area that can still be matched when the object is partially occluded can be found from the entire template image patch. It is worth noting that directly using the feature vector pair does not treat different parts differently. In order to adapt to the background changes during training and find the most important parts from them, it is necessary to update the template image patch features by adding additional occlusion-aware feature matching. Occlusion is created by cropping in the original template image using a sliding window method. This strategy aims to enhance the features of the object of interest and suppress the meaningless background to fine-tune the feature vector.

[0075] We exploit cycle consistency as a learning signal for self-training and use visual transformers to capture the relationship between unlabeled video frames. We adopt a cycle-consistent transformer structure to model self-supervised tracking as a cycle prediction problem. In order to learn richer and discriminative feature representations and make the feature representations of the same objects closer, we not only exploit inter-frame correspondences but also establish intra-frame correspondences to effectively establish the correspondence between objects and image frames. In addition, we further improve the performance of the system by specifically configuring the cycle-consistent transformer structure, including temporal sampling strategies, tracking initialization and data augmentation, and by leveraging a large amount of unlabeled data and a small amount of limited manually labeled data for training.

[0076] The cycle-consistent transformer structure consists of 2N−2 (inter-frame correspondence) or 2M−2 (intra-frame correspondence) transformer block stacks, all of which are connected end-to-end to mine spatial information in adjacent frames organized by inter-frame and intra-frame correspondences. In this mechanism, the representation and localization of the object are iteratively updated. Given an object in an initial frame, the cycle-consistent transformer tracks the moving object and adapts to its changing position through the video sequence carrying its identity information. The inherited object image block in the previous frame is input into the encoder of the current transformer block, transferring and enhancing the appearance information of the previously seen object, and the decoder can well localize the object in the current frame and output the position of the corresponding bounding box.

[0077] Video target tracking is to automatically estimate the position and range of the moving target in all subsequent frames in a video sequence after providing the target's location area (using a rectangular box) in the first frame, so as to achieve the positioning and scale estimation of the moving target. It plays an important role in various applications, including intelligent video surveillance, vehicle navigation, human-computer interaction, and activity analysis.

[0078] Video is an information-intensive medium with spatiotemporal continuity. In the self-supervised video moving object tracking scenario, it is necessary to maximize cycle consistency and effectively capture the intra-frame relationship and inter-frame long-distance relationship of the target to the frame. Given a set of video sequences V from a large-scale video benchmark, the model uses both inter-frame and intra-frame correspondences in a self-supervised manner to learn more discriminative feature embeddings. Two adjacent frames in the same video can be regarded as a natural enhancement of the same moving target in time. Self-supervised learning using inter-frame correspondences can learn the invariance of target features from viewpoint changes and appearance deformations over time. In addition to inter-frame correspondences, intra-frame correspondences construct enhancements of different views and transformations of the same target, and learn the invariance of feature enhancements with viewpoint changes. When tracking video targets, inter-frame correspondences and intra-frame correspondences are established simultaneously, and inter-frame correspondences and intra-frame correspondences are selected with equal probability to form tracking palindrome sequences as data samples.

[0079] Given a video sampling subsequence, a small square area is cropped out with the center of the label image as the frame center as the initial template image block, and the large square image block in the center of the remaining frame is used as the search image. The cropping is performed in this way because the target object located in the center is unlikely to move out of the cropped area in a short time, which can better provide good samples for target tracking. When cropping the square area, additional context margins are added around the bounding box to form a small square area containing the target and the background as the initial template image block. The area is enlarged at the initial template image block, and a set of corresponding search image blocks are obtained from the remaining frames. Further, if the position of the bounding box is close to the edge of the original image and the calculated search image block area exceeds the boundary of the original image, the search area will be moved in the opposite direction without changing the size of the search image area, and the part that still exceeds the original image area will be filled with pixels. In addition, the above template image block and search image block will be further scaled to adjust the size.

[0080] According to one aspect of the present invention, an image transformation enhancement operation is performed on a given target to simulate potential target motion changes in a tracking scene. The work first crops a given target object image block from an initial frame, then performs image transformation enhancement on the cropped image block, and finally pastes the transformed image block onto a new video frame where background interference may exist. This data enhancement method helps to synthesize data of diverse transformations in tracking scenes, but the image generated by pasting onto a new video frame is different from the real image, which violates the principle that the target changes continuously over time. On the basis of image transformation enhancement, spatial enhancement (e.g., horizontal flipping, scaling, and position jittering) and color enhancement (e.g., grayscale, blurring, and color jittering) can be used to enhance the images in the sampling subsequence after the tracking initialization phase with a probability p. These enhancements will simulate target scale changes, deformations, motion, and illumination changes, etc., and enrich the scene of moving target tracking during the tracking process. In addition, when obtaining the search image block according to the center (Xcenter, Ycenter) of the initial template image block (ie, the starting frame It), in order to avoid the position of the predicted target in the search image block always being a fixed area, Gaussian noise can be added to the center Tcenter of the template image block to perform a position jitter operation.

[0081] Given a video sequence consisting of multiple image frames as input, only the predicted bounding box of the last frame is output during the training process. By cleverly avoiding tracking errors on intermediate frames, the cycle-consistent correspondence of the tracked target is achieved. In the first stage, given a video frame of a tracking palindrome sequence, the standard CNN module extracts the features of a single template image patch and the search region, and then concatenates the template features and multiple search features in frame order to form a feature vector. In the second stage, these feature vectors are propagated spatiotemporally through a series of weight-sharing transformer blocks. Specifically, the frame vector is input to the transformer encoder to establish the spatial dependency between the template target and the current search region. Then, the tracking query and the output of the above encoder are input to the transformer decoder, where the tracking query from the previous transformer summarizes the target information from the past frames to highlight the target in the current frame. The tracking query follows the moving target, retains its identity information and adapts to its changing position, shape and appearance. These tracking queries that retain context-rich target clues are used to predict the bounding box of the last frame. In order to better convey the information between the previous frame and the current frame, the relationship between the targets in the adjacent frames of the context must be made clear. Making full use of the characteristics of cycle consistency is not only crucial for cross-frame target association, but also beneficial to the overall accuracy of moving target tracking.

[0082] Take multiple sets of features as input to the monitoring system, compare different features, and obtain monitoring results as output, comparing the monitoring results of different information feature inputs.

[0083] Specifically, a sliding window is first used to slide between consecutive frames of the video, and the current video frame in the sliding window and the video frame previously scanned by the window are assumed to be abnormal and normal classes respectively, which are used to train a logical classifier. If the training accuracy is very high, it proves that the current video frame and the previous video frame in the sliding window can be easily distinguished, indicating that there is a big difference between the current video frame and the previous video frame, and the degree of abnormality is high. Normal events and abnormal events are relative and abstract concepts, and do not have clear semantic information of other absolute and specific concepts. In some cases, their definitions will change with time and environmental changes. An event identified as abnormal in one scene may be normal in another scene. This makes it very difficult to define normal samples and abnormal samples before training, and it needs to be adjusted dynamically according to the actual application scenario.

[0084] According to one aspect of the present invention, multiple groups of facial features of personnel can be used as inputs of a personnel fatigue status monitoring model, the monitoring results of the personnel fatigue status monitoring model with different information feature inputs can be compared, the personnel fatigue status can be used as output, and the performance of the personnel fatigue status monitoring method can be verified. Considering that there may be nonlinear correlations between fatigue features, the nonlinear correlations between fatigue features can be calculated by exponential curve fitting, logarithmic curve fitting, polynomial curve fitting, and power function fitting.

[0085] First, extract the facial features of the person in each frame of the video to be tested to describe their behavior, and construct a facial feature map to represent the extracted features, and decompose it into two feature components: global overall features and eye feature changes; then, use the global and eye features as inputs, and use an anomaly detector to classify all facial feature maps to obtain an initial normal face set and an initial abnormal face set; finally, use the self-training regression strategy based on the initial set as training data to iteratively update an anomaly score module. After the iteration, use the trained module to recalculate all facial anomaly scores for fine detection of abnormal events related to people.

[0086] According to another aspect of the present invention, multiple groups of human skeleton features can be used as inputs to a human status monitoring model, and the skeleton feature information of different people can be compared. The status of whether the person has fallen can be used as output, and the performance of the human fall status monitoring method can be verified.

[0087] The skeleton trajectory in the normal video sequence can be extracted to describe the human behavior, and it can be decomposed into global and local feature components to respectively characterize the overall movement of the human body and the local posture change. Then, the global and local components are jointly modeled to establish a human behavior detection model. Finally, given the video sequence to be tested, after extracting the skeleton trajectory of all individuals, each trajectory is reconstructed and predicted simultaneously using the trained human behavior detection model, and the reconstruction error and prediction error are fused for abnormal trajectory detection, thereby realizing the recognition of abnormal events of people falling.

[0088] First, the skeleton features of pedestrians in each frame of the video to be tested are extracted to describe their behavior, and a spatial skeleton graph is constructed to represent the extracted features. At the same time, it is decomposed into two feature components: global overall motion and local posture change. Then, with global and local features as input, an anomaly detector is used to classify all skeleton graphs to obtain an initial normal skeleton set and an initial abnormal skeleton set. Finally, based on the initial set as training data, a self-training regression strategy is used to iteratively update an anomaly score module. After the iteration, the trained module is used to recalculate the anomaly scores of all skeleton individuals for fine detection of abnormal events related to people.

[0089] According to one aspect of the present invention, a video image is input into a deep learning neural network algorithm to obtain a key point heat map; the annotated key points are calculated in a downsampling manner to obtain the key points of the training image.

[0090] Specifically, when training the key point prediction network, the video image is annotated to obtain the annotated key points, the position of the annotated key points is P, and the annotated key points are calculated in a downsampling manner to obtain the key points of the training image. Here, the annotated key points GT on the video image are downsampled (low resolution processing) to obtain a high-definition training image.

[0091] In the training image, the key points of the training image are distributed to the key point heat map by a Gaussian filtering algorithm in a manner of downsampling the marked key points; specifically, in the training image, the key points of the training image are distributed to the heat map by a Gaussian filtering algorithm in a manner of downsampling the marked key points. The key points of the training image and the predicted key points are corrected by a loss function to obtain a first corrected difference; specifically, the key points of the training image and the predicted key points are corrected by a loss function to obtain a first corrected difference. The second preset condition is: set the number of training times, if the first corrected difference tends to be stable within the number of training times, the correction of the key points and the predicted key points of the training image is completed.

[0092] The initial bias value is set, and the initial bias value is trained through the loss function to obtain the bias value; when the bias value reaches the first preset condition, the predicted key point is corrected by the bias value to obtain the corrected predicted key point; specifically, since the video image is down-sampled, there will be a certain error, so the bias value is set, and the predicted key point is compensated by the bias value. The initial bias value is set, and the initial bias value is trained through the loss function to obtain the bias value. The first preset condition is: set the number of corrections, when the bias value tends to be stable within the number of corrections, it means that the bias value meets the accuracy requirement, at this time, the predicted key point is corrected by the bias value to obtain the corrected predicted key point. When the first correction difference meets the second preset condition, the correction of the key points of the training image and the predicted key points is completed, and the human posture detection model is obtained according to the corrected predicted key points.

[0093] A portable explosion-proof pan / tilt camera according to the present invention, such as Figure 1 As shown. It includes a base, a mounting box is rotatably connected above the base, a connecting pipe is installed at one end of the mounting box, a rotating shaft 1 is rotatably connected inside the connecting pipe, a shell is installed at one end of the rotating shaft 1 away from the mounting box, a camera body is installed inside the shell, the mounting box and the shell are both made of stainless steel or explosion-proof material, so that they have explosion-proof and corrosion-resistant properties, a lens is installed at the front end of the shell in front of the camera body, a wiping mechanism is installed on the surface of the lens, and a pickup is installed inside the shell below the camera body; The camera body is used to continuously collect video data, form a real-time video data stream and send the real-time video data stream to the storage and communication module; A communication module is used to send the real-time video data stream collected by the camera body to the server in the monitoring center; The video monitoring module is used to receive the real-time video data stream collected by the camera body, process the data stream, and then send the processing result to the monitoring center through the communication module; The video monitoring module is also used to achieve: annotating the real-time video images collected by the portable explosion-proof pan-tilt camera in real time to obtain the annotated key points; performing video target segmentation on the annotated video images, establishing the correspondence between the same targets across frames, and achieving a more detailed representation of the target object. It divides the pixels into the foreground target of interest and the background area in the subsequent entire video sequence according to the target object mask given in the first frame, and generates a segmentation mask for the foreground target; performing video target tracking on the video images after video target segmentation, predicting the exact position of the target in the video image in the subsequent video sequence according to the rectangular box of the target of interest given in the first frame, and generating a rectangular bounding box; establishing a visual correspondence based on the cycle consistency of the video images after video target tracking, and establishing a feature representation of the correspondence between targets across video frames; using a sliding window to slide between consecutive video frames, and comparing the current video frame in the sliding window with the video frame previously scanned by the window, outputting a normal or abnormal result according to the comparison result, and issuing an alarm message when an abnormal result is output.

[0094] According to one aspect of the present invention, the communication module may be a 5G wireless communication module for communicating with other explosion-proof PTZ cameras or monitoring centers through a 5G network. The communication module may also be a 6G wireless communication module, a 4G wireless communication module, a wifi wireless communication module, a near field communication module, a Bluetooth communication module, or a wired communication module.

[0095] The portable explosion-proof pan / tilt camera of the present invention comprises a base, an installation box 2 is rotatably connected above the base, the installation box can be rotated 360 degrees to realize all-round shooting of the camera, a connecting pipe is installed at one end of the installation box, a rotating shaft 1 is rotatably connected inside the connecting pipe, a shell is installed at one end of the rotating shaft 1 away from the installation box, the installation box and the shell are made of stainless steel, have explosion-proof and corrosion-resistant characteristics, strong applicability, and are suitable for use in various flammable and explosive environments, the shell is used to install the camera body, the camera body is installed inside the shell, the camera body is used to shoot the picture, and The captured images are transmitted to the PTZ for real-time viewing at a long distance. A lens is installed at the front end of the shell in front of the camera body. The lens is made of special explosion-proof glass, the surface of which is processed by nanotechnology and coated with anti-reflection film on both sides. The transmittance in a specific band is greater than 99%. A wiping mechanism is installed on the surface of the lens to wipe the surface of the lens to prevent dust from being adsorbed on the surface of the lens 7. A microphone is installed inside the shell below the camera body to collect the audio at the monitoring scene and transmit the audio. The audio signal at the monitoring scene can be understood, effectively improving the effectiveness of the video image.

[0096] Specifically, the wiping mechanism includes a wiper motor, a brush rod and a brush strip. The wiper motor is installed inside the shell above the camera body. The output end of the wiper motor passes through the front end of the shell, and the brush rod is installed at the output end of the wiper motor. The brush strip is detachably installed at the rear end of the brush rod. The brush strip is in contact with the surface of the lens. The wiper motor rotates and drives the brush rod to swing. The brush strip at the rear end of the brush rod wipes the surface of the lens, so as to wipe the dust on the surface of the lens and ensure the clarity of the captured picture.

[0097] Furthermore, clamping plates are installed on the upper and lower surfaces of the rear end of the brush rod, and the brush strip is clamped inside the two clamping plates. The brush strip is installed inside the two clamping plates in a clamping manner, which makes it convenient to replace the brush strip after using it for a period of time.

[0098] Furthermore, a connecting block is installed on the upper end of the base through a through hole, a connecting shaft is installed on the upper end of the connecting block, the upper end of the connecting shaft passes through the installation box, and the connecting shaft and the installation box are rotatably connected, a turbine 1 is installed on the upper end of the connecting shaft, and fixed frames are installed at the front and rear parts of the bottom of the installation box on one side of the turbine 1, and a rotating shaft 2 is rotatably installed between the fixed frames on one side of the turbine 1, and a worm 1 is installed on the surface of the rotating shaft 2, and the turbine 1 and the worm 1 are meshed. By rotating the rotating shaft 2, the rotating shaft 2 drives the worm 1 to rotate, and because the worm 1 and the turbine 1 are meshed, the worm 1 rotates along the turbine 1, and at the same time drives the installation box to rotate, thereby realizing 360° all-round shooting of the camera body and a wide shooting range.

[0099] Furthermore, the front end of the second rotating shaft passes through the fixed frame and is equipped with a pulley one. A motor one is installed between the fixed frames above the second rotating shaft. The output end of the motor one passes through the fixed frame and is equipped with a pulley two. A belt strip is installed between the pulley two and the pulley one. The motor one drives the pulley two to rotate, and the pulley two drives the pulley one to rotate through the belt strip. The pulley one drives the second rotating shaft to rotate, providing power for the rotation of the second rotating shaft. The motor one is connected to a forward and reverse circuit and can rotate forward and reverse.

[0100] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0104] In certain embodiments, the control module includes one or more nodes or devices each having an integrated user interface, and a user or operator can interact with a process control system via the integrated user interface to perform related activities (e.g., configuration, viewing, monitoring, testing, diagnosis, sorting, planning, scheduling, annotation, and / or other activities). Examples of these user interface nodes or devices include mobile or stationary computing devices, workstations, handheld devices, tablet devices, surface computing devices, and any other computing devices with processors, memory, and integrated user interfaces. The integrated user interface can include a screen, keyboard, keypad, mouse, button, touch screen, touch template, biometric interface, speaker and microphone, camera, and / or any other user interface technology. Each user interface device can include one or more integrated user interfaces. A user interface node or device can include a direct connection to a cloud computer network system of distributed big data real-time exchange sharing, or can include an indirect connection to a cloud computer network system of distributed big data real-time exchange sharing, for example, via an access point or gateway. The user interface device can be connected to a cloud computer network system of distributed big data real-time exchange sharing in a wired manner and / or in a wireless manner. In some embodiments, the user interface device can be connected to a distributed big data real-time exchange and sharing cloud computer network system in various communication modes.

[0105] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A monitoring method based on video images collected by a portable explosion-proof PTZ camera, comprising the following steps: Step S100, the portable explosion-proof PTZ cameras deployed at various locations collect real-time video images from various directions in real time; Step S200, annotating the video image to obtain annotated key points; Step S300, performing video object segmentation on the annotated video image, establishing a correspondence between the same objects across frames, achieving a more detailed representation of the target object, dividing the pixels into the foreground object of interest and the background area in the subsequent entire video sequence according to the target object mask given in the first frame, and generating a segmentation mask of the foreground object; Step S400, performing video target tracking on the video image after the video target segmentation, predicting the accurate position of the target in the video image in the subsequent video sequence according to the rectangular frame of the target of interest given in the first frame, and generating a rectangular bounding box; Step S500, for the video image after video target tracking, a visual correspondence is established based on cycle consistency, and a feature representation of the target correspondence across video frames is established; Step S600, slide a sliding window between consecutive video frames, and compare the current video frame in the sliding window with the video frame previously scanned by the window, output a normal or abnormal result based on the comparison result, and issue an alarm message when an abnormal result is output.

2. The monitoring method according to claim 1, characterized in that: Video images are annotated by performing refined inference prediction on the label classification of each pixel of the input image, where each pixel is annotated according to the category of the target entity it surrounds.

3. The monitoring method according to claim 1, characterized in that: When segmenting video objects, the objects in different scenes are classified by subject to achieve semantic analysis of video images.

4. The monitoring method according to claim 3, characterized in that: Convolutional neural network (CNN) is used to classify the objects in different scenarios as the backbone network of the semantic segmentation model.

5. The monitoring method according to claim 4, characterized in that: The convolutional neural network uses a transposed convolution layer to gradually transform the feature map of the intermediate layer into the input image size, ensuring the consistency of the spatial dimension, and represents the output of the channel dimension as the category prediction of the pixel corresponding to the corresponding position.

6. The monitoring method according to claim 1, characterized in that: When establishing feature representation of cross-video frame target correspondence, the features of the video image are projected from two-dimensional coordinates to a three-dimensional ground coordinate system for feature representation.

7. The monitoring method according to claim 1, characterized in that: When establishing feature representations of target correspondences across video frames, inter-frame correspondences and intra-frame correspondences are established respectively, effectively establishing correspondences from targets to image frames.

8. A portable explosion-proof pan / tilt camera, comprising a base, a mounting box rotatably connected to the top of the base, a connecting pipe installed at one end of the mounting box, a rotating shaft 1 rotatably connected inside the connecting pipe, a shell installed at one end of the rotating shaft 1 away from the mounting box, a camera body installed inside the shell, the mounting box and the shell are both made of stainless steel or explosion-proof material, so that they have explosion-proof and corrosion-resistant properties, a lens is installed at the front end of the shell in front of the camera body, a wiping mechanism is installed on the surface of the lens, and a microphone is installed inside the shell below the camera body; The camera body is used to continuously collect video data, form a real-time video data stream and send the real-time video data stream to the storage and communication module; Communication module, used to send the real-time video data stream collected by the camera body to the server in the monitoring center, as well as other convenient explosion-proof PTZ cameras; The video monitoring module is used to receive the real-time video data stream collected by the camera body, process the data stream, and then send the processing result to the monitoring center through the communication module; The video monitoring module is also used to implement: Annotate the video images collected in real time by the portable explosion-proof PTZ camera to obtain the annotated key points; Perform video object segmentation on the annotated video images, establish the correspondence between the same objects across frames, achieve more detailed representation of the target objects, divide the pixels into the foreground objects of interest and the background areas in the subsequent entire video sequence according to the target object mask given in the first frame, and generate the segmentation mask of the foreground object; Perform video target tracking on the video image after video target segmentation, predict the accurate position of the target in the video image in the subsequent video sequence according to the rectangular frame of the target of interest given in the first frame, and generate a rectangular bounding box; For video images that have been tracked, visual correspondence is established based on cycle consistency, and feature representation of cross-video frame target correspondence is established; A sliding window is used to slide between consecutive video frames, and the current video frame in the sliding window is compared with the video frame previously scanned by the window. According to the comparison result, a normal or abnormal result is output, and an alarm message is issued when an abnormal result is output.

9. A portable explosion-proof PTZ camera according to claim 8, characterized in that: The communication module is a 5G wireless communication module, which is used to communicate with other explosion-proof PTZ cameras or monitoring centers through the 5G network.

10. A monitoring system based on a portable explosion-proof PTZ camera, the monitoring system comprising: A plurality of portable explosion-proof PTZ cameras as claimed in claim 8 arranged in various positions; A server, connected to the plurality of portable explosion-proof PTZ cameras to transmit video data streams in real time; A storage array, connected to the server, for storing the video data stream received by the server; The video analysis and processing module is connected to the server and the storage array respectively, and is used to analyze and process the received video data stream, so as to realize the analysis and processing of the video data stream collected by all the portable explosion-proof PTZ cameras; The server is also used to receive the video detection results sent by the portable explosion-proof PTZ camera and respond to the alarm information.

Images