Deep learning supported campus monitoring video anomaly detection method and system

Abstract

The invention relates to a deep learning supported campus monitoring video anomaly detection method and system, and relates to the field of campus security monitoring, and the method comprises the steps: receiving a personnel positioning point cloud and video information, and obtaining a monitored crowd positioning point cloud through deep learning analysis; monitoring objects are divided into first-class objects coinciding with the intra-school positioning point cloud and second-class objects not coinciding with the intra-school positioning point cloud. And performing occupational recognition on the second-class objects. And when the first-class object enters the forbidden zone and the second-class object does not move according to the preset path or the occupational identifier is unknown, performing abnormal identification. The technical problem that the intelligent level and the safety guarantee capability of campus monitoring are insufficient due to the fact that existing campus monitoring cannot perform differential monitoring and accurate abnormal behavior recognition on different monitoring objects is solved, and accurate positioning, classification and abnormal identification are realized on the on-campus personnel and the foreign personnel based on deep learning. And the intelligent level and the safety guarantee capability of campus monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the field of campus security monitoring, and in particular to a method and system for detecting anomalies in campus monitoring videos supported by deep learning. Background Art

[0002] As campus security issues are increasingly valued, campus monitoring has become an indispensable means of maintaining campus security. However, most of the existing campus monitoring adopts traditional video monitoring methods, which has the following shortcomings: First, the existing campus monitoring usually treats all monitoring objects equally, and cannot differentiate and monitor according to different identities, which makes it difficult to accurately distinguish between people in the school and outsiders, and cannot identify suspicious behaviors and potential threats in a targeted manner; secondly, traditional campus monitoring mainly relies on manual observation and judgment, lacks intelligent abnormal behavior recognition capabilities, and when abnormal situations appear in the monitoring screen, it is difficult to discover and warn in real time, which poses a safety hazard; thirdly, the existing campus monitoring has limited precise positioning and behavior analysis capabilities in complex campus environments, and the tracking and identification accuracy of monitoring objects is not high, which is prone to missed reports and false reports. Therefore, how to achieve differentiated monitoring and accurate behavior analysis of different monitoring objects in campus scenes, improve the intelligence level of campus monitoring, and then comprehensively strengthen the campus security protection capabilities, is an urgent problem to be solved. Summary of the invention

[0003] The present invention aims to solve the technical problem that existing campus monitoring is unable to perform differentiated monitoring and accurate abnormal behavior identification on different monitored objects, resulting in insufficient intelligence level and security assurance capability of campus monitoring, and provides a campus monitoring video anomaly detection method and system supported by deep learning.

[0004] The technical solution of the present invention to solve the above technical problems is as follows: In a first aspect, the present invention provides a campus surveillance video anomaly detection method supported by deep learning, including: communicating with an on-campus personnel positioning terminal and receiving an on-campus personnel positioning point cloud; communicating with a surveillance camera array and receiving surveillance video information, performing deep learning fusion analysis on the surveillance video information to obtain a monitoring crowd positioning point cloud, wherein any point of the on-campus personnel positioning point cloud and the monitoring crowd positioning point cloud represent a monitoring object; extracting the monitoring objects that overlap with the on-campus personnel positioning point cloud in the monitoring crowd positioning point cloud and setting them as a type of monitoring object; extracting the monitoring objects that do not overlap with the on-campus personnel positioning point cloud in the monitoring crowd positioning point cloud The monitoring objects are set as Class II monitoring objects; the wearing images and the images of the tools carried by the Class II monitoring objects are collected, and occupational identification is performed based on the wearing images and the images of the tools carried to obtain occupational identification results, wherein the occupational identification results include known occupational identifications and unknown occupational identifications; when the Class I monitoring objects enter the restricted movement zone, the Class I monitoring objects are marked as abnormal; when the Class II monitoring objects have the known occupational identification and do not move according to the preset identification path, the Class II monitoring objects are marked as abnormal; when the Class II monitoring objects have the unknown occupational identification, the Class II monitoring objects are marked as abnormal.

[0005] In the second aspect, the present invention provides a campus surveillance video anomaly detection system supported by deep learning, including: a positioning data receiving module, used to communicate with the campus personnel positioning terminal and receive the campus personnel positioning point cloud; a video fusion analysis module, used to communicate with the monitoring camera array and receive monitoring video information, perform deep learning fusion analysis on the monitoring video information, and obtain a monitoring crowd positioning point cloud, wherein the campus personnel positioning point cloud and any one point of the monitoring crowd positioning point cloud represent a monitoring object; a campus personnel screening module, used to extract the monitoring objects that overlap with the campus personnel positioning point cloud in the monitoring crowd positioning point cloud, and set them as a type of monitoring objects; an outsider screening module, used to extract the monitoring objects that overlap with the campus personnel positioning point cloud in the monitoring crowd positioning point cloud The monitoring objects whose positioning point clouds of the on-campus personnel do not overlap are set as Class II monitoring objects; an occupational identification analysis module is used to collect the wearing images and the carrying tool images of the Class II monitoring objects, perform occupational identification based on the wearing images and the carrying tool images, and obtain occupational identification results, wherein the occupational identification results include known occupational identifications and unknown occupational identifications; an abnormal identification module is used to mark the Class I monitoring objects as abnormal when the Class I monitoring objects enter the restricted movement zone, mark the Class II monitoring objects as abnormal when the Class II monitoring objects have the known occupational identification and do not move according to the preset identification path, and mark the Class II monitoring objects as abnormal when the Class II monitoring objects have the unknown occupational identification.

[0006] In a third aspect, the present application provides an electronic device comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is used to execute the campus surveillance video anomaly detection method supported by deep learning provided in the present application.

[0007] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which is used to execute the campus surveillance video anomaly detection method supported by deep learning provided in the present application.

[0008] The beneficial effects of the present invention are: It communicates with the on-campus personnel positioning terminal and receives the on-campus personnel positioning point cloud to obtain the precise location information of the on-campus personnel; it communicates with the surveillance camera array and receives the surveillance video information, performs deep learning fusion analysis on the surveillance video information, and obtains the monitoring crowd positioning point cloud, wherein any point of the on-campus personnel positioning point cloud and the monitoring crowd positioning point cloud represents a monitoring object, and the monitoring video is processed through deep learning to obtain the location point cloud of all monitoring objects, laying the foundation for subsequent classification; the monitoring objects that overlap with the on-campus personnel positioning point cloud in the monitoring crowd positioning point cloud are extracted and set as a type of monitoring object, and the on-campus personnel are identified through point cloud matching and classified as a type of monitoring object to distinguish between people inside and outside the school; the monitoring crowd positioning point cloud is extracted, The monitoring objects that do not overlap with the location point cloud of the personnel on campus are set as Class II monitoring objects; the images of the clothes and tools carried by the Class II monitoring objects are collected, and occupational identification is performed based on the images of the clothes and tools carried to obtain occupational identification results, where the occupational identification results include known occupational identification and unknown occupational identification. By identifying the clothes and objects carried by the Class II monitoring objects, it is determined whether they are known occupational personnel or unknown suspicious personnel; when the Class I monitoring objects enter the restricted area, the Class I monitoring objects are marked as abnormal; when the Class II monitoring objects have known occupational identification and do not move according to the preset identification path, the Class II monitoring objects are marked as abnormal; when the Class II monitoring objects have unknown occupational identification, the Class II monitoring objects are marked as abnormal. According to the type, location trajectory and movement path of different monitoring objects, abnormal behavior is identified and marked. Through the differentiated monitoring system for campus scenes, the location, movement and behavior of different types of personnel are accurately analyzed, and timely warnings can be issued once abnormal identification appears, thereby significantly improving the intelligence level and security assurance capabilities of campus monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of the process of the campus surveillance video anomaly detection method supported by deep learning provided by the present invention; Figure 2 A schematic diagram of the structure of the campus surveillance video anomaly detection system supported by deep learning provided by the present invention; Figure 3A schematic diagram of the structure of an electronic device provided by the present invention; Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention.

[0010] In the accompanying drawings, the components represented by the reference numerals are as follows: Positioning data receiving module 11, video fusion analysis module 12, on-campus personnel screening module 13, outsider screening module 14, occupational identification analysis module 15, abnormal identification module 16, electronic device 200, memory 210, processor 220, first computer program 211, computer readable storage medium 300, second computer program 311. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0012] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0013] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0014] Embodiment 1: like Figure 1As shown, an embodiment of the present invention provides a campus surveillance video anomaly detection method supported by deep learning, which is applied to a campus surveillance video anomaly detection system supported by deep learning, which is communicatively connected to an on-campus personnel positioning terminal and a surveillance camera array.

[0015] Specifically, the provided campus surveillance video anomaly detection method supported by deep learning is applied to the campus surveillance video anomaly detection system supported by deep learning. The system establishes a communication connection with the on-campus personnel positioning terminal and the surveillance camera array. Among them, the on-campus personnel positioning terminal can be a device with positioning function carried by the on-campus personnel, such as a mobile phone, a tablet computer, a smart watch, etc. The surveillance camera array consists of multiple cameras arranged in different areas of the campus to monitor the campus scene in all directions. By communicating with the on-campus personnel positioning terminal, the real-time location information of the on-campus personnel can be obtained; by communicating with the surveillance camera array, the real-time surveillance video data of various areas of the campus can be obtained. This lays a data foundation for the subsequent fusion analysis of on-campus personnel and surveillance videos to achieve accurate anomaly identification.

[0016] Campus surveillance video anomaly detection methods include: S100: Communicate with the on-campus personnel positioning terminal and receive the on-campus personnel positioning point cloud.

[0017] Specifically, by establishing a communication connection with the on-campus personnel positioning terminal, the on-campus personnel positioning point cloud uploaded by the on-campus personnel positioning terminal is received. The on-campus personnel positioning terminal can be a device with positioning function carried by on-campus personnel, such as a smart phone, tablet computer, smart watch, etc. The on-campus personnel positioning terminal can obtain the wearer's spatial position coordinates in real time through various positioning technologies such as GPS, Wi-Fi, Bluetooth, etc., and package it into a point cloud data format for uploading to obtain the on-campus personnel positioning point cloud.

[0018] Point cloud data is a spatial data format composed of discrete three-dimensional coordinate points. Each point contains the X, Y, and Z coordinate information in its three-dimensional space. The position coordinates of all people on campus at a certain moment are aggregated to form the location point cloud of people on campus at that moment. The location point cloud of people on campus intuitively reflects the real-time distribution of people on campus in the space of the campus, and provides the necessary data basis for the subsequent comparison and fusion with surveillance video information.

[0019] By continuously receiving point cloud data uploaded by the positioning terminal, the spatial distribution information of people on campus can be grasped in real time and dynamically, which is the basis for distinguishing people inside and outside the campus, realizing differentiated monitoring and abnormal behavior analysis. By accurately grasping the location distribution of people on campus and comparing it with the personnel information in the surveillance video, it is possible to identify outsiders and suspicious behaviors, thus realizing accurate and intelligent campus safety management.

[0020] S200: Communicate with the surveillance camera array and receive surveillance video information, perform deep learning fusion analysis on the surveillance video information, and obtain a surveillance crowd positioning point cloud, wherein any point of the on-campus personnel positioning point cloud and the surveillance crowd positioning point cloud represents a surveillance object.

[0021] Specifically, by establishing a communication connection with the surveillance camera arrays arranged in various areas of the campus, the surveillance video information collected by the surveillance camera array in real time is received. The surveillance video information refers to the continuous image sequence captured by the camera, which records the real-time dynamic images of the campus scene.

[0022] After receiving the surveillance video information, the video analysis model built based on deep learning technology is used to perform intelligent fusion analysis and processing on the surveillance video information. Through training on a large number of campus surveillance video samples, the video analysis model can automatically extract the appearance features, motion features, etc. of people from the video screen, and realize the understanding and semantic analysis of the video content. Among them, during the fusion analysis and processing, first, the target detection algorithm is used to identify the position and bounding box of the human target in the video frame image of the surveillance video information; then, multi-target tracking is used to associate the detection results of the same person in different frame images according to the consistency of the appearance and motion of the human target between consecutive frames to form a motion trajectory; then, the trajectory of the person under the perspective of multiple cameras at the same time is spatially fused to generate a unified point cloud of the person's position in the campus coordinate system, that is, the monitoring crowd positioning point cloud.

[0023] The monitoring crowd positioning point cloud and the campus personnel positioning point cloud use the same data format, and each point represents the spatial position of a human target detected in a monitoring screen in the campus coordinate system. By comparing and analyzing the two types of point clouds, it is possible to distinguish and identify campus personnel and outsiders.

[0024] Through intelligent video analysis, the location distribution of all people appearing in the surveillance video is obtained, achieving comprehensive and real-time monitoring of campus personnel.

[0025] S300: extracting the monitoring objects that overlap with the positioning point cloud of the on-campus personnel from the positioning point cloud of the monitored crowd and setting them as a type of monitoring objects.

[0026] Specifically, through deep learning fusion analysis, a monitoring crowd positioning point cloud containing the location information of people in all monitoring images is obtained. However, the monitoring crowd includes both on-campus personnel and outsiders. In order to treat and manage the two types of personnel differently, it is necessary to accurately identify on-campus personnel from the monitoring crowd.

[0027] The spatial matching of point cloud data is used to realize the positioning and identification of people in the school. First, the spatial coordinates of the obtained positioning point cloud of people in the school are compared with the generated positioning point cloud of the monitored crowd to find the coincidence points of the two types of point clouds at the same time. Since the positioning point cloud of people in the school is directly collected and uploaded by the positioning terminal of people in the school, it can definitely represent the position of people in the school. If the coordinates of a point in the positioning point cloud of the monitored crowd are consistent with those of a point in the point cloud of people in the school or the distance is within an acceptable error range, it can be considered that the point of the monitored person represents a person in the school.

[0028] Through spatial coordinate matching, all personnel points that appear in both the monitoring crowd positioning point cloud and the on-campus personnel positioning point cloud are extracted and marked as a type of monitoring object, representing the on-campus personnel that appear in the monitoring video. The location information of the first type of monitoring object will be used for subsequent abnormal behavior analysis to achieve accurate monitoring and management of on-campus personnel.

[0029] By comparing the positioning point cloud of people on campus and the positioning point cloud of monitored people, people on campus can be quickly and accurately identified from surveillance videos, providing reliable data support for achieving differentiated campus safety management.

[0030] S400: extracting monitoring objects from the monitoring crowd positioning point cloud that do not overlap with the on-campus personnel positioning point cloud, and setting them as second-category monitoring objects.

[0031] Specifically, in S300, the on-campus personnel in the surveillance video are identified through spatial matching of the on-campus personnel positioning point cloud and the monitored crowd positioning point cloud, and are marked as a first-class monitoring object. In contrast, the remaining points in the monitored crowd point cloud that do not overlap with the on-campus personnel point cloud represent outsiders appearing in the surveillance video. These non-on-campus personnel points are extracted from the monitored crowd positioning point cloud and marked as a second-class monitoring object, representing potential outsiders.

[0032] The identification and positioning of the second-class monitoring objects lays the foundation for intelligent campus security management. Based on the second-class monitoring objects, the behavior patterns and movement trajectories of outsiders can be further analyzed to timely detect suspicious abnormal behaviors. In addition, the identification results of outsiders can also be used for differentiated management from on-campus personnel, realizing classified monitoring and services for on-campus and off-campus personnel.

[0033] By quickly locating the monitoring objects that do not overlap with the location point cloud of the people on campus from the surveillance video, they are set as Class II monitoring objects, laying a data foundation for subsequent analysis and anomaly detection of outsiders.

[0034] S500: Collect clothing images and images of tools carried by the two types of monitored objects, perform occupational identification based on the clothing images and the images of tools carried, and obtain occupational identification results, wherein the occupational identification results include known occupational identifications and unknown occupational identifications.

[0035] Specifically, after identifying the second-category monitoring objects, their occupational identities are identified and judged in order to achieve more refined personnel classification management.

[0036] In campus management, outsiders entering the campus may have different identities and occupations, such as couriers, takeaway deliverymen, decoration workers, visitors, etc. For outsiders of different occupations, their activity areas, behavior patterns and other characteristics are different, so different management and monitoring strategies are required. By identifying the occupations of the second type of monitoring objects, the occupational identification results are obtained, which provides a basis for differentiated management of outsiders.

[0037] First, by tracking the movement trajectory of the two types of monitored objects, clear human images are captured at different monitoring angles. Using image processing technology, the human body area is segmented from the captured image to obtain the image of the clothes and the image of the tools carried, and the human body's clothing features and the features of the items carried are further extracted. Among them, the clothing features include the style, color, pattern, etc. of the clothes, and the features of the items carried include backpacks, tool boxes, express boxes, etc. Then, the extracted clothing features and the features of the items carried are input into the pre-trained occupational recognition model. Among them, the occupational recognition model uses deep learning technology and is trained with a large number of professional personnel image samples. It can automatically learn and extract the significant features of different occupations from visual features. Subsequently, the occupational recognition model gives the most likely occupational category of the two types of monitored objects based on the input features, and marks them as corresponding occupational identifications to obtain occupational identification results. If the occupational identification result of the second category of monitoring objects is within the preset known occupational categories, such as couriers, deliverymen, firefighters, etc., the second category of monitoring objects will be directly identified as the corresponding known occupations, and the corresponding known occupation identification will be obtained; if the occupational identification result of the second category of monitoring objects is judged not to exist in the preset known occupational categories, the second category of monitoring objects will be identified as unknown occupations, and an unknown occupation identification will be obtained.

[0038] By automatically judging the occupational identity of outsiders based on visual features, campus security management can obtain more comprehensive and fine-grained personnel information, laying the foundation for intelligent behavior analysis and anomaly detection. Based on occupational identification information, differentiated management rules and monitoring strategies can be formulated for personnel of different occupations, so as to achieve refined control of personnel identities and improve the overall level of campus security management.

[0039] S600: When a monitored object belonging to the first category enters a restricted movement zone, the monitored object of the first category is marked as abnormal; when a monitored object of the second category has the known occupational identification and does not move according to the preset identification path, the monitored object of the second category is marked as abnormal; when the monitored object of the second category has the unknown occupational identification, the monitored object of the second category is marked as abnormal.

[0040] Specifically, for different categories of monitoring objects, differentiated abnormal behavior analysis methods are used. Specifically, for a type of monitoring object identified, that is, school personnel, attention is paid to whether they enter the prohibited area on campus, that is, the mobile restricted area, such as the dangerous area under construction. By comparing the real-time position of a type of monitoring object with the pre-calibrated campus restricted area position, it is determined whether the school personnel have entered the restricted area. Once the location point is found to fall within the range of the mobile restricted area, the type of monitoring object is marked as abnormal, and an abnormal behavior warning is immediately triggered so that campus managers can take safety precautions in time.

[0041] The second type of monitoring objects identified, namely outsiders, include outsiders with known occupational identification and outsiders with unknown occupational identification. For outsiders with known occupational identification, such as couriers, takeout deliverymen, firefighters, etc., the standard behavior paths of these known occupational personnel are pre-set. The actual movement trajectory of outsiders is compared with the standard path of their occupation in real time. If the actual trajectory is found to deviate significantly from the standard path, the second type of monitoring object is marked as abnormal, so as to effectively monitor the activity trajectory of outsiders with known occupational identification and detect violations in time. For outsiders with unknown occupational identification, since their normal behavior patterns cannot be known, the second type of monitoring object is marked as abnormal as soon as they appear; at the same time, outsiders with unknown occupational identification are continuously tracked and all their activity trajectories are monitored. Once discovered, the highest level of abnormal warning is triggered, prompting campus management personnel to pay close attention to the movement of the person and take strong control measures when necessary.

[0042] By making full use of the location, identity and behavioral characteristics of different monitoring objects, differentiated abnormal judgments are established, and all-round, classified abnormal behavior monitoring and early warning of both on-campus and outsiders are achieved. Various abnormal situations on campus can be discovered in a timely and accurate manner, improving the intelligence level and security assurance capabilities of campus monitoring, and providing strong technical support for campus safety management.

[0043] Furthermore, communicating with the surveillance camera array and receiving surveillance video information, performing deep learning fusion analysis on the surveillance video information, and obtaining a surveillance crowd positioning point cloud, including: S210: The surveillance video information includes the surveillance video of the first camera to the surveillance video of the Nth camera; S220: performing semantic segmentation on the first camera surveillance video to the Nth camera surveillance video through a semantic segmentation network of a deep learning fusion model, and obtaining the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information; S230: fusing the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information through a point cloud fusion network of a deep learning fusion model to obtain global point cloud positioning information at a first moment to global point cloud positioning information at an Mth moment; S240: Add the global point cloud positioning information from the first moment to the global point cloud positioning information at the Mth moment into the monitored crowd positioning point cloud.

[0044] In a feasible implementation, the surveillance video information includes video clips collected by multiple campus surveillance cameras, from the first camera surveillance video to the Nth camera surveillance video, covering real-time surveillance images of different areas in the campus. These video data provide raw materials for subsequent video fusion analysis. Then, the surveillance videos of multiple cameras are processed using the semantic segmentation network in the deep learning fusion model. Among them, semantic segmentation is a pixel-level classification method that can semantically annotate and classify each pixel in the video screen. Through the semantic segmentation network, the human target area is accurately segmented from the video frame image, and the position, size, pixel and other information of the human target in each frame image are extracted. All surveillance videos from the first camera to the Nth camera are processed with semantic segmentation in turn to obtain the personnel position point cloud data under each camera's perspective, that is, the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information. The point cloud positioning timing information of each camera is composed of a personnel position point cloud sequence extracted frame by frame from the perspective, reflecting the personnel position distribution and movement trajectory under the perspective.

[0045] Subsequently, the point cloud fusion network in the deep learning fusion model is used to integrate the point cloud positioning time sequence information of multiple cameras. Since there are differences in perspectives and spatial transformation relationships between the video images shot by different cameras, it is necessary to perform spatial registration and coordinate transformation on the point cloud data from different perspectives and fuse them into a unified point cloud of personnel positions in the campus coordinate system. The point cloud fusion network achieves spatiotemporal synchronization and coordinate unification of multi-perspective point clouds through operations such as feature matching and spatial transformation. The fusion result is a series of global point cloud positioning information, from the global point cloud positioning information at the first moment to the global point cloud positioning information at the Mth moment. The global point cloud at each moment contains the personnel position point cloud from the perspective of all cameras at that moment, reflecting the distribution of personnel throughout the school. Afterwards, the global point cloud positioning information from the first moment to the Mth moment is arranged in chronological order to form a time-series global personnel position point cloud set, namely the monitoring crowd positioning point cloud, which contains the location information of all monitored personnel in the campus for M consecutive moments, laying a data foundation for subsequent tasks such as on-campus personnel identification and abnormal behavior analysis.

[0046] Through the organic combination of semantic segmentation network and point cloud fusion network, the information extraction and fusion from local single-view surveillance video to global multi-view personnel location point cloud is realized, providing high-quality data support for campus surveillance video anomaly detection.

[0047] Furthermore, the semantic segmentation network includes a first convolution semantic segmentation base channel to a Qth convolution semantic segmentation base channel. The semantic segmentation network of the deep learning fusion model is used to perform semantic segmentation on the first camera surveillance video to the Nth camera surveillance video, and obtain the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information, including: S221: Segment the first camera surveillance video at a certain time to obtain a surveillance image at a first time; S222: Processing the first moment monitoring image for semantic segmentation through the first convolution semantic segmentation base channel to obtain first base channel point cloud positioning information; S223: until the Qth convolution semantic segmentation base channel is passed, the first moment monitoring image is processed for semantic segmentation to obtain the Qth base channel point cloud positioning information; S224: Perform a union full connection on the first base channel point cloud positioning information until the Q-th base channel point cloud positioning information to obtain the first camera point cloud positioning information at the first moment, and add it to the first camera point cloud positioning timing information.

[0048] In a preferred embodiment, the semantic segmentation network is composed of multiple parallel convolutional semantic segmentation base channels, namely the first convolutional semantic segmentation base channel to the Qth convolutional semantic segmentation base channel. Each convolutional semantic segmentation base channel is an independent convolutional neural network responsible for extracting specific semantic information from the input image. Multiple convolutional semantic segmentation base channels work in parallel to understand the image semantically from different angles, and finally fuse the semantic information extracted by each to obtain comprehensive and accurate camera point cloud positioning timing information.

[0049] Taking the semantic segmentation processing of the surveillance video of the first camera as an example, first, the surveillance video of the first camera is segmented at a certain time, and the continuous video stream is cut into independent frames. For each frame of the image, the corresponding time information is extracted to form a series of discrete surveillance images, such as the surveillance image at the first moment and the surveillance image at the second moment, so as to prepare for the subsequent semantic segmentation processing. Then, the surveillance image at the first moment is processed using the first convolutional semantic segmentation base channel in the semantic segmentation network. The base channel extracts low-level semantic features in the image, such as edges, textures, colors, etc., through a convolutional neural network, and generates a rough segmentation result to identify the human target area in the image. The segmentation result is represented in the form of a point cloud, where each point corresponds to a pixel in the image and carries the category label and spatial coordinate information of the pixel. The segmentation result obtained by processing the surveillance image at the first moment by the first convolutional semantic segmentation base channel is the first base channel point cloud positioning information.

[0050] Repeat the processing process of the convolution semantic segmentation base channel, and use the second convolution semantic segmentation base channel until the Qth convolution semantic segmentation base channel in turn to perform semantic segmentation on the surveillance image at the first moment. Different base channels extract semantic features at different levels and aspects to generate multiple complementary segmentation results. After each convolution semantic segmentation base channel processes the image, a corresponding point cloud positioning information can be obtained, such as the point cloud positioning information of the second base channel until the point cloud positioning information of the Qth base channel. After that, all the point cloud positioning information generated by the first convolution semantic segmentation base channel to the Qth convolution semantic segmentation base channel are integrated and fused, and the semantic features extracted by different base channels are fused to generate a unified and high-quality semantic segmentation result, that is, the point cloud positioning information of the first camera at the first moment. The point cloud contains the pixel-level segmentation masks and spatial coordinate information of all human targets under the perspective of the first camera at the first moment. The point cloud positioning information of the first camera at the first moment is added to the point cloud positioning timing information of the first camera, and together with the point clouds at other moments, it constitutes the complete timing information of the first camera personnel position.

[0051] Through the design of the multi-channel convolution semantic segmentation network, the complementary advantages of different convolution channels in feature extraction and semantic segmentation are fully utilized, and rich and diverse semantic features can be learned from the input image, and robust and accurate human target segmentation results can be generated. By fusing the segmentation results of different channels, the performance and reliability of semantic segmentation are further improved, providing solid data support for subsequent monitoring object positioning, anomaly detection, etc.

[0052] Furthermore, the point cloud fusion network includes a point cloud cleaning channel and a point cloud fully connected channel. Through the point cloud fusion network of the deep learning fusion model, the first camera point cloud positioning timing information until the Nth camera point cloud positioning timing information is fused to obtain the first moment global point cloud positioning information until the Mth moment global point cloud positioning information, including: S231: cleaning and analyzing the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information through a point cloud cleaning channel, and extracting the first camera point cloud positioning information at the first moment; S232: until the point cloud positioning information of the Nth camera at the first moment is extracted according to the point cloud positioning timing information of the Nth camera; S233: Performing a union and full connection on the point cloud positioning information of the first camera at the first moment to the point cloud positioning information of the Nth camera at the first moment through a point cloud full connection channel to obtain the global point cloud positioning information at the first moment.

[0053] In a preferred embodiment, the point cloud fusion network consists of a point cloud cleaning channel and a point cloud fully connected channel. The point cloud cleaning channel is responsible for preprocessing and cleaning the input point cloud data, removing noise points and outliers, and extracting high-quality and reliable point clouds. The point cloud fully connected channel uses a fully connected neural network to perform spatial transformation and feature fusion on the cleaned point clouds from different camera perspectives to generate a unified global point cloud representation.

[0054] Taking the generation of global point cloud positioning information at the first moment as an example, first, the point cloud cleaning channel is used to clean and analyze the point cloud positioning timing information from the first camera to the Nth camera. The cleaning process automatically detects and removes noise points, outliers and error points in the point cloud sequence of each camera through statistical methods and geometric constraints. These points usually come from local errors or random disturbances in the semantic segmentation results, which will affect the accuracy of subsequent fusion. After cleaning, the point cloud positioning information of each camera at the first moment is extracted to form the point cloud positioning information of the first camera at the first moment, which is used as the input for subsequent fusion. Subsequently, the cleaning process is repeated to extract the point cloud positioning information at the first moment from the point cloud positioning timing information of the second camera to the Nth camera in turn. The point clouds at the first moment under these different camera perspectives are represented as the point cloud positioning information of the second camera at the first moment to the point cloud positioning information of the Nth camera at the first moment. Afterwards, the point cloud fully connected channel is used to perform a full connection operation on the point cloud positioning information of the first camera at the first moment to the point cloud positioning information of the Nth camera at the first moment, and the spatial correspondence between point clouds of different perspectives is converted to a unified global coordinate system. The fusion result is the globally consistent and complete global point cloud positioning information at the first moment, which includes the point cloud of all personnel positions detected in the entire school at the first moment, and each point carries its global coordinates and identity attribute information. The above steps are repeated M times, and the global point cloud positioning information at the first moment can be obtained in sequence until the global point cloud positioning information at the Mth moment, forming a complete campus personnel position point cloud time series information.

[0055] By utilizing the point cloud cleaning channel, the original multi-view point cloud is intelligently enhanced and improved in quality, noise and erroneous information is removed. Then, the point cloud fully connected channel is used to learn the spatial mapping relationship of point clouds from different viewpoints, and feature extraction and information aggregation are performed on point clouds from all viewpoints to obtain a global point cloud representation, providing a high-quality data foundation for subsequent personnel identification and trajectory analysis.

[0056] Furthermore, the point cloud cleaning channel construction steps include: S251: Collecting the time sequence record information of the point cloud positioning of the first camera until the time sequence record information of the point cloud positioning of the Nth camera; S252: extracting first positioning point record information from the first camera point cloud positioning time series record information, wherein the first positioning record information includes a body shape record image, a clothing record image, a carrying tool record image and a first positioning point record time; S253: According to the body shape recorded image, the clothing recorded image, the carried tool recorded image and the first positioning point recorded time, the elapsed time is traversed in the first camera point cloud positioning time series recorded information to obtain the elapsed time detection position time series information of the first positioning point of the single camera; S254: extracting the point cloud positioning timing record information of adjacent cameras from the second camera point cloud positioning timing record information to the Nth camera point cloud positioning timing record information according to the first camera; S255: According to the body shape record image, the clothing record image, the carrying tool record image and the first positioning point record time, the elapsed time is traversed in the adjacent camera point cloud positioning time sequence record information to obtain the elapsed time detection position time sequence information of the first positioning point of multiple cameras; S256: When the proportion of the number of first positioning loss moments in the time sequence information of the position detection elapsed by the first positioning point of the single camera is greater than or equal to the positioning loss moment ratio threshold, and the proportion of the number of second positioning loss moments in the time sequence information of the position detection elapsed by the first positioning point of the multiple cameras is greater than or equal to the positioning loss moment ratio threshold, the first positioning point in the first camera point cloud positioning time sequence record information is cleaned and added to the cleaning mark result; S257: Using the cleaning mark as supervision data and the first camera point cloud positioning timing record information to the Nth camera point cloud positioning timing record information as input data, training the point cloud cleaning channel.

[0057] In a preferred embodiment, in order to construct a point cloud cleaning channel, first, data collection is performed to obtain the point cloud positioning time series record information from the first camera to the Nth camera in a historical period as the original data set for cleaning model training. The point cloud positioning time series record information of each camera contains a sequence of personnel position point clouds at multiple consecutive moments, as well as the attribute information of each detected personnel target. Then, the first positioning point record information of interest is extracted from the first camera point cloud positioning time series record information. The first positioning point represents a personnel target to be analyzed. Its record information includes a body shape record image (describing the human body shape), a wearing record image (describing clothing features), a carrying tool record image (describing personal belongings) and a first positioning point record time (describing the timestamp of the target's appearance), and these attribute information are used to subsequently determine the continuity and reliability of the positioning point. Subsequently, based on the attribute information of the first positioning point (i.e., the body shape record image, the wearing record image, the carrying tool record image and the first positioning point record time), the historical moments are traversed in the first camera point cloud positioning time series record information. By comparing the attributes of the point clouds at historical moments, the point cloud position sequence matching the first positioning point is identified, that is, the position timing information of the first positioning point of a single camera at the elapsed time is detected, reflecting the motion trajectory of the first positioning point under the viewing angle of the camera.

[0058] Subsequently, according to the topological adjacency relationship of the first camera, the point cloud positioning time series record information of other cameras adjacent to it (the second camera to the Nth camera) is extracted as the point cloud positioning time series record information of the adjacent cameras to prepare for cross-camera analysis. Next, based on the attribute information of the first positioning point, the historical moments are traversed in the point cloud positioning time series record information of the adjacent cameras. Through attribute comparison, the cross-camera point cloud position sequence matching the first positioning point is identified, that is, the time series information of the position detected by the multi-camera first positioning point at the elapsed time, which reflects the spatial transfer and time continuity of the first positioning point between adjacent cameras. Thereafter, the single-camera and multi-camera detection position time series information of the first positioning point is analyzed, and the positioning loss ratio is counted to obtain the proportion of the number of first positioning loss moments in the time series information of the detection position of the single-camera first positioning point at the elapsed time and the number of second positioning loss moments in the time series information of the detection position of the multi-camera first positioning point at the elapsed time. When the proportion of positioning loss moments in a single-camera sequence (i.e., the proportion of the number of first positioning loss moments) exceeds the positioning loss moment ratio threshold, and the proportion of positioning loss moments in a multi-camera sequence (i.e., the number of second positioning loss moments) also exceeds the positioning loss moment ratio threshold, it is considered that the detection result of the first positioning point under the perspective of the first camera is unreliable and needs to be cleaned. A cleaning mark is added to the cleaning mark result. After that, the generated cleaning mark result set is used as supervision information, and the point cloud positioning time sequence record information from the first camera to the Nth camera is used as input to train the point cloud cleaning channel in a data-driven manner. Through end-to-end learning, discriminative features are automatically extracted, cleaning strategies are optimized, and a robust point cloud cleaning channel with strong generalization capabilities is formed.

[0059] By making full use of the spatiotemporal continuity of human targets under the perspective of multiple cameras, noise points and outliers can be automatically identified through cross-camera and cross-time information fusion and comparison. This prior knowledge is introduced into the training process of the point cloud cleaning channel to form data-driven point cloud cleaning, which significantly improves the accuracy and adaptability of point cloud cleaning.

[0060] Furthermore, collecting the wearing images and the carrying tool images of the two types of monitoring objects, performing occupation identification based on the wearing images and the carrying tool images, and obtaining occupation identification results, including: S510: performing occupation identification based on the wearing images and the carrying tool images of the second type of monitoring objects to obtain an initial occupation identification result, wherein the initial occupation identification result includes an initial known occupation identification and an initial unknown occupation identification; S520: When the initial occupation identification result is the initial known occupation identification, setting the occupation identification result as the known occupation identification; S530: When the initial occupation identification result is the initial unknown occupation identification, performing trajectory accompanying identification on the second type of monitoring object to obtain a trajectory accompanying probability; S540: When the trajectory accompanying probability is greater than or equal to the trajectory accompanying probability threshold, converting the second type of monitoring object into the same type as the trajectory accompanying object; S550: When the trajectory accompanying probability is less than the trajectory accompanying probability threshold, the occupation identification result is set as the unknown occupation identification.

[0061] In a preferred embodiment, for outsiders (i.e., second-category monitoring objects) detected in video surveillance, their occupational identities are determined by analyzing the characteristics of the clothes and items they carry, combined with trajectory accompanying identification.

[0062] First, feature extraction and pattern recognition are performed on the images of clothing and tools carried by the second category of monitoring objects to give the initial occupational determination results. The initial occupational identification results can be divided into two types, namely, the initial known occupational identification (matching the predefined occupational category) and the initial unknown occupational identification (unable to determine the clear occupation). Through the intelligent analysis of clothing and personal belongings, the occupation of the outsider can be preliminarily judged, providing a reference for further confirmation. Then, the initial occupational identification result is determined. If the initial occupational identification result is the initial known occupational identification, it means that the occupational identity of the outsider can be relatively certain through visual feature analysis, such as couriers, firefighters, etc. At this time, the final occupational identification result is directly set to the known occupational identification.

[0063] In the case where the initial occupational identification is an initial unknown occupational identification, it is difficult to determine the occupational attributes of the outsider only by appearance features. At this time, the trajectory accompanying recognition mechanism is introduced to analyze the movement trajectory of the outsider on campus to determine whether he has a long-term trajectory companionship and interaction with a known identity person (such as students, teachers). The recognition result is quantified by the trajectory accompanying probability, reflecting the proportion of time that the outsider travels with the known identity person. According to the level of the trajectory accompanying probability, the identity type of the outsider is determined. When the trajectory accompanying probability exceeds the preset trajectory accompanying probability threshold, it indicates that the outsider has a significant trajectory correlation with the corresponding known identity person, and it can be inferred that the outsider is likely to have the same identity category as the person. At this time, the type of the outsider is transformed and classified into the category to which the trajectory accompanying object belongs (such as visitors, parents, etc.), and the identity confirmation is completed. When the trajectory accompanying probability is less than the trajectory accompanying probability threshold, it indicates that the outsider has no significant trajectory correlation with any known identity person, and there is a lack of effective context clues to infer his identity. At this time, the occupational identification result of the outsider is maintained as an unknown occupational identification person.

[0064] By combining visual feature analysis and trajectory analysis, we first use appearance features such as clothing to make preliminary judgments. For those who cannot be identified initially, we then introduce trajectory analysis and infer their identities through spatial association with known persons. Through the multi-strategy collaborative recognition mechanism, we can effectively improve the coverage and accuracy of occupational recognition and provide more comprehensive and reliable personnel identity information for campus security.

[0065] Further, when the initial occupation identification result is the initial unknown occupation identification, trajectory accompanying identification is performed on the second type of monitoring object to obtain the trajectory accompanying probability, including: S531: Obtaining first historical movement trajectory information of the second type of monitoring object; S532: extracting a plurality of first point cloud historical movement trajectory information of the on-campus personnel positioning point cloud according to the trajectory statistical time sequence of the first historical movement trajectory information; S533: traversing the plurality of first point cloud historical movement trajectory information and the first historical movement trajectory information to perform trajectory accompanying degree calculation, obtaining a plurality of first trajectory accompanying time proportions, extracting a maximum value of the plurality of first trajectory accompanying time proportions, and setting it as the trajectory accompanying probability; S534: extracting a plurality of second point cloud historical movement trajectory information of the monitoring object point cloud having the known occupation identification according to the trajectory statistical time sequence of the first historical movement trajectory information; S535: traversing the plurality of second point cloud historical movement trajectory information and the first historical movement trajectory information to perform trajectory accompanying degree calculation, obtaining a plurality of second trajectory accompanying time proportions, extracting a maximum value of the plurality of first trajectory accompanying time proportions, and setting it as the trajectory accompanying probability; Among them, when the positioning distance at the same time is less than or equal to the positioning distance threshold, it is considered as trajectory accompaniment.

[0066] Specifically, for outsiders whose initial occupations cannot be determined, the trajectory accompaniment relationship with on-campus personnel and personnel of other known occupations is analyzed, the trajectory accompaniment probability is calculated, and then the identity of the person is determined.

[0067] First, the historical movement trajectory information of the outsider (second-class monitoring object) to be identified is obtained and recorded as the first historical movement trajectory information. The trajectory information is composed of a series of positioning points of the person on campus, reflecting the spatial activity sequence within a certain time range. Then, with the trajectory statistical time sequence of the first historical movement trajectory information as a reference, several first point cloud historical movement trajectory information overlapping with it in time is extracted from the positioning point cloud data of the on-campus personnel. Each first point cloud historical movement trajectory information corresponds to the movement sequence of an on-campus person in the same time range. Through time alignment, the trajectory fragments in which the outsider and the on-campus person may interact in space are obtained. Subsequently, several first point cloud historical movement trajectory information are traversed, and the point-by-point distance calculation is performed with the first historical movement trajectory information to obtain the spatial distance sequence of the two trajectory information at each moment. Then, the number of moments in the entire sequence when the spatial distance between the two trajectories is less than the positioning distance threshold (regarded as trajectory accompaniment) is counted, and the proportion of the total number of moments is recorded as a first trajectory accompaniment moment ratio. Repeat this process to obtain the first trajectory accompaniment moment ratio of the outsider and each on-campus person, and then extract the maximum value from it as the trajectory accompaniment probability of the outsider and the on-campus person group. This probability reflects the degree to which outsiders travel and interact with on-campus personnel in both time and space.

[0068] At the same time, the object of the extracted trajectory is changed from the on-campus personnel to other monitoring objects with known occupational identification, and several pieces of second point cloud historical movement trajectory information are obtained. Then, several pieces of second point cloud historical movement trajectory information are traversed and compared point by point with the first historical movement trajectory information. A series of second trajectory accompanying time proportions are obtained through trajectory accompanying degree calculation, and the maximum value is extracted from them as the trajectory accompanying probability of outsiders and known occupational personnel groups. At this point, the correlation measurement of outsiders with on-campus personnel and other known occupational personnel in the spatiotemporal trajectory is obtained, namely, the trajectory accompanying probability, which describes the possibility of the identity of outsiders.

[0069] Traditional methods mainly rely on visual features to determine identity, and it is difficult to identify people with unclear features. However, through the indirect identity inference mechanism based on the correlation of spatiotemporal trajectories, the strong correlation between identity and mobility patterns of people in campus scenes is fully utilized. By analyzing the accompanying interaction rules between outsiders and other people with known identities in the spatiotemporal dimension, the process-oriented behavioral characteristics are used to infer identity attribution, which makes up for the difficulty of obtaining identity using only visual features.

[0070] The campus surveillance video anomaly detection method supported by deep learning provided by the embodiment of the present invention has at least the following technical effects: Communicate with the on-campus personnel positioning terminal and receive the on-campus personnel positioning point cloud to provide data support for the subsequent distinction between on-campus and off-campus personnel. Communicate with the surveillance camera array and receive surveillance video information, perform deep learning fusion analysis on the surveillance video information, and obtain the surveillance crowd positioning point cloud, where any point in the on-campus personnel positioning point cloud and the surveillance crowd positioning point cloud represents a surveillance object, laying the foundation for subsequent classification and abnormal identification. Extract the surveillance objects that overlap with the on-campus personnel positioning point cloud in the surveillance crowd positioning point cloud and set them as the first-class surveillance objects, and the surveillance objects that do not overlap with the on-campus personnel positioning point cloud are set as the second-class surveillance objects to achieve the distinction between on-campus and off-campus personnel. Collect the clothing images and tool images of the second-class surveillance objects, perform occupational identification based on the clothing images and tool images, and obtain the occupational identification results, where the occupational identification results include known occupational identifications and unknown occupational identifications, providing a basis for abnormal behavior identification. When a Category I monitoring object enters a restricted area of ​​movement, the Category I monitoring object will be marked as abnormal; when a Category II monitoring object has a known occupational identification and does not move according to the preset identification path, the Category II monitoring object will be marked as abnormal; when a Category II monitoring object has an unknown occupational identification, the Category II monitoring object will be marked as abnormal. This enables accurate identification, classification and abnormal behavior analysis of on-campus personnel and different types of off-campus personnel, improving the intelligence and security level of campus monitoring.

[0071] Embodiment 2: like Figure 2 As shown, based on the same inventive concept as the campus surveillance video anomaly detection method supported by deep learning provided in Example 1, an embodiment of the present invention also provides a campus surveillance video anomaly detection system supported by deep learning, the system is communicatively connected with the campus personnel positioning terminal and the surveillance camera array, including: The positioning data receiving module 11 is used to communicate with the on-campus personnel positioning terminal and receive the on-campus personnel positioning point cloud; The video fusion analysis module 12 is used to communicate with the surveillance camera array and receive surveillance video information, perform deep learning fusion analysis on the surveillance video information, and obtain a surveillance crowd positioning point cloud, wherein any point of the campus personnel positioning point cloud and the surveillance crowd positioning point cloud represents a surveillance object; The on-campus personnel screening module 13 is used to extract the monitoring objects that overlap with the on-campus personnel positioning point cloud from the monitoring crowd positioning point cloud and set them as a type of monitoring objects; The outsider screening module 14 is used to extract the monitoring objects that do not overlap with the positioning point cloud of the on-campus personnel from the monitoring crowd positioning point cloud and set them as the second-class monitoring objects; The occupation identification analysis module 15 is used to collect the wearing images and the carrying tool images of the two types of monitoring objects, perform occupation identification based on the wearing images and the carrying tool images, and obtain occupation identification results, wherein the occupation identification results include known occupation identifications and unknown occupation identifications; The abnormal identification module 16 is used to mark the first category of monitored objects as abnormal when the first category of monitored objects enters the restricted movement zone, mark the second category of monitored objects as abnormal when the second category of monitored objects have the known occupational identification and do not move according to the preset identification path, and mark the second category of monitored objects as abnormal when the second category of monitored objects have the unknown occupational identification.

[0072] Furthermore, the video fusion analysis module 12 includes the following execution steps: The surveillance video information includes the surveillance video of the first camera to the surveillance video of the Nth camera; Perform semantic segmentation on the first camera surveillance video to the Nth camera surveillance video through a semantic segmentation network of a deep learning fusion model to obtain the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information; The first camera point cloud positioning timing information up to the N-th camera point cloud positioning timing information are fused through a point cloud fusion network of a deep learning fusion model to obtain global point cloud positioning information at a first moment up to global point cloud positioning information at an M-th moment; The global point cloud positioning information from the first moment to the global point cloud positioning information at the Mth moment is added into the monitored crowd positioning point cloud.

[0073] Furthermore, the semantic segmentation network includes a first convolutional semantic segmentation base channel to a Qth convolutional semantic segmentation base channel, and the video fusion analysis module 12 further includes the following execution steps: Segmenting the first camera surveillance video at a certain time to obtain a surveillance image at a first time; Processing the first moment monitoring image for semantic segmentation through the first convolution semantic segmentation base channel to obtain first base channel point cloud positioning information; Until the monitoring image at the first moment is processed through the Qth convolution semantic segmentation base channel to perform semantic segmentation and obtain the Qth base channel point cloud positioning information; Perform a union and full connection on the first base channel point cloud positioning information until the Q-th base channel point cloud positioning information to obtain the first camera point cloud positioning information at the first moment, and add it to the first camera point cloud positioning timing information.

[0074] Furthermore, the point cloud fusion network includes a point cloud cleaning channel and a point cloud fully connected channel. Through the point cloud fusion network of the deep learning fusion model, the video fusion analysis module 12 also includes the following execution steps: Clean and analyze the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information through a point cloud cleaning channel, and extract the first camera point cloud positioning information at the first moment; Until the point cloud positioning information of the Nth camera at the first moment is extracted according to the point cloud positioning timing information of the Nth camera; The point cloud positioning information of the first camera at the first moment to the point cloud positioning information of the Nth camera at the first moment are fully connected through the point cloud fully connected channel to obtain the global point cloud positioning information at the first moment.

[0075] Furthermore, the point cloud cleaning channel construction step includes: Collect the time sequence record information of the point cloud positioning of the first camera until the time sequence record information of the point cloud positioning of the Nth camera; Extracting first positioning point record information from the first camera point cloud positioning time series record information, wherein the first positioning record information includes a body shape record image, a clothing record image, a carrying tool record image and a first positioning point record time; According to the body shape record image, the clothing record image, the carrying tool record image and the first positioning point record time, the elapsed time is traversed in the first camera point cloud positioning time series record information to obtain the elapsed time detection position time series information of the first positioning point of the single camera; According to the first camera, extracting the point cloud positioning timing record information of adjacent cameras from the second camera point cloud positioning timing record information to the Nth camera point cloud positioning timing record information; According to the body shape record image, the clothing record image, the carrying tool record image and the first positioning point record time, the elapsed time is traversed in the adjacent camera point cloud positioning time sequence record information to obtain the elapsed time detection position time sequence information of the first positioning point of multiple cameras; When the proportion of the number of first positioning loss moments in the time sequence information of the detected position of the first positioning point of the single camera is greater than or equal to the positioning loss moment ratio threshold, and the proportion of the number of second positioning loss moments in the time sequence information of the detected position of the first positioning point of the multi-camera is greater than or equal to the positioning loss moment ratio threshold, the first positioning point in the first camera point cloud positioning time sequence record information is cleaned and added to the cleaning mark result; The point cloud cleaning channel is trained by using the cleaning mark as supervision data and the first camera point cloud positioning timing record information to the Nth camera point cloud positioning timing record information as input data.

[0076] Furthermore, the occupation identification analysis module 15 includes the following execution steps: Perform occupation identification based on the wearing images and the carrying tool images of the two types of monitoring objects to obtain an initial occupation identification result, wherein the initial occupation identification result includes an initial known occupation identification and an initial unknown occupation identification; When the initial occupation identification result is the initial known occupation identification, setting the occupation identification result as the known occupation identification; When the initial occupation identification result is the initial unknown occupation identification, performing trajectory accompanying identification on the second type of monitoring object to obtain a trajectory accompanying probability; When the trajectory accompanying probability is greater than or equal to the trajectory accompanying probability threshold, the second type of monitoring object is converted into the same type as the trajectory accompanying object; When the trajectory accompanying probability is less than the trajectory accompanying probability threshold, the occupation identification result is set as the unknown occupation identification.

[0077] Furthermore, the occupation identification analysis module 15 also includes the following execution steps: Obtaining first historical movement trajectory information of the second type of monitoring objects; Extracting a plurality of first point cloud historical movement trajectory information of the on-campus personnel positioning point cloud according to the trajectory statistical time sequence of the first historical movement trajectory information; Traversing the plurality of first point cloud historical movement trajectory information and the first historical movement trajectory information to perform trajectory accompanying degree calculation, obtaining a plurality of first trajectory accompanying time proportions, extracting a maximum value of the plurality of first trajectory accompanying time proportions, and setting it as the trajectory accompanying probability; Extracting a plurality of second point cloud historical movement trajectory information of the monitoring object point cloud having the known occupation identification according to the trajectory statistical time sequence of the first historical movement trajectory information; Traversing the plurality of second point cloud historical movement trajectory information and the first historical movement trajectory information to calculate the trajectory accompanying degree, obtaining a plurality of second trajectory accompanying time proportions, extracting the maximum value of the plurality of first trajectory accompanying time proportions, and setting it as the trajectory accompanying probability; Among them, when the positioning distance at the same time is less than or equal to the positioning distance threshold, it is considered as trajectory accompaniment.

[0078] Embodiment three: See also Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3As shown, an electronic device 200 provided by an embodiment of the present invention includes a memory 210, a processor 220, and a first computer program 211 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the first computer program 211, a campus surveillance video anomaly detection method supported by deep learning is implemented.

[0079] Embodiment 4: See also Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 300 on which a second computer program 311 is stored. When the second computer program 311 is executed by a processor, a campus surveillance video anomaly detection method supported by deep learning is implemented.

[0080] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] Although preferred embodiments of the present invention have been described, additional changes and modifications may occur to these embodiments once those skilled in the art understand the basic inventive concepts.

[0086] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. A campus surveillance video anomaly detection method supported by deep learning, characterized in that: A campus surveillance video anomaly detection system supported by deep learning is applied, wherein the system is connected to a campus personnel positioning terminal and a surveillance camera array for communication, and includes: Communicate with the on-campus personnel positioning terminal and receive the on-campus personnel positioning point cloud; Communicate with the surveillance camera array and receive surveillance video information, perform deep learning fusion analysis on the surveillance video information, and obtain a surveillance crowd positioning point cloud, wherein any point of the campus personnel positioning point cloud and the surveillance crowd positioning point cloud represents a surveillance object; Extracting the monitoring objects that overlap with the positioning point cloud of the on-campus personnel from the positioning point cloud of the monitored crowd and setting them as a type of monitoring objects; Extracting the monitoring objects in the monitoring crowd positioning point cloud that do not overlap with the positioning point cloud of the on-campus personnel and setting them as the second-category monitoring objects; Collecting clothing images and carrying tool images of the two types of monitoring objects, performing occupation identification based on the clothing images and the carrying tool images, and obtaining occupation identification results, wherein the occupation identification results include known occupation identifications and unknown occupation identifications; When the first category of monitored objects enters the restricted movement zone, the first category of monitored objects are marked as abnormal; when the second category of monitored objects have the known occupational identification and do not move according to the preset identification path, the second category of monitored objects are marked as abnormal; when the second category of monitored objects have the unknown occupational identification, the second category of monitored objects are marked as abnormal.

2. The method according to claim 1, characterized in that Communicate with the surveillance camera array and receive surveillance video information, perform deep learning fusion analysis on the surveillance video information, and obtain the surveillance crowd positioning point cloud, including: The surveillance video information includes the surveillance video of the first camera to the surveillance video of the Nth camera; Perform semantic segmentation on the first camera surveillance video to the Nth camera surveillance video through a semantic segmentation network of a deep learning fusion model to obtain the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information; The first camera point cloud positioning timing information up to the N-th camera point cloud positioning timing information are fused through a point cloud fusion network of a deep learning fusion model to obtain global point cloud positioning information at a first moment up to global point cloud positioning information at an M-th moment; The global point cloud positioning information from the first moment to the global point cloud positioning information at the Mth moment is added into the monitored crowd positioning point cloud.

3. The method according to claim 2, characterized in that The semantic segmentation network includes a first convolution semantic segmentation base channel to a Qth convolution semantic segmentation base channel, and semantic segmentation is performed on the first camera surveillance video to the Nth camera surveillance video through the semantic segmentation network of the deep learning fusion model to obtain the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information, including: Segmenting the first camera surveillance video at a certain time to obtain a surveillance image at a first time; Processing the first moment monitoring image for semantic segmentation through the first convolution semantic segmentation base channel to obtain first base channel point cloud positioning information; Until the monitoring image at the first moment is processed through the Qth convolution semantic segmentation base channel to perform semantic segmentation and obtain the Qth base channel point cloud positioning information; Perform a union and full connection on the first base channel point cloud positioning information until the Q-th base channel point cloud positioning information to obtain the first camera point cloud positioning information at the first moment, and add it to the first camera point cloud positioning timing information.

4. The method according to claim 2, characterized in that The point cloud fusion network includes a point cloud cleaning channel and a point cloud fully connected channel. The point cloud fusion network of the deep learning fusion model fuses the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information to obtain the global point cloud positioning information at the first moment to the global point cloud positioning information at the Mth moment, including: Clean and analyze the first camera point cloud positioning timing information to the Nth camera point cloud positioning timing information through a point cloud cleaning channel, and extract the first camera point cloud positioning information at the first moment; Until the point cloud positioning information of the Nth camera at the first moment is extracted according to the point cloud positioning timing information of the Nth camera; The point cloud positioning information of the first camera at the first moment to the point cloud positioning information of the Nth camera at the first moment are fully connected through the point cloud fully connected channel to obtain the global point cloud positioning information at the first moment.

5. The method according to claim 4, characterized in that The point cloud cleaning channel construction step includes: Collect the time sequence record information of the point cloud positioning of the first camera until the time sequence record information of the point cloud positioning of the Nth camera; Extracting first positioning point record information from the first camera point cloud positioning time series record information, wherein the first positioning record information includes a body shape record image, a clothing record image, a carrying tool record image and a first positioning point record time; According to the body shape record image, the clothing record image, the carrying tool record image and the first positioning point record time, the elapsed time is traversed in the first camera point cloud positioning time series record information to obtain the elapsed time detection position time series information of the first positioning point of the single camera; According to the first camera, extracting the point cloud positioning timing record information of adjacent cameras from the second camera point cloud positioning timing record information to the Nth camera point cloud positioning timing record information; According to the body shape record image, the clothing record image, the carrying tool record image and the first positioning point record time, the elapsed time is traversed in the adjacent camera point cloud positioning time sequence record information to obtain the elapsed time detection position time sequence information of the first positioning point of multiple cameras; When the proportion of the number of first positioning loss moments in the time sequence information of the detected position of the first positioning point of the single camera is greater than or equal to the positioning loss moment ratio threshold, and the proportion of the number of second positioning loss moments in the time sequence information of the detected position of the first positioning point of the multi-camera is greater than or equal to the positioning loss moment ratio threshold, the first positioning point in the first camera point cloud positioning time sequence record information is cleaned and added to the cleaning mark result; The point cloud cleaning channel is trained by using the cleaning mark as supervision data and the first camera point cloud positioning timing record information to the Nth camera point cloud positioning timing record information as input data.

6. The method according to claim 1, characterized in that Collecting the wearing images and the carrying tool images of the two types of monitoring objects, performing occupation identification based on the wearing images and the carrying tool images, and obtaining occupation identification results, including: Perform occupation identification based on the wearing images and the carrying tool images of the two types of monitoring objects to obtain an initial occupation identification result, wherein the initial occupation identification result includes an initial known occupation identification and an initial unknown occupation identification; When the initial occupation identification result is the initial known occupation identification, setting the occupation identification result as the known occupation identification; When the initial occupation identification result is the initial unknown occupation identification, performing trajectory accompanying identification on the second type of monitoring object to obtain a trajectory accompanying probability; When the trajectory accompanying probability is greater than or equal to the trajectory accompanying probability threshold, the second type of monitoring object is converted into the same type as the trajectory accompanying object; When the trajectory accompanying probability is less than the trajectory accompanying probability threshold, the occupation identification result is set as the unknown occupation identification.

7. The method according to claim 6, characterized in that When the initial occupation identification result is the initial unknown occupation identification, performing trajectory accompanying identification on the second type of monitoring object to obtain trajectory accompanying probability includes: Obtaining first historical movement trajectory information of the second type of monitoring objects; Extracting a plurality of first point cloud historical movement trajectory information of the on-campus personnel positioning point cloud according to the trajectory statistical time sequence of the first historical movement trajectory information; Traversing the plurality of first point cloud historical movement trajectory information and the first historical movement trajectory information to perform trajectory accompanying degree calculation, obtaining a plurality of first trajectory accompanying time proportions, extracting a maximum value of the plurality of first trajectory accompanying time proportions, and setting it as the trajectory accompanying probability; Extracting a plurality of second point cloud historical movement trajectory information of the monitoring object point cloud having the known occupation identification according to the trajectory statistical time sequence of the first historical movement trajectory information; Traversing the plurality of second point cloud historical movement trajectory information and the first historical movement trajectory information to perform trajectory accompanying degree calculation, obtaining a plurality of second trajectory accompanying time proportions, extracting the maximum value of the plurality of first trajectory accompanying time proportions, and setting it as the trajectory accompanying probability; Among them, when the positioning distance at the same time is less than or equal to the positioning distance threshold, it is considered as trajectory accompaniment.

8. The campus surveillance video anomaly detection system supported by deep learning is characterized by: The method for detecting anomalies in campus surveillance videos supported by deep learning according to any one of claims 1 to 7 is used for implementing the method, wherein the system is communicatively connected with a campus personnel positioning terminal and a surveillance camera array, and comprises: A positioning data receiving module, the positioning data receiving module is used to communicate with the on-campus personnel positioning terminal and receive the on-campus personnel positioning point cloud; A video fusion analysis module, which is used to communicate with the surveillance camera array and receive surveillance video information, perform deep learning fusion analysis on the surveillance video information, and obtain a surveillance crowd positioning point cloud, wherein any point of the campus personnel positioning point cloud and the surveillance crowd positioning point cloud represents a surveillance object; A school personnel screening module, the school personnel screening module is used to extract the monitoring objects that overlap with the school personnel positioning point cloud from the monitoring crowd positioning point cloud and set them as a type of monitoring objects; An outsider screening module, the outsider screening module is used to extract the monitoring objects that do not overlap with the on-campus personnel positioning point cloud from the monitoring crowd positioning point cloud and set them as the second-class monitoring objects; an occupation identification and analysis module, the occupation identification and analysis module is used to collect the wearing images and the carrying tool images of the two types of monitoring objects, perform occupation identification based on the wearing images and the carrying tool images, and obtain occupation identification results, wherein the occupation identification results include known occupation identifications and unknown occupation identifications; An abnormal identification module is used to mark the first category of monitored objects as abnormal when the first category of monitored objects enters a restricted movement zone, mark the second category of monitored objects as abnormal when the second category of monitored objects have the known occupational identification and do not move according to the preset identification path, and mark the second category of monitored objects as abnormal when the second category of monitored objects have the unknown occupational identification.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the campus surveillance video anomaly detection method supported by deep learning as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, which, when executed by a processor, implements the campus surveillance video anomaly detection method supported by deep learning as described in any one of claims 1 to 7.

Images