Campus abnormal event monitoring video analysis method based on graph neural network

Abstract

The invention provides a campus abnormal event monitoring video analysis method based on a graph neural network. The method comprises the following steps: collecting a plurality of video pictures in a campus; based on a campus abnormal behavior event relevancy judgment model pre-constructed by utilizing a graph neural network, determining relevancy of each video picture; sorting the video pictures from large to small according to the corresponding relevancy to obtain a video picture sequence; and pushing the video picture sequence to campus safety management personnel. According to the campus abnormal behavior monitoring technology based on the graph neural network, a worker can quickly and effectively check the video clip most possibly containing the abnormal behavior, accurate judgment and confirmation of the abnormal behavior are facilitated, the efficiency and accuracy of processing the campus abnormal behavior event are greatly improved, and the user experience is improved. And corresponding measures are taken in time to protect the safety and rights of the students.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior monitoring, and in particular to a method for analyzing abnormal event monitoring videos on campus based on a graph neural network. Background Art

[0002] Abnormal behaviors on campus usually occur all over the campus. After receiving an alarm about abnormal behaviors on campus, it is necessary to conduct an investigation in time to stop and punish the relevant personnel. However, the densely distributed cameras on campus contain a large amount of video data, which brings great difficulty and pressure to the investigation. It is urgent to develop an intelligent campus abnormal behavior investigation method and technology based on graph neural network to assist schools in improving the efficiency of investigating abnormal behaviors on campus.

[0003] However, existing technologies and methods mainly involve in-depth analysis of a large amount of video data captured by numerous cameras on campus. Due to the vast scope of the campus and the large number of cameras, the amount of video data collected is huge, including various complex scenes and activities. If all the video data are processed without distinction, the processing time will be too long and the processing efficiency will be relatively low. Summary of the invention

[0004] The present invention provides a campus abnormal event monitoring video analysis method based on graph neural network, comprising:

[0005] Collect multiple video images on campus;

[0006] Determine the relevance of each of the video images based on a campus abnormal behavior event relevance determination model pre-constructed using a graph neural network;

[0007] Sort the video frames in descending order according to the corresponding relevance to obtain a video frame sequence;

[0008] The video image sequence is pushed to campus security management personnel.

[0009] Optionally, the collecting of multiple video images on campus includes:

[0010] Capture multiple video images on campus through video equipment pre-installed in multiple key areas on campus;

[0011] The key areas include at least: playground, classroom, library and cafeteria.

[0012] Optionally, the structure of pre-building a campus abnormal behavior event correlation determination model based on a graph neural network includes:

[0013] The input layer, encoder, decoder, fully connected layer, graph convolutional layer and output layer are connected in sequence;

[0014] Among them, the input layer is used to input video images;

[0015] The encoder is used to convert the input video image into a feature vector W X1 ; W X1 =σ(A1X1+b 11 ), where σ represents the activation function in the graph neural network, A1 represents the weight parameter matrix of the graph neural network, X1 represents the input video screen, and b 11 represents the bias parameter matrix;

[0016] The decoder is used to transform the feature vector W X1 Transformed into video feature extraction result Y1; Y1 = σ(C1W X1 +b 12 ), σ represents the activation function in the graph neural network, C1 represents the weight parameter matrix in the decoder model, and b 12 represents the bias parameter matrix;

[0017] The fully connected layer is used to fully connect the video feature extraction result Y1 with the abnormal behavior event related information, and set the initial weight of each edge after the full connection to the same initial value to obtain a fully connected graph;

[0018] The graph convolution layer is used to transform the fully connected graph into a result graph;

[0019] The output layer is used to extract the edge weight values ​​connecting the video features with the relevant information from the result graph and use them as the relevance.

[0020] Optionally, the determining the relevance of each of the video images based on the campus abnormal behavior event relevance determination model pre-constructed using a graph neural network includes:

[0021] Traversing each of the video screens in sequence, and each time traversing, inputting the traversed video screen into the campus abnormal behavior event relevance determination model pre-built based on the graph neural network to obtain the relevance of the traversed video screen;

[0022] After traversing each of the video frames in sequence, the relevance of each of the video frames is obtained.

[0023] Optionally, the campus abnormal event monitoring video analysis technology based on graph neural network also includes:

[0024] When the campus security manager verifies that abnormal campus behavior occurs on campus based on the video picture sequence, the campus security manager is assisted in handling the abnormal campus behavior.

[0025] The present invention provides a campus abnormal behavior monitoring system based on graph neural network, comprising:

[0026] The acquisition module is used to acquire multiple video images on campus;

[0027] A determination module, used to determine the relevance of each of the video images based on a campus abnormal behavior event relevance determination model pre-constructed using a graph neural network;

[0028] A sorting module, used for sorting the video frames from large to small according to the corresponding relevance, to obtain a video frame sequence;

[0029] The push module is used to push the video picture sequence to campus security management personnel.

[0030] The present invention provides an electronic device, comprising:

[0031] at least one processor; and

[0032] A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any of the above-mentioned methods for monitoring abnormal behavior on campus based on graph neural networks.

[0033] The present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement any of the above-mentioned methods for monitoring abnormal behavior on campus based on graph neural networks when executed.

[0034] The present invention provides a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device executes the computer program code of any of the above-mentioned methods for monitoring abnormal behavior on campus based on graph neural networks.

[0035] The beneficial effects achieved by the present invention are as follows:

[0036] The present invention enables staff to quickly and effectively view video clips that are most likely to contain abnormal behavior, which helps to accurately identify and confirm abnormal behavior, thereby greatly improving the efficiency and accuracy of handling abnormal behavior incidents on campus and taking corresponding measures in a timely manner to protect students' safety and rights.

[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0038] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0040] Figure 1 It is a flow chart of a campus abnormal event monitoring video analysis method based on graph neural network in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a modeling process in an embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of the deployment mode of the campus abnormal behavior prevention and control model system based on privacy protection in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the application of the campus abnormal behavior event correlation determination model in an embodiment of the present invention;

[0044] Figure 5 Schematic diagram of a campus abnormal behavior monitoring system based on graph neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0046] The present invention provides a campus abnormal event monitoring video analysis method based on graph neural network, such as Figure 1 As shown, including:

[0047] S1, collect multiple video images on campus;

[0048] S2. Determine the relevance of each of the video images based on a campus abnormal behavior event relevance determination model pre-constructed using a graph neural network;

[0049] S3, sorting the video frames from large to small according to the corresponding relevance to obtain a video frame sequence;

[0050] S4. Pushing the video image sequence to campus security management personnel.

[0051] Relevance refers to the relevance of video images to abnormal behavior events on campus; the video images are sorted according to the relevance to abnormal behavior events on campus and then presented to relevant processing personnel, which reduces the time spent by personnel in screening video materials and improves the efficiency of managers in handling abnormal behavior events. Based on the graph neural network, a relevance determination model for abnormal behavior events on campus is pre-built, and the relevance of each video image to abnormal behavior events on campus is determined using the relevance determination model for abnormal behavior events on campus pre-built based on the graph neural network.

[0052] In one embodiment, the collecting of multiple video images on campus includes:

[0053] Capture multiple video images on campus through video equipment pre-installed in multiple key areas on campus;

[0054] The key areas include at least: playground, classroom, library and cafeteria.

[0055] Camera equipment can be set up in advance at key areas on campus where abnormal behavior incidents may occur to collect video footage.

[0056] In one embodiment, the structure of the campus abnormal behavior event correlation determination model pre-built based on the graph neural network includes:

[0057] The input layer, encoder, decoder, fully connected layer, graph convolutional layer and output layer are connected in sequence;

[0058] Among them, the input layer is used to input video images;

[0059] The encoder is used to convert the input video image into a feature vector W X1 ; W X1 =σ(A1X1+b 11 ), where σ represents the activation function in the graph neural network, A1 represents the weight parameter matrix of the graph neural network, X1 represents the input video screen, and b 11 represents the bias parameter matrix;

[0060] The decoder is used to transform the feature vector W X1 Transformed into video feature extraction result Y1; Y1 = σ(C1W X1 +b 12 ), σ represents the activation function in the graph neural network, C1 represents the weight parameter matrix in the decoder model, and b 12 represents the bias parameter matrix;

[0061] The fully connected layer is used to fully connect the video feature extraction result Y1 with the abnormal behavior event related information, and set the initial weight of each edge after the full connection to the same initial value to obtain a fully connected graph;

[0062] The graph convolution layer is used to transform the fully connected graph into a result graph;

[0063] The output layer is used to extract the edge weight values ​​connecting the video features with the relevant information from the result graph and use them as the relevance.

[0064] like Figure 2 As shown, in a specific application, the video image X1 is input into the campus abnormal behavior event relevance determination model. After the video image X1 passes through the encoder of the campus abnormal behavior event relevance determination model (i.e., the part of the graph neural network, which is used to convert the input data into a high-dimensional representation of the feature vector), the feature vector W is calculated. X1 =σ(A1X1+b 11 ), where σ represents the activation function in the network, A1 represents the weight parameter matrix in the model network, which is used for linear transformation operations and represents the relationship between input features and output features, and b 11 Represents the bias parameter matrix, which is used for the offset term in the linear transformation operation to translate the input data;

[0065] The feature vector W X1 After passing through another decoder of the campus abnormal behavior event correlation determination model (i.e., the part of the neural network, which is used to convert the encoded information back to the original data space or the target output form, reconstruct the original information from the compressed or abstract representation or generate a new output), the video feature extraction result Y1=σ(C1W X1 +b 12 ), σ represents the activation function in the network, C1 represents the weight parameter matrix in the decoder model, which is used for linear transformation operations and represents the relationship between input features and output features, b 12 Represents the bias parameter matrix, which is used for the offset term in the linear transformation operation to translate the input data. The optimization goal is L1=min||A L1 Y1-Y true || 2 +β1||A L1 ||2, where β1 is a hyperparameter, ||A L1 ||2 is the 2-norm constraint. As part of the loss function, a penalty term based on the parameter 2-norm is added to the original loss function. By limiting the size of the model parameters, the complexity of the model is reduced and the generalization ability is improved.

[0066] The video feature extraction result Y1 and the abnormal behavior event related information X2 input in the form of feature vectors are fully connected, and the initial weight (initial weight refers to the initial value assigned to each edge at the beginning of model training; usually these weights will be adjusted through optimization algorithms during training to make the model better fit the data) of each edge (edge ​​is the association between video features and abnormal behavior event related information nodes. Through these edges, information can be transmitted between nodes) is set to the same initial value (initial value is the weight value assigned to each edge before model training. Setting the same initial value means that all edges have the same importance in the initial stage), and then transformed into a fully connected graph X3, which is input into the graph convolution layer (graph convolution layer is a neural network structure specially used for processing graph data. It can effectively extract node features and edge features by applying convolution operations on the structure of the graph. The graph convolution layer is used to process graph structure data related to video features and abnormal behavior events. It will learn and predict based on the input graph data to identify abnormal behavior in the video. The propagation mode of each layer of the graph neural network in the graph convolution layer (that is, the way information is transmitted between different layers in the graph neural network) is (This formula shows the transfer mode from layer l to layer l+1), where H (l) is the input feature of the lth layer (i.e., the input received by each layer of the graph neural network), W (l) is the weight matrix of the lth layer (used to perform linear transformation on the input features to help the model learn node features), σ is the nonlinear activation function (used to introduce nonlinear characteristics so that the neural network can express complex functional relationships), I is the identity matrix, yes The degree matrix of j is the index of all nodes in the graph;

[0067] After the fully connected graph X3 is propagated through the multi-layer graph neural network, the final result graph Y2 is obtained. The optimization target (that is, the loss function that needs to be minimized when training the model to guide the update of model parameters. The optimization target here is the cross entropy loss function, which is used to measure the difference between the predicted category distribution and the actual category distribution) is where y L is the index set of all samples in the training set, F is the number of categories, and Y lf is the true label of sample l belonging to category f, and Z is the forward propagation function in, is the forward propagation function, σ is the nonlinear activation function (used to introduce nonlinear characteristics so that the neural network can express complex functional relationships), H (l)is the input feature of the lth layer (i.e., the input received by each layer of the graph neural network), W (l) The weight matrix of the lth layer (used to perform linear transformation on the input features to help the model learn node features), is a symmetric and normalized matrix after standardization. D is The degree matrix of j is the index of all nodes in the graph;

[0068] From the result graph Y2, the value of the edge weight connecting the video feature and the related information is extracted, that is, the correlation degree Y3 is obtained.

[0069] like Figure 3 As shown in the figure, a large amount of video data is recorded by cameras in the activity places of middle school students in the school. The collected data is input into the campus abnormal behavior video data relevance judgment system based on graph neural network for relevance judgment. Finally, according to the judgment results of the model, the video data is provided to the processing personnel according to the relevance for identification, to verify the authenticity and degree of the abnormal behavior incidents on campus, and to realize the pre-processing of the video data, which can effectively improve the efficiency of campus staff in handling abnormal behavior incidents on campus.

[0070] like Figure 4 In the system architecture shown, first of all, high-definition camera equipment needs to be deployed and installed in key areas of the school's student activity areas, such as playgrounds, classrooms, libraries, and canteens. These cameras are responsible for continuously collecting video data of students' daily activities, thus forming a comprehensive monitoring network. These video data will be uploaded to the school's remote server in real time or periodically for subsequent data analysis and processing.

[0071] When the model system is running, the system first receives and stores the video data collected by the camera. Once the school personnel receive a report of abnormal behavior incidents on campus, the system will start. The system is built based on a graph neural network and is specifically used to judge and analyze the relevance of video materials of abnormal behavior incidents on campus. Based on the information provided in the report, the system will judge the relevance of video material clips divided by time and place. Then, the video material clips will be sorted according to relevance, and the video clips with high relevance will be presented to campus security personnel and management for further verification and analysis.

[0072] The system enables staff to quickly and effectively review video clips that are most likely to contain abnormal behavior, which helps to accurately identify and confirm abnormal behavior, thereby greatly improving the efficiency and accuracy of handling abnormal behavior incidents on campus and taking appropriate measures in a timely manner to protect students' safety and rights.

[0073] In one embodiment, the relevance of each of the video images is determined based on the campus abnormal behavior event relevance determination model pre-built using a graph neural network, including:

[0074] Traversing each of the video screens in sequence, and each time traversing, inputting the traversed video screen into the campus abnormal behavior event relevance determination model pre-built based on the graph neural network to obtain the relevance of the traversed video screen;

[0075] After traversing each of the video frames in sequence, the relevance of each of the video frames is obtained.

[0076] Based on the campus abnormal behavior event relevance determination model pre-built using the graph neural network, when determining the relevance of each of the video frames, traverse each video frame in turn, and each time traversal is completed, it is input into the campus abnormal behavior event relevance determination model pre-built using the graph neural network for processing to obtain the relevance.

[0077] In one embodiment, the campus abnormal event monitoring video analysis technology based on graph neural network also includes:

[0078] When the campus security manager verifies that abnormal campus behavior occurs on campus based on the video picture sequence, the campus security manager is assisted in handling the abnormal campus behavior.

[0079] When assisting, it can be determined whether to call police officers based on the severity of the abnormal behavior on campus.

[0080] The present invention provides a campus abnormal behavior monitoring system based on graph neural network, such as Figure 5 As shown, including:

[0081] The acquisition module 1 is used to acquire multiple video images on campus;

[0082] Determination module 2, used to determine the relevance of each of the video images based on a campus abnormal behavior event relevance determination model pre-constructed using a graph neural network;

[0083] A sorting module 3 is used to sort the video frames from large to small according to the corresponding relevance to obtain a video frame sequence;

[0084] The push module 4 is used to push the video picture sequence to campus security management personnel.

[0085] The present invention provides an electronic device, comprising:

[0086] at least one processor; and

[0087] A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any of the above-mentioned methods for monitoring abnormal behavior on campus based on graph neural networks.

[0088] The present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement any of the above-mentioned methods for monitoring abnormal behavior on campus based on graph neural networks when executed.

[0089] The present invention provides a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device executes the computer program code of any of the above-mentioned methods for monitoring abnormal behavior on campus based on graph neural networks.

[0090] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A campus abnormal event monitoring video analysis method based on graph neural network, characterized in that: include: Collect multiple video images on campus; Determine the relevance of each of the video images based on a campus abnormal behavior event relevance determination model pre-constructed using a graph neural network; Sort the video frames in descending order according to the corresponding relevance to obtain a video frame sequence; The video image sequence is pushed to campus security management personnel.

2. A campus abnormal event monitoring video analysis method based on graph neural network as claimed in claim 1, characterized in that: The collecting of multiple video images on campus includes: Capture multiple video images on campus through video equipment pre-installed in multiple key areas on campus; The key areas include at least: playground, classroom, library and cafeteria.

3. A campus abnormal event monitoring video analysis method based on graph neural network as claimed in claim 1, characterized in that: The structure of the campus abnormal behavior event correlation determination model pre-built based on graph neural network includes: The input layer, encoder, decoder, fully connected layer, graph convolutional layer and output layer are connected in sequence; Among them, the input layer is used to input video images; The encoder is used to convert the input video image into a feature vector W X1 ; W X1 =σ(A1X1+b 11 ), where σ represents the activation function in the graph neural network, A1 represents the weight parameter matrix of the graph neural network, X1 represents the input video image, and b 11 represents the bias parameter matrix; The decoder is used to transform the feature vector W X1 Transformed into video feature extraction result Y1; Y1 = σ(C1W X1 +b 12 ), σ represents the activation function in the graph neural network, C1 represents the weight parameter matrix in the decoder model, and b 12 represents the bias parameter matrix; The fully connected layer is used to fully connect the video feature extraction result Y1 with the abnormal behavior event related information, and set the initial weight of each edge after the full connection to the same initial value to obtain a fully connected graph; The graph convolution layer is used to transform the fully connected graph into a result graph; The output layer is used to extract the edge weight values ​​connecting the video features with the relevant information from the result graph and use them as the relevance.

4. A campus abnormal event monitoring video analysis method based on graph neural network as claimed in claim 1, characterized in that: The relevance determination model for abnormal campus behavior events pre-built using a graph neural network is used to determine the relevance of each of the video images, including: Traversing each of the video screens in sequence, and each time traversing, inputting the traversed video screen into the campus abnormal behavior event relevance determination model pre-built based on the graph neural network to obtain the relevance of the traversed video screen; After traversing each of the video frames in sequence, the relevance of each of the video frames is obtained.

5. A campus abnormal behavior monitoring system based on graph neural network, characterized in that: include: The acquisition module is used to acquire multiple video images on campus; A determination module, used to determine the relevance of each of the video images based on a campus abnormal behavior event relevance determination model pre-constructed using a graph neural network; A sorting module, used for sorting the video frames from large to small according to the corresponding relevance, to obtain a video frame sequence; The push module is used to push the video picture sequence to campus security management personnel.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and: A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for monitoring abnormal behavior on campus based on a graph neural network as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the campus abnormal behavior monitoring method based on graph neural network according to any one of claims 1 to 4 when executed.

8. A computer program product, characterized in that The computer program product includes computer instructions, and when executed by an electronic device, the electronic device executes the computer program code of the method for monitoring abnormal behavior on campus based on graph neural network as described in any one of claims 1-4.

Images