Method and system for modeling and detecting abnormal behavior based on construction personnel adjacency graph

Abstract

The application provides a kind of based on the abnormal behavior modeling and detection method and system of constructing personnel adjacency graph, comprising: step S1: personnel are tracked, obtain personnel number, personnel appear frame sequence number and in each frame position area coordinate information;Step S2: with each personnel target as node, establish node adjacency matrix, calculate the adjacency relationship between nodes, with the coordinate distance of current node and adjacent node as the weight of adjacent edge, construct personnel adjacency graph structure;Step S3: calculate the position area coordinate change of each node, set node features;Step S4: build network structure, input graph data, output each frame detection result, detect whether each graph contains abnormal node, if there is, it is considered that there is abnormal behavior in the frame, calculate loss and update network parameters.The Sub-GCN network proposed in the application aims to focus on the difference between different nodes in the same graph, and is more suitable for personnel adjacency graph model compared with other networks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital video processing, in particular, to a video abnormal behavior detection method, more particularly, to an abnormal behavior modeling and detection method and system based on constructing personnel adjacency relation graph. BACKGROUND

[0002] Video abnormal behavior detection is a task of identifying and detecting behaviors that may threaten social order and public safety in monitoring videos. With the improvement of social security level, more and more public places are equipped with monitoring cameras, which can be used to monitor and review the events occurred in the past. However, whether it is real-time monitoring of abnormal behaviors on the monitoring screen or searching for past events, it requires a lot of manpower and time. Video abnormal behavior detection aims to use computer vision methods to detect abnormal behaviors as a special classification task, extract features for modeling, and use deep networks to learn features, so as to automate the process of obtaining abnormal analysis results from monitoring videos, save human resources, and ensure the accuracy and efficiency of detection.

[0003] The research of abnormal behavior detection task has developed for many years and has formed a certain system. According to the technical route classification, it can be divided into classification model and reconstruction model; according to the supervision method classification, it can be divided into supervised, unsupervised, mixed, weakly supervised and self-supervised learning; according to the use of features, it can be divided into using appearance features and motion information combination, using skeleton features, etc.

[0004] “Abnormal” is a relative concept without fixed semantics, and its definition often depends on the context, which is ambiguous. The same event may be assigned different labels in the abnormal detection task because of the different scenes, which leads to the fact that although abnormal detection seems to be an action classification task, it is very different from action classification. It is this feature that the event is abnormal or not depends on the scene of occurrence, which makes it difficult to break through the bottleneck of the current industry application method, and it is difficult to balance accuracy and speed. In addition, the method applied in the industry mostly locates the time segment of abnormal occurrence, but does not specifically identify the location of the abnormal behavior in the picture.

[0005] Patent document CN114913599A discloses a video anomaly behavior detection method and system based on an auto-encoder, including video downsampling preprocessing, combining a background extraction algorithm with an image segmentation network to separate foreground and background from the preprocessed video image sequence, inputting the foreground and background into different channels of a dual-channel auto-encoder to extract features, and performing feature fusion in the hidden layer, and finally sending the fused features to a decoder training method. However, the invention directly extracts the features of the RGB picture, and does not convert the detection of whether each frame contains abnormal behavior into the detection of whether each graph contains an abnormal node.

[0006] Patent CN113313037A discloses a self-attention mechanism-based generative adversarial network video anomaly detection method, including video preprocessing, modeling motion information using a liteFlownet network, and introducing a generator and a Markov discriminator composed of a generative adversarial network with a self-attention mechanism. However, the invention does not use a personnel adjacency graph to model motion information, and does not introduce a Sub-GCN network for training and detection.

[0007] The paper Semi-Supervised Classification With Graph Convolutional Networks proposes the widely used GCN model, which uses neighborhood aggregation in the propagation process, aggregates the features of neighboring nodes to represent the current node, and combines the connection characteristics of the topological graph to achieve various tasks, including but not limited to social network prediction, traffic prediction, etc. However, in the anomaly detection task, the difference between nodes is the key to distinguishing node anomalies, therefore, the representation method of features needs to be further studied, and the invention does not use a new aggregation representation method and does not focus on the difference between the current node and its adjacent nodes in the same graph.

[0008] Patent CN109543539A discloses a crowd anomaly behavior detection method, the main features of which include: first step: collecting video data of the monitoring area and preprocessing; second step: using a mixture Gaussian model to model the background and separate the background pixels from the original image; third step: extracting the moving target from the monitoring video image;

[0009] Fourth step: record the trajectory information of the moving target and mark it; fifth step: track and count the marked moving target to calculate the traffic flow; sixth step: when the motion trajectory of the moving target exceeds the set threshold, an alarm information is sent. However, the invention does not introduce a Sub-GCN network for training and detection. SUMMARY

[0010] In view of the defects in the prior art, the present application aims to provide a method and system for modeling and detecting abnormal behavior based on a constructed personnel adjacency graph.

[0011] According to the present application, a method for modeling and detecting abnormal behavior based on a constructed personnel adjacency graph is provided, comprising:

[0012] Step S1: target tracking is performed on personnel to obtain personnel numbers, frame sequence numbers in which the personnel appear, and position region coordinate information in each frame;

[0013] Step S2: a node adjacency matrix is established with each personnel target as a node, the adjacency relationship between nodes is calculated, the coordinate distance between the current node and the adjacent node is taken as the adjacency edge weight, and a personnel adjacency graph structure is constructed;

[0014] Step S3: the position region coordinate change of each node is calculated, and node features are set;

[0015] Step S4: a network structure is constructed, graph data is input, a detection result of each frame is output, it is detected whether an abnormal node is contained in each graph, if there is, it is considered that abnormal behavior exists in the frame, a loss is calculated to update network parameters.

[0016] Preferably, in the step S1:

[0017] The original video data is processed by using a target tracker to obtain a tracking information sequence corresponding to each video, each piece of information in the sequence is composed of a frame number f, a personnel identity number p, a left upper corner coordinate (x1, y1) of a personnel identification frame, and a right lower corner coordinate (x2, y2);

[0018] For each piece of information in the information sequence, the coordinate (x, y) of the center of the current personnel is calculated f，p :

[0019]

[0020] Wherein, x, x1, x2 all represent horizontal coordinates, y, y1, y all represent vertical coordinates, f represents a frame number, and p represents a personnel identity number.

[0021] Preferably, in the step S2:

[0022] For each tracking information sequence of a video, the set of all personnel identity numbers p contained in each frame is the set of nodes in the topological graph to be constructed for each frame, and the set size is n;

[0023] For each video, the total number of frames is m, and the calculated personnel center coordinates (x, y) are data, a matrix d with a dimension of m×n×2 is established;

[0024] For each pair f, p:

[0025] d f，p = (x, y) f，p (2)

[0026] For the elements in the matrix which are not assigned values after the above calculation:

[0027]

[0028] Wherein:

[0029]

[0030]

[0031]

[0032]

[0033] Wherein, db is the frame in which the current personnel appears for the first time from the current frame, and the position data of the personnel in the frame is taken, f b Indicates the serial number of the frame found forwardly; df is the frame in which the current personnel appears for the first time from the current frame, and the position data of the personnel in the frame is taken, f f Indicates the serial number of the frame found backwardly;

[0034] For each two nodes in each frame, the distance of the node center position is calculated, and the reciprocal of the distance is taken as the edge weight to establish a non-directed edge for the two nodes:

[0035]

[0036]

[0037] Wherein f is the current frame, a and b are nodes, dis f，a，b is the distance between a and b in the frame f, w a，b is the weight of the edge with a and b as endpoints; d f，a is the position data of the personnel a in the frame f, d f，b is the position data of the personnel b in the frame f;

[0038] The nodes are established and the edge connection is given, and a personnel relationship topology graph is constructed for each frame of video.

[0039] Preferably, in the step S3:

[0040] The position area coordinate change of each node in the adjacent frames before and after is calculated, and the coordinate distance change amount of the current node and other nodes, the motion speed change amount and the motion direction change amount of the next time of the node are taken as the characteristics of each node;

[0041] For the personnel relationship topology graph G corresponding to frame f f , the adjacency matrix A f (n, n), the absolute value of the difference between the adjacency matrices of the adjacent two frames of personnel relationship topology graph is taken element by element, and then the row is added to obtain the feature value F measuring the positional relationship change of each node with other personnel in the current frame ADJ (n):

[0042] F ADJ (n) = sum(|A f (n, n) - A f-1 (n, n)|, axis = 0) (10)

[0043] For the personnel relationship topology graph G corresponding to frame f f , the node list in the frame, the difference between the center coordinates (x, y) of each node and the previous frame is constructed to construct the node motion change list loc f (n), and the horizontal coordinate x in it is constructed to construct the node horizontal coordinate change list x f (n), and the vertical coordinate y in it is constructed to construct the node vertical coordinate change list y f (n), the change amount in the horizontal and vertical directions is synthesized, and the feature value F recording the motion change information of each node in the current frame is obtained V (n):

[0044]

[0045]

[0046]

[0047]

[0048] Wherein, loc is the position information (x, y) containing horizontal coordinate and vertical coordinate;

[0049] For the personnel relationship topology graph G corresponding to frame f f , the node list in the frame, the information recorded by loc f (n) is the motion change amount, or the motion vector of the node from the previous frame position to the current frame position; take the cosine value of the included angle of the two motions recorded in the current frame, to measure the change of the motion direction; since the cosine function is a monotonically decreasing function in the interval [0, π], here the opposite number +1 is taken to obtain a function with a value range [0, 1] and a monotonically increasing function, and the value is taken as the feature value F recording the motion direction change information ANGLE (n):

[0050]

[0051] wherein,

[0052]

[0053] The motion features are combined into node features F:

[0054]

[0055] Preferably, in the step S4:

[0056] The Sub-GCN network structure is constructed, the detection of whether the abnormal behavior is contained in each frame is converted into the detection of whether the abnormal node is contained in each graph, the training process supervises whether the abnormal behavior is contained in each frame, the graph data is input, the detection result of each frame is output, the loss is calculated, and the network parameters are updated;

[0057] The adjacency matrix calculation method of Sub-GCN:

[0058]

[0059] wherein, is the adjacency matrix after adding a self-loop to the original graph, is is the corresponding degree matrix, and I is a unit matrix;

[0060] The weighted binary cross-entropy loss function is adopted:

[0061]

[0062] wherein, is the binary cross-entropy loss function, β is a weight, and depends on the proportion of positive and negative samples, y i represents a true value, represents the probability that the network output prediction is a normal class;

[0063] The 2-layer Sub-GCN structure is adopted, the Relu activation function is adopted between layers, the Softmax function is applied after convolution in the last layer to obtain the output probability, iterative training is performed under different hyperparameters, and the optimal model is obtained;

[0064] The test set divided out is subjected to the same preprocessing step, feature extraction step and modeling step to construct a personnel relationship topological graph, the constructed graph data is input into the trained Sub-GCN network, and a recognition result is obtained.

[0065] According to the abnormal behavior modeling and detection system based on construction of a personnel adjacency relationship graph provided by the application, the system comprises:

[0066] Module M1: target tracking is performed on the personnel to obtain personnel number, frame number where the personnel appears, and position area coordinate information in each frame;

[0067] Module M2: a node adjacency matrix is established with each personnel target as a node, the adjacency relationship between nodes is calculated, the coordinate distance of the current node and the adjacent node is taken as the adjacency edge weight, and a personnel adjacency relationship graph structure is constructed;

[0068] Module M3: the position area coordinate change of each node is calculated, and the node features are set;

[0069] Module M4: a network structure is constructed, graph data is input, and each frame detection result is output, whether each graph contains an abnormal node is detected, if there is, it is considered that there is an abnormal behavior in the frame, and the loss is calculated to update the network parameters.

[0070] Preferably, in the module M1:

[0071] The original video data is processed by using a target tracker to obtain a tracking information sequence corresponding to each video, and each piece of information in the sequence is composed of frame number f, personnel identity number p, left upper corner coordinate (x1, y1) of the personnel recognition frame, and right lower corner coordinate (x2, y2);

[0072] For each piece of information in the information sequence, the coordinate (x, y) of the center of the current personnel is calculated f，p :

[0073]

[0074] Wherein, x, x1, x2 represent horizontal coordinates, y, y1, y represent vertical coordinates, f represents frame number, and p represents personnel identity number.

[0075] Preferably, in the module M2:

[0076] For each video tracking information sequence, the set of all personnel identity numbers p contained in each frame is the set of nodes in the topological graph to be constructed for each frame, and the set size is n;

[0077] For each video, the total number of frames is m, and the personnel center coordinates (x, y) calculated are data, a matrix d with a dimension of mxn×2 is established;

[0078] For each pair f, p:

[0079] d f，p = (x, y) f，p (2)

[0080] For the elements in the matrix that have not been assigned values after the above calculation:

[0081]

[0082] wherein:

[0083]

[0084]

[0085]

[0086]

[0087] wherein db is the frame number of the nearest frame in which the current person appears, and the position data of the current person in the frame is taken, df is the frame number of the nearest frame in which the current person appears, and the position data of the current person in the frame is taken, b wherein db is the frame number of the nearest frame in which the current person appears, and the position data of the current person in the frame is taken, df is the frame number of the nearest frame in which the current person appears, and the position data of the current person in the frame is taken, f wherein db is the frame number of the nearest frame in which the current person appears, and the position data of the current person in the frame is taken, df is the frame number of the nearest frame in which the current person appears, and the position data of the current person in the frame is taken,

[0088] For each two nodes in each frame, the distance of the node center position is calculated, and the reciprocal of the distance is taken as the edge weight to establish a non-directed edge between the two nodes:

[0089]

[0090]

[0091] wherein f is the current frame, a and b are nodes, dis f，a，b is the distance between a and b in frame f, w a，b is the weight of the edge with a and b as endpoints; d f，a is the position data of person a in frame f, d f，b is the position data of person b in frame f;

[0092] The nodes are established and the edges are connected, and a personnel relationship topology graph is constructed for each frame of video.

[0093] Preferably, in the module M3:

[0094] The coordinate variation of each node in the adjacent frames is calculated, and the coordinate distance variation, the node next time motion speed variation and the motion direction variation of the current node and other nodes are taken as the characteristics of each node;

[0095] For the personnel relationship topology graph G f of frame f, there is an adjacency matrix A f (n, n), the adjacency matrices of the adjacent two frames of personnel relationship topology graph are subtracted by elements to take absolute values, and then added by rows to obtain a characteristic value F ADJ(n):

[0096] F ADJ (n)=sum(|A f (n,n)-A f-1 (n, n)|,axis=0) (10)

[0097] For the personnel relationship topology graph G corresponding to frame f f The list of nodes in the image is used to construct a list of node motion changes based on the difference between the center coordinates (x, y) of each node and the previous frame. f (n), and construct a list of node x-coordinate changes using the x-coordinates of each node. f (n), and construct a list of node y-coordinate changes using the y-coordinates of (n). f (n) is used to synthesize the changes in the horizontal and vertical directions, resulting in the feature value F of the motion change information of each node recorded in the current frame. V (n):

[0098]

[0099]

[0100]

[0101]

[0102] Where loc contains the position information (x, y) including the horizontal and vertical coordinates;

[0103] For the personnel relationship topology graph G corresponding to frame f f The list of nodes in loc f (n) The recorded information is the amount of motion change, or the motion vector of a node moving from the position of the previous frame to the position of the current frame; take the two motions recorded in the previous frame, calculate the cosine of the angle between the two motion vectors, and measure the change in motion direction; since the cosine function is a monotonically decreasing function in the interval [0, π], take the opposite number + 1 to get a function with a range of [0, 1] and a monotonically increasing value, and its value is used as the feature value F for recording the change in motion direction. ANGLE (n):

[0104]

[0105] in,

[0106]

[0107] The motion features are combined into node features F:

[0108]

[0109] Preferably, in the module M4:

[0110] The Sub-GCN network structure is constructed, detection of whether abnormal behavior is contained in each frame is converted into detection of whether abnormal nodes are contained in each graph, a training process supervises whether abnormal behavior is contained in each frame, graph data are input, and each frame detection result is output, a loss is calculated, and network parameters are updated;

[0111] The adjacency matrix calculation method of Sub-GCN is as follows:

[0112]

[0113] Wherein, is the adjacency matrix after adding a self-loop to the original graph, is is the corresponding degree matrix, and I is the unit matrix;

[0114] A weighted binary cross-entropy loss function is used:

[0115]

[0116] Wherein, is the binary cross-entropy loss function, beta is a weight, and depends on the proportion of positive and negative samples, y i represents a true value, represents the probability that the network output prediction is a normal class;

[0117] A 2-layer Sub-GCN structure is used, a Relu activation function is used between layers, a Softmax function is applied after convolution in the last layer to obtain an output probability, and the optimal model is obtained by iterative training under different hyperparameters;

[0118] The test set divided out is subjected to the same preprocessing step, feature extraction step and modeling step to construct a personnel relationship topological graph, and the constructed graph data are input into the trained Sub-GCN network to obtain a recognition result.

[0119] Compared with the prior art, the present application has the following beneficial effects:

[0120] 1. The present application proposes the concept of "personnel adjacency relationship graph", embeds the position correlation of different personnel in space and the time correlation of the same personnel in the movement process into the graph structure, represents complex image information in a clear and simple topological graph structure, eliminates the interference of background noise on detection, and simultaneously pays attention to the personnel characteristics and the relationship characteristics of other personnel in the scene;

[0121] 2. The Sub-GCN network proposed in the present application aims to pay attention to the differences between different nodes in the same graph, and is more suitable for the personnel adjacency relationship graph model than other networks.

[0122] 3. The application is a method that can detect abnormal behavior in monitoring video and locate the position of abnormal frames. The new modeling method can complete the abnormal detection task with less calculation amount, has the advantages of high detection accuracy and high detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0123] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments read in conjunction with the accompanying drawings:

[0124] Fig. 1 The training and detection logic flow chart of the method of the present application;

[0125] Fig. 2 The Sub-GCN network structure diagram of the method of the present application. DETAILED DESCRIPTION

[0126] The present application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present application. These are within the scope of protection of the present application.

[0127] Example 1

[0128] The abnormal behavior modeling and detection method based on constructing personnel adjacency relationship graph proposed by the present application aims to improve the detection accuracy while reducing the calculation amount, improve the detection efficiency, and identify the position of the abnormality in the picture.

[0129] The application provides a video abnormal behavior modeling and detection method based on constructing personnel adjacency graph, first, personnel in the video are tracked to obtain personnel number, frame number of each personnel appearing in the video and position area coordinate information of each personnel in each frame; then, personnel adjacency graph is constructed for each frame in the video, each personnel target is taken as a node, a node adjacency matrix is established to calculate the adjacency relationship between nodes, the coordinate distance between the current node and the adjacent node is taken as the adjacency edge weight to construct the personnel adjacency graph structure; further, the position area coordinate change of each node in the adjacent frames before and after is calculated, the coordinate distance change amount of the current node and other nodes, the node next time motion speed change amount and the motion direction change amount are taken as the characteristics of each node; a Sub-GCN network structure is designed to train and detect the constructed graph structure data, whether each frame contains abnormal behavior is detected is converted into whether each graph contains abnormal node, and whether each frame contains abnormal behavior is supervised and labeled in the training process; in the detection process, target tracking, feature extraction and establishment of the personnel adjacency graph of each frame are carried out on the unknown video, the graph data is input into the Sub-GCN network, whether each graph contains abnormal node is identified, and if the abnormal node exists, it is considered that the abnormal behavior exists in the frame.

[0130] The "personnel adjacency graph" concept provided by the application models each frame in the video as a topological graph composed of nodes and edges, embeds the position correlation of different personnel in space and the time correlation of the same personnel in the motion process into the graph structure, represents the complex image information in a clear and concise topological graph structure, eliminates the interference of background noise on detection, and simultaneously pays attention to the personnel characteristics and the relationship characteristics of other personnel in the scene. The local graph convolution network (Sub-GCN) is designed to pay attention to the difference between different nodes in the same graph, and is more suitable for the personnel relationship graph model than other networks. Through the two designs, the existence of the abnormal in the monitoring video can be detected, and the abnormal frame position can be located, and the detection accuracy and efficiency are high.

[0131] According to the abnormal behavior modeling and detection method based on constructing personnel adjacency graph provided by the application, as shown in the figure, Figs. 1-2 , the method comprises the following steps.

[0132] Step S1: personnel are tracked to obtain personnel number, frame number of personnel appearing and position area coordinate information in each frame;

[0133] Specifically, in the step S1, the personnel are tracked by using a target tracker to obtain a tracking information sequence corresponding to each video, and each piece of information in the sequence is composed of frame number f, personnel identity number p, personnel recognition frame upper left corner coordinate (x1, y1) and lower right corner coordinate (x2, y2).

[0134] The original video data is processed by using a target tracker to obtain a tracking information sequence corresponding to each video, and each piece of information in the sequence is composed of frame number f, personnel identity number p, personnel recognition frame upper left corner coordinate (x1, y1) and lower right corner coordinate (x2, y2).

[0135] For each piece of information in the information sequence, calculate the coordinates (x, y) where the current person center is located f，p :

[0136]

[0137] Wherein, x, x1, x2 represent the horizontal coordinates, y, y1, y represent the vertical coordinates, f represents the frame number, and p represents the personnel identity number.

[0138] Step S2: establishing a node adjacency matrix with each personnel target as a node, calculating the adjacency relationship between nodes, taking the coordinate distance of the current node and the adjacent node as the adjacency edge weight, and constructing a personnel adjacency relationship graph structure;

[0139] Specifically, in the step S2:

[0140] For each video tracking information sequence, the set of all personnel identity numbers p contained in each frame is the set of nodes required to be constructed in the topological graph of each frame, and the set size is n;

[0141] For each video, the total number of frames is m, and the personnel center coordinates (x, y) calculated are data, and a matrix d with dimensions m*n*2 is established;

[0142] For each pair f, p:

[0143] d f，p = (x, y) f，p (2)

[0144] For the elements in the matrix that have not been assigned values after the above calculation:

[0145]

[0146] Wherein:

[0147]

[0148]

[0149]

[0150]

[0151] Wherein, db is the frame in which the current personnel appears, and the position data of the personnel in the frame is taken, f b represents the serial number of the frame found forwardly; df is the frame in which the current personnel appears, and the position data of the personnel in the frame is taken, f f represents the serial number of the frame found backwardly.

[0152] For each two nodes in each frame, the distance of the node center position is calculated, and a non-directed edge is established between the two nodes with the reciprocal of the distance as the edge weight:

[0153]

[0154]

[0155] wherein f is the current frame, a, b are nodes, dis f，a，b is the distance between nodes a, b in frame f, w a，b is the weight of the edge with a, b as endpoints; d f，a is the position data of person a in frame f, d f，b is the position data of person b in frame f;

[0156] The nodes are established and the edge connections are given, and a personnel relationship topology graph is constructed for each frame of video.

[0157] Step S3: calculate the position region coordinate change of each node, and set the node features;

[0158] Specifically, in the step S3:

[0159] The position region coordinate change of each node in adjacent frames is calculated, and the coordinate distance change amount of the current node and other nodes, the node next time motion speed change amount and the motion direction change amount are taken as the features of each node;

[0160] For the personnel relationship topology graph G f corresponding to frame f, there is an adjacency matrix A f (n, n), the adjacency matrices of adjacent two frames of personnel relationship topology graph are subtracted element by element to take absolute values, and then added by rows to obtain a feature value F ADJ (n) that measures the position relationship change of each node and other persons in the current frame:

[0161] F ADJ (n) = sum(|A f (n, n) - A f-1 (n, n)|, axis = 0) (10)

[0162] For the node list in the personnel relationship topology graph G f corresponding to frame f, a node motion change list loc f (n) is constructed with the difference value of the center coordinates (x, y) of each node and the previous frame, and a node horizontal coordinate change list x f (n) is constructed with the horizontal coordinates x therein, and a node vertical coordinate change list y f(n) is used to synthesize the changes in the horizontal and vertical directions, resulting in the feature value F of the motion change information of each node recorded in the current frame. V (n):

[0163]

[0164]

[0165]

[0166]

[0167] Where l0c contains the position information (x, y) including the horizontal and vertical coordinates;

[0168] For the personnel relationship topology graph G corresponding to frame f f The list of nodes in loc f (n) The recorded information is the amount of motion change, or the motion vector of a node moving from the position of the previous frame to the position of the current frame; take the two motions recorded in the current frame backward, calculate the cosine value of the angle between the two motion vectors, and measure the change in motion direction; since the cosine function is a monotonically decreasing function in the interval [0, π], here we take the opposite number + 1 to obtain a function with a range of [0, 1] and a monotonically increasing value, and its value is used as the feature value F for recording the change in motion direction information. ANGLE (n):

[0169]

[0170] in,

[0171]

[0172] The motion features are combined into node features F:

[0173]

[0174] Step S4: Construct the network structure, input graph data, output the detection results for each frame, detect whether each graph contains abnormal nodes, if so, consider that there is abnormal behavior in that frame, calculate the loss and update the network parameters.

[0175] Specifically, in step S4:

[0176] Construct a Sub-GCN network structure, transforming the detection of whether there is abnormal behavior in each frame into the detection of whether there is abnormal node in each graph. During the training process, supervise and label whether there is abnormal behavior in each frame. Input graph data, output the detection results of each frame, calculate the loss and update the network parameters.

[0177] Method for calculating the adjacency matrix of Sub-GCN:

[0178]

[0179] wherein, is the adjacency matrix of the original graph with self-loops, is is the corresponding degree matrix, and I is the identity matrix;

[0180] The weighted binary cross-entropy loss function is used:

[0181]

[0182] wherein, is the binary cross-entropy loss function, and β is the weight, which depends on the proportion of positive and negative samples, y i represents the true value, represents the probability that the network output predicts as a normal class;

[0183] The 2-layer Sub-GCN structure is used, the Relu activation function is used between layers, the Softmax function is applied after the last layer of convolution to obtain the output probability, and the optimal model is obtained after iterative training under different hyperparameters.

[0184] The test set divided out is subjected to the same preprocessing step, feature extraction step, and modeling step to construct a personnel relationship topology graph, and the constructed graph data is input into the trained Sub-GCN network to obtain the recognition result.

[0185] Embodiment 2:

[0186] Embodiment 2 is a preferred example of Embodiment 1, which more specifically illustrates the present application.

[0187] The present application also provides a personnel adjacency relationship graph-based abnormal behavior modeling and detection system, which can be realized by executing the process steps of the personnel adjacency relationship graph-based abnormal behavior modeling and detection method, i.e., the personnel adjacency relationship graph-based abnormal behavior modeling and detection method can be understood by those skilled in the art as a preferred embodiment of the personnel adjacency relationship graph-based abnormal behavior modeling and detection system.

[0188] According to the personnel adjacency relationship graph-based abnormal behavior modeling and detection system provided by the present application, the personnel adjacency relationship graph-based abnormal behavior modeling and detection system comprises:

[0189] Module M1: target tracking of personnel, obtaining personnel number, frame sequence number of personnel appearance, and position area coordinate information in each frame;

[0190] Specifically, in the module M1:

[0191] The original video data is processed by using a target tracker to obtain a tracking information sequence corresponding to each video, each information in the sequence being composed of a frame number f, a personnel identity number p, a left upper corner coordinate (x1, y1) of a personnel recognition frame, and a right lower corner coordinate (x2, y2);

[0192] For each information in the information sequence, a coordinate (x, y) where a current personnel center is located is calculated f，p

[0193]

[0194] wherein x, x1, x2 all represent horizontal coordinates, y, y1, y all represent vertical coordinates, f represents a frame number, and p represents a personnel identity number.

[0195] Module M2: a node adjacency matrix is established with each personnel target as a node, an adjacency relationship between nodes is calculated, a coordinate distance of a current node and an adjacent node is taken as an adjacency edge weight, and a personnel adjacency relationship graph structure is constructed;

[0196] Specifically, in the module M2:

[0197] For each tracking information sequence of each video, a set of all personnel identity numbers p contained in each frame is a set of nodes required to be constructed in a topological graph of each frame, and the set size is n;

[0198] For each video, a matrix d with a dimension of mxn*2 is established with a total number of frames m and the calculated personnel center coordinates (x, y) as data.

[0199] For each pair f, p:

[0200] d f，p = (x, y) f，p (2)

[0201] For the elements in the matrix that are not assigned values after the above calculation:

[0202]

[0203] wherein:

[0204]

[0205]

[0206]

[0207]

[0208] ​wherein db is the frame found from the current frame backwardly to the nearest frame in which the current person appears, taking the position data of the person, f b denotes the serial number of the frame found forwardly; df is the frame found from the current frame forwardly to the nearest frame in which the current person appears, taking the position data of the person, f f denotes the serial number of the frame found backwardly;

[0209] For each two nodes in each frame, the distance of the node center position is calculated, and the reciprocal of the distance is taken as the edge weight to establish a non-directed edge between the two nodes:

[0210]

[0211]

[0212] wherein f is the current frame, a and b are nodes, dis f，a，b is the distance between a and b in the frame f, w a，b is the weight of the edge with a and b as endpoints; d f，a is the position data of the person a in the frame f, d f，b is the position data of the person b in the frame f;

[0213] The nodes are established and the edge connection is given, and a personnel relationship topology graph is constructed for each frame of video.

[0214] Module M3: calculating the position region coordinate change of each node and setting the node features;

[0215] Specifically, in the module M3:

[0216] The position region coordinate change of each node in the adjacent frames before and after is calculated, and the coordinate distance change amount of the current node and other nodes, the node next time motion speed change amount and the motion direction change amount are taken as the features of each node;

[0217] For the personnel relationship topology graph G f corresponding to the frame f, there is an adjacency matrix A f (n, n), the adjacency matrices of the adjacent two frames of personnel relationship topology graph are subtracted element by element to take the absolute value, and then the rows are added to obtain a feature value F ADJ (n) that measures the position relationship change of each node with other persons in the current frame:

[0218] F ADJ (n) = sum (|A f (n, n) - A f-1 (n, n) |, axis = 0) (10)

[0219] For the personnel relationship topology graph G floc is a list of nodes in the frame f, and the motion change list loc of each node is constructed by the difference between the center coordinate (x, y) of each node and the center coordinate (x, y) of the previous frame f (n), and the node horizontal coordinate change list x is constructed by the horizontal coordinate x in x f (n), and the node vertical coordinate change list y is constructed by the vertical coordinate y in y f (n), and the motion change information of each node in the current frame is recorded in the feature value F V (n):

[0220]

[0221]

[0222]

[0223]

[0224] wherein loc is the position information (x, y) containing the horizontal coordinate and the vertical coordinate;

[0225] For the personnel relationship topology graph G corresponding to the frame f f loc is a list of nodes in the frame f f (n) records the motion change amount, or the motion vector of the node moving from the position of the previous frame to the position of the current frame; taking the two motions recorded in the current frame, the cosine value of the included angle of the two motion vectors is calculated to measure the change of the motion direction; since the cosine function is a monotonically decreasing function in the interval [0, π], the inverse number +1 is taken to obtain a function with a value range [0, 1] and a monotonically increasing function, and the value is taken as the feature value F recording the motion direction change information ANGLE (n):

[0226]

[0227] wherein,

[0228]

[0229] The motion features are combined into the node feature F:

[0230]

[0231] Module M4: constructing a network structure, inputting graph data, outputting each frame detection result, detecting whether each graph contains an abnormal node, if there is, considering that there is an abnormal behavior in the frame, calculating a loss and updating network parameters.

[0232] Specifically, in the module M4:

[0233] The Sub-GCN network structure is constructed, detection of whether an abnormal behavior is contained in each frame is converted into detection of whether an abnormal node is contained in each graph, a training process supervises whether an abnormal behavior is contained in each frame, input graph data, and output of each frame detection result is obtained, and network parameters are updated by calculating loss;

[0234] The adjacency matrix calculation method of the Sub-GCN is as follows:

[0235]

[0236] Wherein, The adjacency matrix after adding a self-loop to the original graph is, The degree matrix is, The corresponding degree matrix is, and I is a unit matrix.

[0237] A weighted binary cross-entropy loss function is used:

[0238]

[0239] Wherein, The binary cross-entropy loss function is, β is a weight, and depends on the proportion of positive and negative samples, y i represents a true value, represents a probability that the network output predicts as a normal class;

[0240] A 2-layer Sub-GCN structure is used, a Relu activation function is used between layers, a Softmax function is applied after convolution in the last layer to obtain an output probability, and an optimal model is obtained by iterative training under different hyperparameters.

[0241] The test set divided out is subjected to the same preprocessing step, feature extraction step and modeling step to construct a personnel relationship topology graph, and the constructed graph data is input into the trained Sub-GCN network to obtain a recognition result.

[0242] Embodiment 3:

[0243] Embodiment 3 is a preferred example of Embodiment 1, and is used to more specifically illustrate the present application.

[0244] The present application is directed to video data such as a monitoring camera, and a modeling method based on computer vision is designed by extracting target personnel features and associated information with surrounding personnel, a topology graph structure-personnel adjacency relationship graph is established in the scene from the video frame, a Sub-GCN network structure is designed for training, so that the abnormality of the video frame can be judged, and in addition, the position of the abnormal personnel in the scene in the abnormal frame can also be obtained.

[0245] According to the abnormal behavior modeling and detection method based on the construction of the personnel adjacency relationship graph provided by the present application, the method comprises:

[0246] a tracking information extraction step of performing target tracking on the personnel in the video to obtain personnel numbers in the video, frame numbers at which each personnel appears in the video, and position region coordinate information in each frame;

[0247] a personnel adjacency graph construction step of taking each personnel target as a node, establishing a node adjacency matrix, calculating adjacency relationships between the nodes, taking coordinate distances of the current node and the adjacent nodes as adjacency edge weights, and constructing a personnel adjacency graph structure;

[0248] a node feature extraction step of calculating position region coordinate changes of each node in adjacent frames before and after the current node, and taking coordinate distance changes of the current node and other nodes, node next-time motion speed changes, and motion direction changes as node features;

[0249] a sub-graph convolution network (Sub-GCN) training step of constructing a Sub-GCN network structure, converting detection of whether each frame contains abnormal behavior into detection of whether each graph contains an abnormal node, requiring supervised labeling of whether each frame contains abnormal behavior in the training process, inputting graph data, outputting each frame detection result, calculating a loss, and updating network parameters;

[0250] an abnormal behavior detection step of performing target tracking, feature extraction, and establishment of a personnel adjacency graph of each frame on unknown videos in the detection process, inputting the graph data into the Sub-GCN network, identifying whether an abnormal node exists in each graph, and considering that a frame contains abnormal behavior if an abnormal node exists;

[0251] Specifically, the tracking information extraction step includes:

[0252] a target tracking step of processing original video data by using a target tracker (such as DeepSort, SiamFC, etc.) to obtain a tracking information sequence corresponding to each video, each piece of information in the sequence consisting of a frame number f, a personnel identity number p, a left upper corner coordinate (x1, y1) of a personnel bounding box, and a right lower corner coordinate (x2, y2);

[0253] a position calculation step of calculating a coordinate (x, y) at which a center of a current personnel is located for each piece of information in the information sequence f，p :

[0254]

[0255] wherein x, x1, and x2 represent horizontal coordinates, y, y1, and y represent vertical coordinates, f represents a frame number, and p represents a personnel identity number.

[0256] Specifically, the personnel adjacency graph construction step includes:

[0257] Building node set: For each video's tracking information sequence, the set of all person identity numbers p contained in each frame is the set of nodes needed to build the topology graph in each frame, and the set size is n.

[0258] Each frame node padding: For each video, the total number of frames is m, and the person center coordinates (x, y) calculated above are the data to establish a matrix d with dimensions m x n x 2.

[0259] For each pair f, p:

[0260] d f，p = (x, y) f，p (2)

[0261] For the elements in the matrix that have not been assigned values after the above calculation:

[0262]

[0263] Where:

[0264]

[0265]

[0266]

[0267]

[0268] Where db is the nearest frame that appears the current person from the current frame, and the position data of the person in the frame is taken, f b represents the serial number of the frame found forward; df is the nearest frame that appears the current person from the current frame, and the position data of the person in the frame is taken, f f represents the serial number of the frame found backward.

[0269] Building edges: For each pair of nodes in each frame, calculate the distance between the node center positions, and take the reciprocal of the distance as the edge weight to build an undirected edge between the two nodes:

[0270]

[0271]

[0272] Where f is the current frame, a and b are nodes, dis f，a，b is the distance between a and b in frame f, w a，b is the weight of the edge with a and b as endpoints. d f，a is the position data of person a in frame f, d f，b is the position data of person b in frame f.

[0273] Constructing graph: Establish nodes and give edge connection by the above method, namely, constructing a personnel relationship topology graph for each frame of video.

[0274] Specifically, the node feature extraction step comprises:

[0275] Calculating adjacency matrix change step: for the personnel relationship topology graph G f , there is an adjacency matrix A f (n, n). The adjacency matrices of adjacent two frames of personnel relationship topology graph are subtracted element by element to take absolute value, and then added by row to obtain a feature value F ADJ (n) that measures the positional relationship change of each node with other personnel in the current frame.

[0276] F ADJ (n) = sum (|A f (n, n) - A f-1 (n, n) |, axis = 0) (10)

[0277] Calculating motion speed change step: for the node list in the personnel relationship topology graph G f of frame f, a node motion change list loc f (n) is constructed with the difference between the center coordinates (x, y) of each node and the previous frame, and a node horizontal coordinate change list x f (n) is constructed with the horizontal coordinates in it, and a node vertical coordinate change list y f (n) is constructed with the vertical coordinates in it. The change amounts in horizontal and vertical directions are synthesized, namely a feature value F V (n) that records the motion change information of each node in the current frame.

[0278]

[0279]

[0280]

[0281]

[0282] Wherein, loc is the position information containing horizontal coordinate and vertical coordinate, namely (x, y);

[0283] Calculating motion direction change step: for the node list in the personnel relationship topology graph G f of frame f, loc f(n) The recorded information can be considered as the motion change amount, or as the motion vector of the node from the previous frame position to the current frame position. Taking the two motions recorded in the current frame forward, the cosine value of the angle between the two motion vectors is calculated, which measures the change in the direction of motion. Since the cosine function is a monotonically decreasing function in the interval [0, π], the opposite number +1 is taken to obtain a monotonically increasing function with a value range of [0, 1], and the value is taken as the characteristic value F of the recorded motion direction change information ANGLE (n):

[0284]

[0285] wherein

[0286]

[0287] The node feature adding step: the above three motion features are combined into a node feature F:

[0288]

[0289] Specifically, the Sub-GCN network training step includes:

[0290] Building a convolutional layer: compared with the traditional GCN, Sub-GCN focuses on the differences between different nodes in the same graph. Therefore, Sub-GCN has a different method of calculating the adjacency matrix compared with the traditional GCN:

[0291]

[0292] wherein: is the adjacency matrix after adding a self-loop to the original graph, is the corresponding degree matrix, and I is the identity matrix.

[0293] Setting the loss function: a weighted binary cross-entropy loss function is used.

[0294]

[0295] wherein, is the binary cross-entropy loss function, β is the weight, which depends on the proportion of positive and negative samples, yi represents the true value, represents the probability that the network output prediction is a normal class.

[0296] Building the network and training the optimal model: a 2-layer Sub-GCN structure is used, and a Relu activation function is used between layers. After the last layer of convolution, a Softmax function is applied to obtain the output probability. Under different hyperparameters, iterative training is performed to obtain the optimal model.

[0297] Specifically, the abnormal behavior detection step comprises: detecting the unknown video by a process of target tracking, feature extraction, establishing a personnel adjacency graph for each frame, inputting the graph data into a Sub-GCN network, identifying whether there is an abnormal node in each graph, and if there is, considering that there is abnormal behavior in the frame.

[0298] Embodiment 4:

[0299] Embodiment 4 is a preferred example of Embodiment 1, which more specifically illustrates the present application.

[0300] The abnormal behavior modeling and detection system based on the construction of personnel adjacency graph provided by the present application can be realized through the step flow of the abnormal behavior modeling and detection method based on the construction of personnel adjacency graph. Those skilled in the art can understand the abnormal behavior modeling and detection method based on the construction of personnel adjacency graph as a preferred example of the abnormal behavior modeling and detection system based on the construction of personnel adjacency graph.

[0301] The following further illustrates a preferred example.

[0302] As shown in Fig. 1 , Fig. 2 , an abnormal behavior modeling and detection method based on the construction of personnel adjacency graph mainly comprises six steps of S1-S6:

[0303] S1: Select normal behavior videos and abnormal behavior videos in a public database ShanghaiTech for model training and testing. The obtained videos are divided into a training set and a test set in a random division manner at a ratio of 7:3. Target tracking is performed on the training set videos to obtain the identity number of each video appearing personnel, the frame number of each personnel appearing in the video and the position area information in the scene. The specific implementation method is:

[0304] (1-1) Apply a Deep-Sort target tracker to process the original video data to obtain a tracking information sequence corresponding to each video, and each information in the sequence is composed of frame number f, personnel identity number p, personnel recognition box upper left corner coordinates (x1, y1), and right bottom corner coordinates (x2, y2);

[0305] (1-2) Position calculation step: for each information in the above information sequence, calculate the coordinates (x, y) of the center of the current personnel f，p :

[0306]

[0307] Wherein, x, x1, x2 represent horizontal coordinates, y, y1, y represent vertical coordinates, f represents frame number, and p represents personnel identity number.

[0308] S2: For each frame in the video, taking each person target with exclusive identity number as node, taking the extracted motion information as node feature, connecting each two nodes with edge, and constructing the personnel relationship topology graph. The specific implementation method is:

[0309] (2-1): For each video tracking information sequence, the set of all personnel identity numbers p contained is the set of nodes in the topology graph to be constructed for each frame, and the set size is n.

[0310] (2-2): For each video, the total number of frames is m, and the personnel center coordinates (x, y) calculated above are data, and a matrix d with dimensions m x n x 2 is established.

[0311] For each pair f, p:

[0312] d f，p = (x, y) f，p (2)

[0313] For the elements in the matrix that have not been assigned values after the above calculation:

[0314]

[0315] Wherein:

[0316]

[0317]

[0318]

[0319]

[0320] Wherein, db is the nearest frame in which the current personnel appears from the current frame, and the position data of the personnel in the frame is taken, f b represents the serial number of the frame found forwardly; df is the nearest frame in which the current personnel appears from the current frame, and the position data of the personnel in the frame is taken, f f represents the serial number of the frame found backwardly. (2-3): For each two nodes in each frame, the distance between the node center positions is calculated, and the reciprocal thereof is taken as the edge weight to establish a non-directed edge between the two nodes:

[0321]

[0322]

[0323] Wherein f is the current frame, a and b are nodes, dis f，a，b is the distance between a and b nodes in f frame, and w a，bis the weight of the edge with endpoints a, b. d f，a is the position data of person a in frame f, d f，b is the position data of person b in frame f.

[0324] (2-4): The nodes are established and the edges are connected by the above method, that is, a personnel relationship topology graph is constructed for each frame of video.

[0325] S3: For each frame in each video, motion information is extracted from the position area information of each person target with a unique identity number in the adjacent frames before and after, and the motion information is added to the personnel relationship topology graph as a node feature. The specific implementation method is:

[0326] (3-1): For the personnel relationship topology graph G f of the frame f, there is an adjacency matrix A f (n, n). The adjacency matrices of the adjacent two frames of personnel relationship topology graph are subtracted element by element to take the absolute value, and then added by row to obtain a feature value F ADJ (n) that measures the positional relationship change of each node with other personnel in the current frame:

[0327] F ADJ (n) = sum (|A f (n, n) - A f-1 (n, n) |, axis = 0) (10)

[0328] (3-2): For the node list in the personnel relationship topology graph G f of the frame f, a node motion change list loc f (n) is constructed with the difference between the center coordinates (x, y) of each node and the previous frame, and a node horizontal coordinate change list x f (n) is constructed with the horizontal coordinates x in it, and a node vertical coordinate change list y f (n) is constructed with the vertical coordinates y in it. The change amounts in the horizontal and vertical directions are synthesized, that is, the feature value F V (n) that records the motion change information of each node in the current frame:

[0329]

[0330]

[0331]

[0332]

[0333] wherein loc is the position information containing horizontal and vertical coordinates, that is, (x, y).

[0334] (3-3): The personnel relationship topology graph G corresponding to frame f f The node list in the frame f, loc f (n) The recorded information can be said to be the motion change amount, or it can be considered that the node moves from the previous frame position to the current frame position. Taking the two motions recorded in the current frame forward, the cosine value of the angle between the two motion vectors is calculated, that is, the change of the motion direction can be measured. Since the cosine function is a monotonically decreasing function in the interval [0, π], here the reciprocal +1 is taken to obtain a function with a value range of [0, 1] and a monotonically increasing function, and the value is taken as the characteristic value F of the recorded motion direction change information ANGLE (n):

[0335]

[0336] wherein

[0337]

[0338] Add node feature step: combine the above three motion features into node feature F:

[0339]

[0340] S4: Train the improved GCN network Sub-GCN with the constructed graph data;

[0341] (4-1): Compared with the traditional GCN, Sub-GCN focuses on the differences between different nodes in the same graph. Therefore, Sub-GCN has a different adjacent matrix calculation method compared with the traditional GCN:

[0342]

[0343] wherein: is the adjacent matrix after adding a self-loop to the original graph, is the corresponding degree matrix, and I is the identity matrix.

[0344] (4-2): Set the loss function: adopt the weighted binary cross-entropy loss function.

[0345]

[0346] wherein, is the binary cross-entropy loss function, β is the weight, which depends on the proportion of positive and negative samples, y i represents the true value, represents the probability that the network output prediction is a normal class.

[0347] Constructing network and training optimal model: 2 layers of Sub-GCN structure are adopted, Relu activation function is adopted between layers, Softmax function is applied after convolution of the last layer to obtain output probability, and optimal model is obtained through iterative training under different hyperparameters.

[0348] (4-3): Constructing network and training optimal model: 2 layers of Sub-GCN structure are adopted, Relu activation function is adopted, Softmax function is applied after convolution of the last layer to obtain output probability, and optimal model is obtained through iterative training under different hyperparameters.

[0349] S5: The divided test set is subjected to the same preprocessing step, feature extraction step and modeling step to construct a personnel relationship topology graph, the constructed graph data is input into the trained Sub-GCN network, and a recognition result is obtained.

[0350] S6: The recognition result is visualized and displayed on the original video frame.

[0351] Those skilled in the art know that, in addition to implementing the system, device and each module thereof provided by the present application in the form of pure computer readable program code, the same program can be realized by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for realizing various programs can also be considered as structures within the hardware component; the modules for realizing various functions can also be considered as both software programs for realizing methods and structures within hardware components.

[0352] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for modeling and detecting abnormal behavior based on constructing a graph of neighborhood relations of a person, characterized in that, Comprise: Step S1: target tracking is carried out on personnel, personnel number, frame sequence number of personnel appearance and position area coordinate information in each frame are obtained; Step S2: node adjacency matrix is established with each personnel target as node, the adjacency relationship between nodes is calculated, the coordinate distance of the current node and the adjacent node is taken as the adjacency edge weight, and a personnel adjacency relationship graph structure is constructed; Step S3: the position area coordinate change of each node is calculated, and the node characteristics are set; Step S4: network structure is constructed, graph data is input, and each frame detection result is output, whether each graph contains an abnormal node is detected, if there is, it is considered that there is an abnormal behavior in the frame, loss is calculated, and network parameters are updated; In the step S4: The Sub-GCN network structure is constructed, whether each frame contains an abnormal behavior is converted into whether each graph contains an abnormal node, whether each frame contains an abnormal behavior is supervised in the training process, graph data is input, each frame detection result is output, loss is calculated, and network parameters are updated; The adjacency matrix calculation method of Sub-GCN is as follows: wherein, adding a self-loop to the adjacency matrix of the original graph, is the corresponding degree matrix, is the identity matrix; A weighted binary cross entropy loss function is used: wherein, is a binary cross-entropy loss function, is a weight depending on the ratio of positive and negative samples, represents the true value, represents the probability that the network output predicts as normal class. A 2-layer Sub-GCN structure is used, a Relu activation function is used between layers, a Softmax function is used after convolution in the last layer to obtain output probability, and the optimal model is obtained by iterative training under different hyperparameters; The test set divided out is subjected to the same preprocessing step, feature extraction step and modeling step to construct a personnel relationship topology graph, the constructed graph data is input into the trained Sub-GCN network, and a recognition result is obtained.

2. The method of claim 1, wherein the method further comprises: In the step S1: The original video data is processed by using a target tracker to obtain a tracking information sequence corresponding to each video, wherein each information in the sequence is composed of a frame number , a personnel identity number , a left upper corner coordinate of a personnel identification frame , and a right lower corner coordinate . For each piece of information in the information sequence, calculate the coordinates in which the current person center is located : wherein, , , both represent the horizontal coordinate, , , both represent the vertical coordinate, represents the frame number, represents the person identity number. 3.The method of claim 1, wherein, In the step S2: For each sequence of tracking information of a video, the set of all person identity numbers contained in each frame The set of nodes in the topological graph to be constructed for each frame, with size ; For each video, the total number of frames is The calculated center coordinates of the personnel For the data, establish dimensions as matrix ; For each pair : For the elements in the matrix that have not been assigned values after the above calculation: Wherein: wherein, to find the nearest frame in which the person appears from the current frame, taking the position data of the person in that frame, denotes the sequence number of the frame found forward; to find the nearest frame in which the person appears from the current frame, taking the position data of the person in that frame, denotes the sequence number of the frame found backward; For each two nodes in each frame, the distance of the node center position is calculated, and the reciprocal of the distance is taken as the edge weight to establish a non-directed edge for the two nodes: wherein is the current frame, is a node, is in the frame distance between nodes, is the weight of an edge with endpoints; is the frame position data of a person in the frame, position data of a person in the frame, position data of a person in the frame, position data of a person in the frame, position data of a person in the frame. Nodes are established and edge connections are given, and a personnel relationship topology graph is constructed for each frame video.

4. The method of claim 1, wherein, In the step S3: The position area coordinate change of each node in adjacent frames before and after is calculated, the coordinate distance change amount of the current node and other nodes, the motion speed change amount of the node next time and the motion direction change amount are taken as the characteristics of each node; For the frame The corresponding personnel relationship topology , the adjacency matrix , the adjacency matrix of the adjacent two frames of personnel relationship topology is subtracted by element and the absolute value is taken, and then the row is added to obtain the characteristic value for measuring the position relationship change of each node and other personnel in the current frame : For the frame corresponding to the personnel relationship topology of the node list in the previous frame, the central coordinates of each node The difference between the nodes and the previous frame builds a node motion change list , and the horizontal coordinates in it respectively Build a node horizontal coordinate change list , the vertical coordinates in it Build a node vertical coordinate change list , and synthesize the change amount in the horizontal and vertical directions to record the motion change information of each node in the current frame. Characteristic value : wherein is position information comprising an abscissa and an ordinate ; For the frame The corresponding personnel relationship topology The node list in the frame, The recorded information is the motion change amount, or the motion vector of the node moving from the previous frame position to the current frame position; take the two motions recorded in the current frame forward, calculate the cosine value of the included angle of the two motion vectors to measure the change of the motion direction; since the cosine function is a monotonically decreasing function in the interval, take the opposite number here to get a function with a value range of and a monotonically increasing function, whose value is used as the characteristic value of the recorded motion direction change information : Wherein, Combining motion features into node features : 。 5. A system for modeling and detecting abnormal behavior based on constructing a personnel adjacency graph, characterized in that, Comprise: Module M1: target tracking is carried out on personnel, personnel number, frame sequence number of personnel appearance and position area coordinate information in each frame are obtained; Module M2: node adjacency matrix is established with each personnel target as node, the adjacency relationship between nodes is calculated, the coordinate distance of the current node and the adjacent node is taken as the adjacency edge weight, and a personnel adjacency relationship graph structure is constructed; Module M3: the position area coordinate change of each node is calculated, and the node characteristics are set; Module M4: network structure is constructed, graph data is input, and each frame detection result is output, whether each graph contains an abnormal node is detected, if there is, it is considered that there is an abnormal behavior in the frame, loss is calculated, and network parameters are updated; In the module M4: The Sub-GCN network structure is constructed, detection of whether an abnormal behavior is contained in each frame is converted into detection of whether an abnormal node is contained in each graph, a training process supervises whether an abnormal behavior is contained in each frame, graph data are input, a detection result of each frame is output, a loss is calculated, and network parameters are updated; The Sub-GCN adjacency matrix calculation method is as follows: wherein, adding a self-loop to the adjacency matrix of the original graph, is the corresponding degree matrix, is the identity matrix; A weighted binary cross-entropy loss function is used: wherein, is a binary cross-entropy loss function, is a weight depending on the ratio of positive and negative samples, represents the true value, represents the probability that the network outputs a prediction of the normal class; A 2-layer Sub-GCN structure is used, a Relu activation function is used between layers, a Softmax function is used after convolution in the last layer to obtain an output probability, and an optimal model is obtained through iterative training under different hyperparameters. The divided test set is subjected to the same preprocessing step, feature extraction step and modeling step to construct a personnel relationship topological graph, the constructed graph data are input into the trained Sub-GCN network, and a recognition result is obtained.

6. The system for abnormal behavior modeling and detection based on the graph of the build staff neighbors of claim 5, wherein, In the module M1: The original video data is processed by using a target tracker to obtain a tracking information sequence corresponding to each video, wherein each information in the sequence is composed of a frame number , a personnel identity number , a left upper corner coordinate of a personnel recognition frame , and a right lower corner coordinate . For each piece of information in the information sequence, calculate the coordinates in which the current person center is located : wherein, , , all represent the horizontal coordinate, , , all represent the vertical coordinate, represents the frame number, represents the person identity number.

7. The system for abnormal behavior modeling and detection based on the graph of the build staff neighbors of claim 5, wherein, In the module M2: For each sequence of tracking information of a video, the set of all person identity numbers contained in each frame The set of nodes in the topological graph to be constructed for each frame, with size ; For each video, the total number of frames is The calculated center coordinates of the personnel For the data, establish dimensions as matrix ; For each pair : For elements in the matrix that have not been assigned values after the above calculation: Wherein: wherein, to find the nearest frame in which the person appears from the current frame, taking the position data of the person in that frame, denotes the sequence number of the frame found forward; to find the nearest frame in which the person appears from the current frame, taking the position data of the person in that frame, denotes the sequence number of the frame found backward; For each two nodes in each frame, the distance of the node center position is calculated, and a reciprocal of the distance is used as an edge weight to establish a non-directed edge for the two nodes. wherein is the current frame, is a node, is in the frame distance between nodes, is the weight of an edge with endpoints; is the frame position data of a person in the frame position data of a person in the frame position data of a person Nodes are established and edge connections are given, and a personnel relationship topological graph is constructed for each frame of video.

8. The system for abnormal behavior modeling and detection based on the graph of the build staff neighbors of claim 5, wherein, In the module M3: The position region coordinate change of each node in adjacent frames is calculated, and the coordinate distance change amount of the current node and other nodes, the node next time motion speed change amount and the motion direction change amount are used as the node features; For the frame The corresponding personnel relationship topology , the adjacency matrix , the adjacency matrix of the adjacent two frames of personnel relationship topology is subtracted by element and the absolute value is taken, and then the row is added to obtain the characteristic value for measuring the position relationship change of each node and other personnel in the current frame : For the frame corresponding to the personnel relationship topology of the node list in each node center coordinates with the previous frame difference value to build a node motion change list , and respectively in the horizontal coordinate build a node horizontal coordinate change list , in which the vertical coordinates build a node vertical coordinate change list , the amount of change in the horizontal and vertical directions is synthesized, and the feature value recording the motion change information of each node in the current frame : wherein is position information comprising an abscissa and an ordinate ; For frame The corresponding personnel relationship topology The node list in the frame, The recorded information is the motion change amount, or the motion vector of the node moving from the previous frame position to the current frame position; take the two motions recorded in the current frame forward, calculate the cosine value of the included angle of the two motion vectors to measure the change of the motion direction; since the cosine function is a monotonically decreasing function in the interval, here we take the opposite number to get a function with a value range of and a monotonically increasing function, whose value is used as the characteristic value of the recorded motion direction change information : Wherein, Combining motion features into node features : 。

Images