Video description method of subway passengers' abnormal behavior enhanced by skeleton key point knowledge
By constructing a knowledge graph of abnormal subway passenger behavior and a knowledge graph of skeletal key points, and combining a spatiotemporal graph convolutional neural network and a cross-modal attention model, the problem of insufficient targeting and accuracy in the description of abnormal subway passenger behavior in existing technologies is solved, and natural language descriptions for daily subway operation management are generated.
Patent Information
- Application Number
- CN202311739530.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-18
AI Technical Summary
Existing video description technologies lack specificity in describing abnormal behavior of subway passengers in real monitoring scenarios, and their accuracy in identifying and describing diverse abnormal behaviors is insufficient, making it difficult to meet the needs of daily subway operation and management.
By constructing a knowledge graph of abnormal behavior of subway passengers and a knowledge graph of skeletal key points, and combining a spatiotemporal graph convolutional neural network and a cross-modal attention knowledge enhancement model, natural language descriptions are generated to improve the accuracy of abnormal behavior video descriptions.
It generates accurate natural language descriptions that can be directly applied to the daily operation and management decisions of the subway, improves the accuracy and usability of video descriptions of abnormal passenger behavior, and promotes the practical application of video description technology.
Smart Images

Figure CN117557945B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video description, and in particular to a method for describing abnormal behavior videos of subway passengers enhanced with skeleton key point knowledge. Background Art
[0002] At present, although a series of theoretical progress has been made in visual description methods designed based on external knowledge and public data sets, there are few studies on video description for real monitoring scenarios. The generated content lacks the pertinence of subway scene applications and is difficult to meet the needs of video description of abnormal behaviors of subway passengers in real monitoring scenarios. In addition, the existing video description technology has deficiencies in the identification of diverse abnormal behaviors of passengers, the accuracy of description, and usability. The above problems have seriously restricted the implementation and promotion of video description technology. In response to the above two problems, the present invention designs a video description method for abnormal behaviors of subway passengers enhanced with skeleton key point knowledge. The present invention can improve the accuracy of identifying diverse abnormal behaviors of passengers by constructing a knowledge graph of abnormal behavior joints of subway passengers and introducing a cross-modal attention knowledge enhancement model. At the same time, on the basis of improving the accuracy of identifying abnormal behaviors of subway passengers, the present invention can generate natural language descriptions that are directly applied to daily operation and management decisions of subways.
[0003] At present, the research on video description methods based on external knowledge can be divided into the following two categories: (1) Video description model based on scene graph: In the model encoding stage, scene graph is introduced to encode fine-grained visual information to improve the diversity of description sentences; (2) Video description model based on knowledge graph: In the model decoding stage, knowledge is extracted from external knowledge base to supplement the generation of description text to generate more natural description sentences. However, the natural language description sentences generated by these two methods lack the application specificity of subway scenes, and the description expression cannot adapt to the actual needs of identifying passengers' diverse abnormal behaviors. Summary of the invention
[0004] The purpose of the present invention is to provide a method for describing subway passengers' abnormal behavior videos with skeleton key point knowledge enhancement, which can be directly applied to the natural language description of daily subway operation management decisions and improve the accuracy of the description of subway passengers' abnormal behavior.
[0005] To achieve the above object, the present invention provides a method for describing a video of abnormal behavior of subway passengers enhanced with skeleton key point knowledge, comprising:
[0006] Get videos of abnormal subway behavior;
[0007] Segmenting the subway abnormal behavior video to obtain multiple abnormal behavior video clips, and extracting the skeletal joint points of each passenger in each abnormal behavior video clip;
[0008] Constructing an abnormal behavior knowledge graph based on subway domain knowledge; the abnormal behavior knowledge graph includes multiple abnormal behavior categories and multiple skeleton joints, each abnormal behavior category is represented by multiple skeleton joints, and each skeleton joint includes multiple attribute features;
[0009] According to the attribute characteristics of the skeletal joint points of each passenger in each abnormal behavior video clip, the skeletal key points of each passenger are screened out from each abnormal behavior video clip, and the visual feature vector of the skeletal key points of each passenger in each abnormal behavior video clip is determined;
[0010] A skeleton key point knowledge graph is constructed based on the abnormal behavior knowledge graph; the skeleton key point knowledge graph includes multiple abnormal behavior categories, skeleton key points of each passenger and skeleton key point knowledge vectors; the skeleton key point knowledge vector includes multiple attribute features of each skeleton key point;
[0011] For any abnormal behavior video clip, according to the skeleton key point visual feature vector and skeleton key point knowledge vector of each passenger in the abnormal behavior video clip, a spatiotemporal graph convolutional neural network is used to determine the abnormal behavior visual representation of the abnormal behavior video clip;
[0012] According to the skeleton key point visual feature vector and the skeleton key point knowledge vector of each passenger in the abnormal behavior video clip, a cross-modal attention knowledge enhancement model is used to determine the knowledge perception context vector of each time step in the abnormal behavior video clip;
[0013] According to the abnormal behavior visual representation of the abnormal behavior video clip and the knowledge-aware context vector of each time step in the abnormal behavior video clip, an abnormal behavior video description generation model is adopted to generate a description sentence of the abnormal behavior video clip.
[0014] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention constructs an abnormal behavior knowledge graph and a skeleton key point knowledge graph, while taking into account the knowledge graphs of two levels, namely, the abnormal behavior of passengers and the skeleton key points of passengers in the surveillance video clips. Compared with a single-level knowledge graph, it can more specifically condense the dynamic semantic association relationship between the skeleton key points of various abnormal behaviors of passengers, and provide more accurate knowledge of abnormal behavior of subway passengers for the abnormal behavior video description generation model. Through the spatiotemporal graph convolutional neural network, the cross-modal attention knowledge enhancement model and the abnormal behavior video description generation model, a finer-grained joint representation of key visual information and natural language is learned, and natural language sentences that accurately describe the abnormal behavior of passengers are generated, thereby improving the accuracy of the description of abnormal behavior of subway passengers. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0016] Figure 1 A flowchart of the method for describing abnormal behavior of subway passengers by video enhanced with skeleton key point knowledge provided by the present invention;
[0017] Figure 2 Schematic diagram of the process of describing the video of abnormal behavior of subway passengers;
[0018] Figure 3 This is a schematic diagram of the processing of the video segmentation algorithm BSN++;
[0019] Figure 4 This is a schematic diagram of the processing of the human skeleton joint point extraction algorithm OpenPose-slim;
[0020] Figure 5 This is a schematic diagram of abnormal behavior of subway passengers in the ConceptNet5.5 dataset;
[0021] Figure 6 Schematic diagram of abnormal behavior knowledge graph;
[0022] Figure 7 This is a schematic diagram of the knowledge graph of skeleton key points;
[0023] Figure 8 Schematic diagram of the visual representation process of abnormal behavior based on spatiotemporal graph convolutional neural network;
[0024] Fig. 9 Schematic diagram of abnormal behavior identification process enhanced by cross-modal attention knowledge;
[0025] Fig.10 Schematic diagram of a model for generating descriptions of abnormal behavior videos;
[0026] Fig.11 An example diagram of a video description of abnormal behavior of subway passengers generated by the present invention;
[0027] Fig.12 An example diagram of a video description of abnormal behavior of subway passengers generated by the present invention;
[0028] Fig.13 A schematic diagram of the subway passenger abnormal behavior video description system enhanced with skeleton key point knowledge provided by the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] The purpose of the present invention is to provide a method for describing abnormal behaviors of subway passengers in videos enhanced with skeleton key point knowledge. By constructing an abnormal behavior knowledge graph and a skeleton key point knowledge graph and introducing a cross-modal attention knowledge enhancement model, it is possible to generate a natural language description that can be directly applied to daily operation and management decisions of the subway on the basis of improving the accuracy of the description of abnormal behaviors of subway passengers.
[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0032] Embodiment 1
[0033] like Figure 1 As shown, this embodiment provides a method for describing subway passengers' abnormal behavior videos enhanced with skeleton key point knowledge, including: abnormal behavior video segmentation and passenger skeleton joint point feature extraction, subway passenger abnormal behavior knowledge graph and skeleton key point knowledge graph construction, and subway passenger abnormal behavior video description enhanced with skeleton key point knowledge. Figure 2 As shown, the specific steps are as follows:
[0034] Step 100: Obtain subway abnormal behavior video.
[0035] Step 200: Segment the subway abnormal behavior video to obtain multiple abnormal behavior video clips, and extract the skeletal joints of each passenger in each abnormal behavior video clip. The abnormal behavior video can be segmented according to the temporal behavior of the passengers, and then the spatiotemporal features of the skeletal joints of the passengers in each abnormal behavior video clip can be extracted.
[0036] Specifically, the video segmentation algorithm BSN++ is used to segment the subway abnormal behavior video to obtain multiple abnormal behavior video clips. In this embodiment, the video segmentation algorithm BSN++ is used to segment the abnormal behavior video clip set of passengers in the subway abnormal behavior video. in, and Represents abnormal behavior video clips The start time and end time of the abnormal behavior video clip are n, n is the number of the abnormal behavior video clip, N is the total number of abnormal behavior video clips, n∈N. It can reduce the interference of irrelevant video clips on passenger behavior recognition and provide effective data information for the step of extracting features of passenger skeleton joints.
[0037] like Figure 3 As shown in the figure, the video segmentation algorithm BSN++ includes a basic module, a complementary boundary generator and a nomination relationship modeling. Among them, the complementary boundary generator is connected to the basic module to generate a boundary map representing boundary information, and the nomination relationship modeling is connected to the basic module to generate a confidence map for confidence evaluation. Finally, the result is output according to the boundary map and the confidence map.
[0038] The human skeleton joint point extraction algorithm OpenPose-slim is used to extract the skeleton joint points of each passenger in each abnormal behavior video clip. In this embodiment, each segmented abnormal behavior video clip is input into the human skeleton joint point extraction algorithm OpenPose-slim to obtain the spatiotemporal sequence features of the skeleton joint points of the passengers in the abnormal behavior video clip. Includes the image number, passenger number, and the locations of all skeletal joints, where: Video clips of abnormal behavior The skeleton joint points of the mth passenger in the image of the tth frame (i.e. time step t), k represents the number of the bone joint point, Video clips of abnormal behavior The coordinates of the kth skeletal joint point of the mth passenger in the tth frame image. The OpenPose-slim algorithm for extracting human skeletal joint points is used to extract the skeletal joint point features of passengers in real video clips of high-density passenger flow in subways, which are used as the entity objects of the skeletal key point knowledge graph in step 500 and the visual feature input of the cross-modal attention knowledge enhancement model in step 700.
[0039] like Figure 4 As shown in the figure, the human skeleton joint point extraction algorithm OpenPose-slim includes a sequentially connected spatial encoding stage, a temporal encoding-decoding stage, and a spatial decoding stage. Among them, the spatial encoding stage includes a sequentially connected convolution layer, a pooling layer, and a convolution layer. The spatial decoding stage includes a sequentially connected deconvolution layer, a depooling layer, and a deconvolution layer.
[0040] Step 300: construct an abnormal behavior knowledge graph based on subway domain knowledge. The abnormal behavior knowledge graph includes multiple abnormal behavior categories and multiple skeleton joints, each abnormal behavior category is represented by multiple skeleton joints, and each skeleton joint includes multiple attribute features.
[0041] In this embodiment, the abnormal behavior knowledge graph includes 8 types of abnormal behaviors, and the behavior of each passenger is represented by 18 skeletal joints, each of which contains 5 types of attribute features.
[0042] Specifically, based on subway domain knowledge and surveillance video feature analysis, the abnormal behaviors of subway passengers in the ConceptNet5.5 dataset and the Subway dataset are integrated, and the abnormal behaviors of subway passengers and passenger skeleton joints are used to construct an abnormal behavior knowledge graph.
[0043] Figure 5 Figure 2 shows the abnormal behavior of subway passengers in the ConceptNet5.5 dataset.
[0044] Abnormal behaviors of subway passengers in the Subway dataset include: climbing over to evade fares, running up and down stairs, going the wrong way on escalators, crawling under to evade fares, leaning out of escalators, slipping on handrails, following others to evade fares, and falling down stairs.
[0045] There are multiple categories of abnormal behaviors, namely: climbing over to evade the fare, crawling under to evade the fare, following to evade the fare, falling down the stairs, running up the stairs, going up the escalator in the opposite direction, leaning out of the escalator and slipping on the handrail.
[0046] The 18 skeletal joints of each passenger are: left hand, right hand, left shoulder, right shoulder, left elbow, right elbow, left knee, right knee, left foot, right foot, left waist, right waist, neck, nose, left eye, right eye, left ear and right ear.
[0047] The multiple attribute features of each bone joint point are: the angle f formed by the bone joint points 1,k , the angle change between the bone joints f 2,k , the acceleration of the bone joint point f 3,k , the distance between the bone joint point and the center of gravity of the skeleton f 4,k and the offset f of the bone joint point 5,k Correspondingly, the feature of each bone joint point is expressed as: F k =[f 1,k ,f 2,k ,f 3,k ,f 4,k ,f 5,k ], where k represents the number of the bone joint point.
[0048] like Figure 6 Shown is the abnormal behavior knowledge graph constructed by the present invention.
[0049] Step 400: According to the attribute characteristics of the skeletal joint points of each passenger in each abnormal behavior video clip, the skeletal key points of each passenger are screened out from each abnormal behavior video clip, and the visual feature vector of the skeletal key points of each passenger in each abnormal behavior video clip is determined.
[0050] Specifically, for any passenger, the difference between each attribute feature of each skeletal joint point of the passenger and the threshold of each attribute feature is calculated respectively. After the difference is sorted from large to small, 10 skeletal joint points with the highest spatiotemporal correlation are selected from each abnormal behavior video clip as skeletal key points.
[0051] Step 500: Construct a skeleton key point knowledge graph based on the abnormal behavior knowledge graph. The skeleton key point knowledge graph includes multiple abnormal behavior categories, skeleton key points of each passenger, and skeleton key point knowledge vectors. The skeleton key point knowledge vector includes multiple attribute features of each skeleton key point. Figure 7 Shown is the skeleton key point knowledge graph constructed by the present invention.
[0052] Specifically, the visual features of the passenger's skeleton key points are used as queries, and the attribute features of the corresponding skeleton joint points are extracted from the abnormal behavior knowledge graph, and then the skeleton key point knowledge graph is constructed. Attribute characteristics of bone key points Connect them to generate the skeleton key point knowledge graph G = {V, E} corresponding to the abnormal behavior video clip, where V = {v a} represents a node set, E = {e ab} represents an edge set, a and b are node numbers, a≠b.
[0053] In the present invention, the abnormal behavior knowledge graph constructed in step 300 and the skeleton key point knowledge graph constructed in step 500 can guide the dynamic semantic association between the skeleton key points that characterize various abnormal behaviors of subway passengers based on the abnormal behavior knowledge. At the same time, the knowledge of abnormal behaviors of passengers and skeleton key points in real surveillance video clips is taken into account. Among them, the abnormal behavior knowledge graph can quantify the knowledge of diverse abnormal behaviors of passengers by measuring the mutual relationship between extracted entities, and the skeleton key point knowledge graph can further condense the dynamic semantic association relationship between the skeleton key points of various abnormal behaviors of passengers.
[0054] Step 600: For any abnormal behavior video clip, based on the skeleton key point visual feature vector and the skeleton key point knowledge vector of each passenger in the abnormal behavior video clip, a spatiotemporal graph convolutional neural network is used to determine the abnormal behavior visual representation of the abnormal behavior video clip.
[0055] Specifically, Figure 8 As shown in the figure, the Skip-Gram model is used to vectorize the abnormal behavior knowledge graph and the visual feature vector of the passenger's skeleton key points The knowledge vector of the passenger's skeleton key points After concatenation, a spatiotemporal graph convolutional neural network is used to learn the visual representation of abnormal behaviors guided by the knowledge of skeletal key points.
[0056] The spatiotemporal graph convolutional neural network uses the following formula to determine abnormal behavior video clips Visual representation of abnormal behavior:
[0057]
[0058]
[0059] Among them, X is the input of the spatiotemporal graph convolutional neural network, Video clips of abnormal behavior The visual feature vector of the skeleton key points of each passenger, Video clips of abnormal behavior The skeleton key point knowledge vector of each passenger in, Video clips of abnormal behavior The skeleton key points of the mth passenger in the image at time step t, B is a vector of all 1s, S is the node feature embedding matrix (the matrix representing the edges or connections between nodes in the forward propagation equation), and Z is the spatiotemporal graph convolutional neural network output matrix, i.e., the abnormal behavior video clip Visual representation of abnormal behavior, including the connection relationship between nodes, X h is the feature of X in the hth layer of the spatiotemporal graph convolutional neural network, is the degree matrix, is the adjacency matrix of the undirected graph with self-connection added, Θ g is the filter parameter matrix, represents the Kronecker product and ⊙ represents the dot product.
[0060] Step 700: Based on the skeletal key point visual feature vector and skeletal key point knowledge vector of each passenger in the abnormal behavior video clip, a cross-modal attention knowledge enhancement model is used to determine the knowledge perception context vector of each time step in the abnormal behavior video clip.
[0061] Specifically, based on the cross-modal attention knowledge enhancement model, the visual context vector and the knowledge context vector are integrated into the knowledge enhancement model of the decoder, and a knowledge-aware context vector is learned through decoder update. Fig. 9 Shown is a schematic diagram of the abnormal behavior identification process of the cross-modal attention knowledge enhancement model.
[0062] The cross-modal attention knowledge enhancement model uses the following formula to determine abnormal behavior video clips The knowledge-aware context vector at time step t in :
[0063]
[0064]
[0065]
[0066] θ(u w ,r,u n ) = f mlp (TransE(u w ,r,u n ));
[0067] in, Video clips of abnormal behavior The knowledge-aware context vector at time step t, i.e., the knowledge edge attention at time step t, U sub is a subgraph generated by the subway passenger abnormal behavior input sequence. The subway passenger abnormal behavior input sequence includes a visual context vector and a knowledge context vector. The visual context vector is determined by the visual feature vector of the skeleton key points, and the knowledge context vector is determined by the knowledge vector of the skeleton key points. u is U sub The nodes in U sub The set of edges containing node u, u w and u n is the node feature embedding generated by the convolutional neural network, and w≠n, w represents the number of the knowledge triple, r is the connection relationship matrix between nodes, α(u) is the importance of the hidden state of the subway passenger abnormal behavior input sequence for determining the next hidden state, γ(·) is the soft alignment of the parameter conversion multi-layer perception calculation, s n,t Abnormal behavior video clips in cross-modal attention knowledge enhancement models The hidden state at time step t, s n,t-1 Abnormal behavior video clips in cross-modal attention knowledge enhancement models The hidden state at time step t-1, y n,t-1 Video clips of abnormal behavior The word generated at time step t-1, e(y n,t-1 ) is y n,t-1 The embedding of n,t Video clips of abnormal behavior The context state at time step t is obtained by feature concatenation, f de (·) is the decoder, f mlp (·) is a multi-layer perceptron, and TransE(·) represents a method of modeling relations by interpreting them as transformations of low-dimensional embedding operations on entities.
[0068] Step 800: Based on the abnormal behavior visual representation of the abnormal behavior video clip and the knowledge-aware context vector of each time step in the abnormal behavior video clip, an abnormal behavior video description generation model is used to generate a description sentence of the abnormal behavior video clip.
[0069] Specifically, a video description generation model is used to learn a finer-grained joint representation of vision and natural language, and natural language sentences are obtained for the video description of abnormal behavior of subway passengers, such as Fig.10 The abnormal behavior video description generation model uses the following formula to generate abnormal behavior video clips. Description statement:
[0070]
[0071]
[0072]
[0073] in, is the input of the abnormal behavior video description generation model, and Z is the abnormal behavior video clip Visual representation of abnormal behavior, Video clips of abnormal behavior The knowledge-aware context vector at time step t in , Video clips of abnormal behavior in the attention model The hidden state at time step t-1, GRU lan () is a video description generation model, Video clips of abnormal behavior in a video description generation model The hidden state of GRU at time step t lan () Through the convolution operation according to and get W p Learning parameters for the video description generation model, b p The bias of the video description generation model, y n,t Video clips of abnormal behavior The word generated at time step t, p(y n,t ) is a video clip of abnormal behavior The distribution of words generated at time step t in p(y n,t ) Identify video clips of abnormal behavior Description statement of p(y n,t ) directly affects the accuracy of the descriptive statement.
[0074] In the above steps 600 to 800, the visual representation of abnormal behavior guided by the knowledge graph of skeleton key points and the abnormal behavior knowledge enhanced by cross-modal attention knowledge are integrated. First, after the visual feature vector of the passenger skeleton key points is concatenated with the passenger key point knowledge vector, the spatiotemporal graph convolutional neural network is used to learn the visual representation of abnormal behavior guided by the skeleton key point knowledge, and the video description generation model is focused on the key abnormal behavior visual information in the video clip through the visual attention mechanism. Secondly, based on the cross-modal attention knowledge enhancement model, the visual context vector and the knowledge context vector are integrated into the knowledge enhancement model of the decoder, and a knowledge-aware context vector is learned through the decoder update. Finally, the output of the skeleton key point representation based on the spatiotemporal graph convolutional neural network and the cross-modal attention skeleton key point knowledge enhancement model are collaboratively input into the abnormal behavior video description generation model, and a video description sentence for the abnormal behavior of subway passengers is obtained by learning a finer-grained joint representation of vision and natural language.
[0075] In order to verify the effectiveness of the present invention, a subway passenger abnormal behavior video description dataset is first constructed based on real surveillance video data. Specifically, the 2,000 passenger abnormal behavior videos collected include 8 types of subway passenger abnormal behaviors, including climbing over to evade tickets, drilling under to evade tickets, following to evade tickets, falling down stairs, running on stairs, going against the escalator, leaning out of the escalator, and slipping on the handrail, with an average of 250 videos in each category; the behavior of each passenger is characterized by 18 skeletal joints, including left hand, right hand, left shoulder, right shoulder, left elbow, right elbow, left knee, right knee, left foot, right foot, left waist, right waist, neck, nose, left eye, right eye, left ear, and right ear; in addition, the 5 types of attribute features contained in each skeletal joint are the angle formed by the joint points, the change in the angle between the joint points, the acceleration of the joint points, the distance between the joint points and the center of gravity of the skeleton, and the offset of the joint points. Each passenger abnormal behavior video corresponds to a natural language description sentence. In particular, the natural language description sentence is a sentence that contains information such as passenger characteristics, the location of the behavior, and the category of abnormal behavior and is expressed coherently. 1600, 200 and 200 video and natural language sentence pairs were extracted from the constructed subway passenger abnormal behavior video description dataset for model training, testing and verification. The experimental results show that the present invention can improve the accuracy and usability of the subway passenger abnormal behavior video description content for real surveillance videos by constructing a subway passenger abnormal behavior-skeleton key point knowledge graph and introducing a cross-modal attention skeleton key point knowledge enhanced abnormal behavior identification model, which is helpful to promote the implementation and promotion of video description technology in the field of daily subway operation management.
[0076] like Fig.11 An example of a video description of abnormal behavior of subway passengers generated by the present invention is shown in FIG. Fig.12Another example of a video description of abnormal behavior of subway passengers generated by the present invention.
[0077] In summary, compared with the prior art, the present invention has the following advantages:
[0078] (1) The present invention can effectively overcome the problem that video description technology is difficult to meet the needs of video description of abnormal behavior of subway passengers in real monitoring scenarios, and the accuracy and usability of identifying and describing diverse abnormal behaviors of passengers are insufficient. It can not only improve the accuracy of the description of abnormal behavior of subway passengers, but also generate natural language descriptions that can be directly applied to daily subway operation and management decisions, promoting the application of video description technology.
[0079] (2) The present invention takes into account both the knowledge graphs of abnormal passenger behaviors and the skeleton key points in the surveillance video clips. Compared with the knowledge graphs of a single level, it can more specifically condense the dynamic semantic association relationship between the skeleton key points of various abnormal behaviors of passengers. It can provide more accurate knowledge of abnormal behavior of subway passengers for the generation model of abnormal behavior video description of subway passengers.
[0080] Embodiment 2
[0081] In order to execute the method corresponding to the above-mentioned embodiment 1 to achieve the corresponding functions and technical effects, a subway passenger abnormal behavior video description system enhanced with skeleton key point knowledge is provided below.
[0082] like Fig.13 As shown, the subway passenger abnormal behavior video description system enhanced with skeleton key point knowledge provided in this embodiment includes: a video acquisition module 21, a skeleton joint point extraction module 22, an abnormal behavior knowledge graph construction module 23, a skeleton key point screening module 24, a skeleton key point knowledge graph construction module 25, an abnormal behavior visual representation module 26, an abnormal behavior identification module 27 and an abnormal behavior description module 28.
[0083] The video acquisition module 21 is used to acquire subway abnormal behavior videos.
[0084] The skeleton joint point extraction module 22 is used to segment the subway abnormal behavior video to obtain multiple abnormal behavior video clips, and extract the skeleton joint points of each passenger in each abnormal behavior video clip.
[0085] Specifically, the skeleton joint point extraction module 22 is composed of a video segmentation algorithm BSN++ and a human skeleton joint point extraction algorithm OpenPose-slim connected in series.
[0086] The abnormal behavior knowledge graph construction module 23 is used to construct an abnormal behavior knowledge graph based on subway domain knowledge. The abnormal behavior knowledge graph includes multiple abnormal behavior categories and multiple skeleton joints, each abnormal behavior category is represented by multiple skeleton joints, and each skeleton joint includes multiple attribute features.
[0087] The skeleton key point screening module 24 is used to screen out the skeleton key points of each passenger from each abnormal behavior video clip according to the attribute characteristics of the skeleton joint points of each passenger in each abnormal behavior video clip, and determine the skeleton key point visual feature vector of each passenger in each abnormal behavior video clip.
[0088] The skeleton key point knowledge graph construction module 25 is used to construct a skeleton key point knowledge graph based on the abnormal behavior knowledge graph. The skeleton key point knowledge graph includes multiple abnormal behavior categories, skeleton key points of each passenger and skeleton key point knowledge vectors. The skeleton key point knowledge vector includes multiple attribute features of each skeleton key point.
[0089] The abnormal behavior visual representation module 26 is used to determine the abnormal behavior visual representation of any abnormal behavior video clip by using a spatiotemporal graph convolutional neural network based on the skeletal key point visual feature vector and skeletal key point knowledge vector of each passenger in the abnormal behavior video clip.
[0090] The abnormal behavior identification module 27 is used to determine the knowledge perception context vector of each time step in the abnormal behavior video clip based on the skeleton key point visual feature vector and the skeleton key point knowledge vector of each passenger in the abnormal behavior video clip, using a cross-modal attention knowledge enhancement model.
[0091] The abnormal behavior description module 28 is used to generate a description sentence of the abnormal behavior video clip using an abnormal behavior video description generation model according to the abnormal behavior visual representation of the abnormal behavior video clip and the knowledge perception context vector of each time step in the abnormal behavior video clip.
[0092] Compared with the prior art, the subway passenger abnormal behavior video description system enhanced with skeleton key point knowledge provided in this embodiment has the same beneficial effects as the subway passenger abnormal behavior video description method enhanced with skeleton key point knowledge provided in Example 1, which will not be repeated here.
[0093] Embodiment 3
[0094] This embodiment provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for describing abnormal behavior of subway passengers by video enhanced with skeleton key point knowledge of embodiment one.
[0095] Optionally, the above-mentioned electronic device may be a server.
[0096] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for describing abnormal behavior of subway passengers by video with enhanced skeleton key point knowledge of embodiment one.
[0097] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0098] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for describing abnormal behavior videos of subway passengers enhanced with skeleton key point knowledge, characterized in that: The method for describing subway passenger abnormal behavior videos enhanced with skeleton key point knowledge includes: Get videos of abnormal subway behavior; Segmenting the subway abnormal behavior video to obtain multiple abnormal behavior video clips, and extracting the skeletal joint points of each passenger in each abnormal behavior video clip; Constructing an abnormal behavior knowledge graph based on subway domain knowledge; the abnormal behavior knowledge graph includes multiple abnormal behavior categories and multiple skeleton joints, each abnormal behavior category is represented by multiple skeleton joints, and each skeleton joint includes multiple attribute features; According to the attribute characteristics of the skeletal joint points of each passenger in each abnormal behavior video clip, the skeletal key points of each passenger are screened out from each abnormal behavior video clip, and the visual feature vector of the skeletal key points of each passenger in each abnormal behavior video clip is determined; A skeleton key point knowledge graph is constructed based on the abnormal behavior knowledge graph; the skeleton key point knowledge graph includes multiple abnormal behavior categories, skeleton key points of each passenger and skeleton key point knowledge vectors; the skeleton key point knowledge vector includes multiple attribute features of each skeleton key point; For any abnormal behavior video clip, according to the skeleton key point visual feature vector and skeleton key point knowledge vector of each passenger in the abnormal behavior video clip, a spatiotemporal graph convolutional neural network is used to determine the abnormal behavior visual representation of the abnormal behavior video clip; The spatiotemporal graph convolutional neural network uses the following formula to determine abnormal behavior video clips Visual representation of abnormal behavior: Among them, X is the input of the spatiotemporal graph convolutional neural network, Video clips of abnormal behavior The visual feature vector of the skeleton key points of each passenger, Video clips of abnormal behavior The skeleton key point knowledge vector of each passenger in, Video clips of abnormal behavior The skeleton key points of the mth passenger in the image at time step t, B is a vector of all 1s, S is the node feature embedding matrix, and Z is the abnormal behavior video clip Visual representation of abnormal behavior, X h is the feature of X in the hth layer of the spatiotemporal graph convolutional neural network, is the degree matrix, is the adjacency matrix of the undirected graph with self-connection added, Θ g is the filter parameter matrix, represents the Kronecker product, ⊙ represents the dot product; According to the skeletal key point visual feature vector and skeletal key point knowledge vector of each passenger in the abnormal behavior video clip, a cross-modal attention knowledge enhancement model is used to determine the knowledge perception context vector of each time step in the abnormal behavior video clip; The cross-modal attention knowledge enhancement model uses the following formula to determine abnormal behavior video clips The knowledge-aware context vector at time step t in : θ(in w ,r,u n )=f mlp (TransE(u w ,r,u n )); in, Video clips of abnormal behavior The knowledge-aware context vector at time step t, U sub is a subgraph generated by the subway passenger abnormal behavior input sequence. The subway passenger abnormal behavior input sequence includes a visual context vector and a knowledge context vector. The visual context vector is determined by the visual feature vector of the skeleton key points, and the knowledge context vector is determined by the knowledge vector of the skeleton key points. u is U sub The nodes in U sub The set of edges containing node u, u w and u n is the node feature embedding generated by the convolutional neural network, w≠n, r is the connection relationship matrix between nodes, α(u) is the importance of the hidden state of the subway passenger abnormal behavior input sequence for determining the next hidden state, γ(·) is the soft alignment of the parameter conversion into multi-layer perception calculation, s n,t Abnormal behavior video clips in cross-modal attention knowledge enhancement models The hidden state at time step t, s n,t-1 Abnormal behavior video clips in cross-modal attention knowledge enhancement models The hidden state at time step t-1, y n,t-1 Video clips of abnormal behavior The word generated at time step t-1, e(y n,t-1 ) is y n,t-1 The embedding of n,t Video clips of abnormal behavior The context state at time step t, f de (·) is the decoder, f mlp (·) is a multi-layer perceptron, and TransE(·) represents a method of modeling relations by interpreting them as transformations of low-dimensional embedding operations on entities; Generate a description sentence of the abnormal behavior video clip using an abnormal behavior video description generation model according to the abnormal behavior visual representation of the abnormal behavior video clip and the knowledge-aware context vector of each time step in the abnormal behavior video clip; The abnormal behavior video description generation model uses the following formula to generate abnormal behavior video clips Description statement: in, The input for the model to generate descriptions of abnormal behavior videos, Attention Model for Video Caption Generation The hidden state at time step t-1, GRU lan () is a video description generation model, Video clips of abnormal behavior in a video description generation model The hidden state at time step t, W p Learning parameters for the video description generation model, b p The bias of the video description generation model, y n,t Video clips of abnormal behavior The word generated at time step t, p(y n,t ) is a video clip of abnormal behavior The distribution of words generated at time step t in p(y n,t ) Identify video clips of abnormal behavior Description statement.
2. The method for describing subway passengers' abnormal behavior videos enhanced with skeleton key point knowledge according to claim 1 is characterized in that: The video segmentation algorithm BSN++ is used to segment the subway abnormal behavior video to obtain multiple abnormal behavior video clips.
3. The method for describing subway passengers' abnormal behavior videos enhanced with skeleton key point knowledge according to claim 1 is characterized in that: The human skeleton joint point extraction algorithm OpenPose-slim is used to extract the skeleton joint points of each passenger in each abnormal behavior video clip.
4. The method for describing subway passengers' abnormal behavior videos enhanced with skeleton key point knowledge according to claim 1 is characterized in that: Construct abnormal behavior knowledge graph based on subway domain knowledge, including: Based on the knowledge of subway domain, the abnormal behaviors of subway passengers in ConceptNet5.5 dataset and Subway dataset are integrated, and the abnormal behaviors of subway passengers and passenger skeleton joints are used to construct an abnormal behavior knowledge graph.
5. The method for describing subway passengers' abnormal behavior videos enhanced with skeleton key point knowledge according to claim 1 is characterized in that: The multiple abnormal behavior categories are: climbing over to evade the fare, crawling under to evade the fare, following to evade the fare, falling down the stairs, running on the stairs, going the wrong way on the escalator, leaning out of the escalator, and slipping on the handrail; The multiple attribute features of each bone joint point are: the angle formed by the bone joint points, the change in the angle between the bone joint points, the acceleration of the bone joint points, the distance between the bone joint point and the center of gravity of the skeleton, and the offset of the bone joint points.
