Examination room abnormal behavior detection method and system based on time sequence key points

Abstract

The invention provides an examination room abnormal behavior detection method and system based on time sequence key points, and belongs to the technical field of human body abnormal behavior detection. The method comprises the following steps: acquiring an examinee image sequence in an examination room video, and extracting time sequence key point data sequence information; the extracted information is input into a trained Transform model, and a single-person feature vector of the examinee is obtained; performing single-person abnormal behavior prediction on the single-person feature vector by using a classifier; and according to a prediction result, constructing a graph structure by taking the single-person feature vector as a vertex and taking the examination room seating table as a topological structure, inputting the graph structure into a graph neural network, and judging whether an abnormal association behavior exists between the examinees in the specific seating relationship or not. The subtle action change behavior characteristics of the examinee can be more accurately captured and recognized, and the detection accuracy is improved. Meanwhile, potential collaborative abnormal behaviors among the examinees can be mined based on analysis of a graph structure, the accuracy of overall abnormal behavior detection is further improved, and the workload of invigilators is relieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human abnormal behavior detection, and particularly relates to a method and system for detecting abnormal behavior in an examination room based on temporal key points. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] The detection of abnormal behavior of students in the examination room usually requires invigilators to patrol and timely discover through video monitoring. However, there are often certain visual blind spots in the invigilators' patrols, making it difficult to detect all abnormal behaviors in the examination room. Most of the video monitoring systems in the examination room are in the traditional mode, and their main functions and applications still remain in shooting and storage. They can only simply record and store the video of the examination scene, unable to effectively detect abnormal behaviors that occur in the examination room in real time. Moreover, during the confirmation process, personnel need to browse a large amount of video data for the positioning and judgment of abnormal behaviors, resulting in low detection efficiency.

[0004] With the breakthrough progress of deep learning technology in the fields of image classification, target recognition, etc., in recent years, there have also been related studies applying deep learning technology to the detection of abnormal behavior in the examination room. However, using a deep network can only detect the abnormal behavior of a single person. In the examination room, abnormal behavior is not limited to a single person, and there are also situations of abnormal behavior of multiple people. Therefore, the above-mentioned abnormal behavior detection method cannot effectively handle the problem of abnormal behavior of complex interactions among multiple people, resulting in inaccurate detection of abnormal behavior. Summary of the Invention

[0005] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for detecting abnormal behavior in an examination room based on temporal key points, effectively solving the problems of low accuracy of the existing examination room abnormal behavior detection technology and difficulty in dealing with abnormal behavior of complex interactions among multiple people, improving the intelligent level and efficiency of examination room monitoring, and providing strong technical support for maintaining the fairness and justice of examinations.

[0006] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of the present invention provides a method for detecting abnormal behavior in an examination room based on temporal key points;

[0008] A method for detecting abnormal behavior in an examination room based on temporal key points includes:

[0009] Obtain the image sequence of candidates in the examination room video, and extract the sequence information of human temporal key point data of candidates in the examination room;

[0010] Input the extracted sequence information of the human body temporal key points of the examinees into the trained Transformer model to obtain the single-person feature vector of each examinee; input the obtained single-person feature vector into the classifier for single-person abnormal behavior prediction;

[0011] According to the single-person abnormal behavior prediction result, use the single-person feature vector as the vertex and the seating chart of the examination room as the topological structure to construct a graph structure; input the constructed graph structure into the graph neural network to determine whether there is an abnormal association behavior between the examinees with specific seating relationships.

[0012] As a further technical solution, during the process of obtaining the image sequence of the examinees in the examination room video, it also includes preprocessing the collected examination video data, and the preprocessing includes image denoising and color correction.

[0013] As a further technical solution, the process of extracting the sequence information of the human body temporal key points of the examinees in the examination room is as follows:

[0014] Use the pre-trained human key point detection model to process the preprocessed examination room video frames to obtain the human body temporal key point information of the examinees;

[0015] Arrange the human body temporal key point information in chronological order to obtain the sequence of the human body temporal key points of each examinee.

[0016] As a further technical solution, the human body temporal key point information of the examinees includes the position coordinates of the key parts of the examinees' heads, necks, shoulders, elbows, wrists, waists, knees, and ankles.

[0017] As a further technical solution, the trained Transformer model uses the encoder to perform position encoding on the input sequence of temporal key points, retaining the order information of the key points in the time series;

[0018] Use the multi-head attention mechanism to learn and fuse the relationships between different time steps and different key points to obtain the high-dimensional feature representation vector of each examinee;

[0019] Use the decoder to convert the high-dimensional feature representation vector into a single-person feature vector suitable for abnormal behavior classification.

[0020] As a further technical solution, the classifier uses the cross-entropy loss function as the optimization objective and continuously adjusts the parameters of the Transformer model through the backpropagation algorithm.

[0021] As a further technical solution, a method for detecting abnormal behaviors in an examination room based on timing key points further includes generating an abnormal behavior report immediately when single or multiple abnormal behaviors are detected; the content of the report includes the time when the abnormal behavior occurs, the seat numbers of the examinees involved, the types of abnormal behaviors, and relevant evidence data.

[0022] The second aspect of the present invention provides a system for detecting abnormal behaviors in an examination room based on timing key points.

[0023] A system for detecting abnormal behaviors in an examination room based on timing key points includes:

[0024] A key point data sequence information acquisition module, configured to: acquire an image sequence of examinees in an examination room video, and extract the human body timing key point data sequence information of the examinees in the examination room;

[0025] A single-person abnormal behavior prediction module, configured to: input the extracted human body timing key point data sequence information of the examinees into a trained Transformer model to obtain a single-person feature vector for each examinee; input the obtained single-person feature vector into a classifier for single-person abnormal behavior prediction;

[0026] A multi-person abnormal behavior prediction module, configured to: according to the single-person abnormal behavior prediction result, construct a graph structure with the single-person feature vector as the vertex and the examination room seating chart as the topological structure; input the constructed graph structure into a graph neural network to determine whether there is an abnormal association behavior between examinees with specific seat relationships.

[0027] The third aspect of the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the steps in a method for detecting abnormal behaviors in an examination room based on timing key points as described in the first aspect of the present invention.

[0028] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in a method for detecting abnormal behaviors in an examination room based on timing key points as described in the first aspect of the present invention.

[0029] The above one or more technical solutions have the following beneficial effects:

[0030] (1) Through the precise analysis of timing key points and the powerful feature fusion ability of Transformer, the present invention can also more accurately capture and identify some subtle but potentially abnormal action change behavior characteristics of candidates, thereby effectively reducing false alarms and missed detections, and improving the accuracy of single-person abnormal behavior detection. At the same time, the multi-person associated behavior analysis based on the graph structure can discover potential collaborative abnormal behaviors among candidates, further improving the overall accuracy of abnormal behavior detection.

[0031] (2) Using the seating chart of the examination room to construct a graph structure can naturally incorporate the relationships among candidates in a multi-person examination room into the analysis framework, fully considering the influence of the spatial position relationships among candidates on their behaviors, and effectively dealing with complex interactive abnormal behavior patterns among multiple people, such as group cheating, etc., providing more comprehensive and in-depth behavior analysis capabilities for examination room supervision.

[0032] (3) The method of the present invention can achieve automatic and real-time analysis of examination room videos without a large amount of manual intervention, greatly reducing the workload of invigilators and improving the efficiency of examination room monitoring. And it can timely detect abnormal behaviors and issue alarms, providing timely and effective technical support for maintaining examination room order and examination fairness.

[0033] Advantages of additional aspects of the present invention will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0035] Figure 1 It is a flowchart of the method for the first embodiment.

[0036] Figure 2 It is a system structure diagram for the second embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0039] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0040] The present invention first performs high-precision human key point detection on candidates in the examination room video to obtain the sequence information of human temporal key point data. Then, the Transformer model is used to perform feature fusion on the sequence information of the temporal key point data of a single person, and a well-trained classifier is used to predict the abnormal behavior of a single person in the examination room. Then, taking the single-person feature vector as the vertex, the seating chart of the examination room constructs a graph structure, and the graph neural network is used to predict the edge information, so as to realize the detection of abnormal behaviors of two or more people in the examination room.

[0041] Embodiment 1

[0042] This embodiment discloses a method for detecting abnormal behaviors in an examination room based on temporal key points;

[0043] As Figure 1 shown, a method for detecting abnormal behaviors in an examination room based on temporal key points includes:

[0044] Step S1, obtaining the sequence of candidate images in the examination room video, and extracting the sequence information of the human temporal key point data of the candidates in the examination room;

[0045] Reasonably arrange multiple high-definition cameras in the examination room to ensure that the entire examination room area can be covered without obvious monitoring blind spots. The cameras need to have a certain resolution and frame rate to obtain clear and smooth examination room video data. Cameras with a resolution of 1920×1080 pixels and a frame rate of 25fps can be selected. Preprocess the examination room video data collected by the cameras, including operations such as image denoising and color correction, to ensure high-quality candidate image sequences are obtained for subsequent key point detection. At the same time, according to the actual layout of the examination room and the camera position information, establish the corresponding seating chart data of the candidates to clarify the seat position information of each candidate in the examination room, providing a basis for subsequent graph structure construction.

[0046] Furthermore, use a pre-trained human key point detection model to process the preprocessed examination room video frames, and accurately extract the human temporal key point information of each candidate. The human temporal key point information includes the position coordinates of the key parts of the candidate's head, neck, shoulders, elbows, wrists, waist, knees, and ankles. These key point information form continuous trajectory data along the time series, providing a basic data source for subsequent behavior analysis;

[0047] For each frame of video image, the human key point detection model outputs the temporal key point information of each candidate's body, and arranges these key point coordinate information in chronological order to form a temporal key point data sequence of each candidate's body. For example, for an examination room video with a duration of T seconds and a frame rate of f frames per second, for a certain candidate, a temporal key point data sequence containing T×f time steps will be obtained, and each time step contains the coordinate information of multiple key points of the candidate.

[0048] Step S2: Input the extracted temporal key point data sequence information of the candidate's body into the trained Transformer model to obtain the single-person feature vector of each candidate; input the obtained single-person feature vector into the classifier for single-person abnormal behavior prediction.

[0049] Input the temporal key point data sequence information of each candidate's body into the trained Transformer model. The encoder part of the Transformer model first performs position encoding on the input temporal key point data sequence information of the body to retain the order information of the key points in the time series. Then, through the multi-head attention mechanism, the relationships between different time steps and different key points are learned and feature fusion is performed, which not only fuses spatial information but also effectively fuses time information. Specifically, by setting 2 layers of encoders, each layer using a multi-head attention mechanism with 8 heads, the multi-head attention mechanism of the Transformer model can automatically learn the spatio-temporal correlation features between key points, such as the relative position change relationship between hand key points and head key points at different times, etc., so as to effectively fuse these features and obtain the high-dimensional feature representation of each candidate; input the high-dimensional feature representation of the candidate into the decoder part, and use the decoder to convert the fused features into a single-person feature vector suitable for abnormal behavior classification.

[0050] Then, based on the obtained single-person feature vector, abnormal behavior prediction is performed through a pre-trained classifier. In this embodiment, the classifier consists of a fully connected layer and a Softmax layer. The classifier is pre-trained with a large amount of candidate temporal key point data labeled with normal and abnormal behaviors, and can judge whether there is a single-person abnormal behavior in the examination room according to the input feature vector, such as staring at non-examination paper areas for a long time (which can be judged by the direction of the head key points and the key point features in the eye area), frequent abnormal hand movements (such as rapid and irregular movement of hand key points), etc. In addition, the classifier uses the cross-entropy loss function as the optimization objective, and continuously adjusts the parameters of the Transformer model through the backpropagation algorithm to improve the prediction accuracy of single-person abnormal behaviors. In practical applications, when the abnormal behavior probability output by the classifier exceeds a preset threshold (such as 0.6), it is determined that the candidate has a single-person abnormal behavior, and relevant information is recorded.

[0051] Step S3, based on the prediction result of abnormal behavior of a single person, a graph is constructed with the feature vector of the single person as the vertex and the seating chart of the examination room as the topological structure; the constructed graph is input into the graph neural network to determine whether there is abnormal correlation behavior between candidates with specific seat relationships.

[0052] Since abnormal behavior in the examination room is not limited to a single person, there are also cases of abnormal behavior of multiple people. Based on this, in this embodiment, according to the prediction results of abnormal behavior of a single person, a graph structure is constructed according to the previously established examination room seating table. Specifically, the feature vector of a single person is used as the vertex of the graph, and the adjacent seats include front and back, left and right, diagonally up, and diagonally down as edges. If two examinees are adjacent in the seating table, a connecting edge is established between the corresponding vertices in the graph. For example, for an examination room with N examinees, a graph structure containing N vertices will be obtained. The edges between the vertices are determined according to the adjacent relationship of the seats. There are edge connections between the vertices of adjacent examinees, and there are no direct edge connections between the vertices of non-adjacent examinees. A graph neural network (such as Graph Convolutional Network, GCN) is used to analyze the constructed graph structure. The GCN model transmits and updates information between the vertices of the graph through a message passing mechanism. In each layer of GCN, each vertex aggregates and updates information based on its own feature vector and the feature vectors of adjacent vertices. For example, for a model with L layers of GCN, in the lth layer (1≤l≤L), the hidden state update formula of vertex i is:

[0053]

[0054] Where N(i) represents the set of neighbor vertices of vertex i. is the normalization coefficient, W l and b l is the learnable parameter of the lth layer, and σ is the activation function. After being processed by multiple layers of GCN, the edge information in the graph can reflect whether there are abnormal association behaviors between adjacent candidates, such as eye contact during collaborative cheating (the association between adjacent candidates through the change of the direction of the key points of the head), passing items (the coordination of the key point movements of the hands between adjacent candidates), etc. Finally, a binary classifier is used to judge the edge information to determine whether there is abnormal behavior of two or more people. When the probability of abnormal behavior of an edge exceeds the preset threshold (such as 0.5), it is determined that there is abnormal association behavior between the corresponding candidates, and the relevant information is recorded.

[0055] Furthermore, in this embodiment, when single or multiple abnormal behaviors are detected, the system immediately generates an abnormal behavior report. The report content includes the time when the abnormal behavior occurred, the seat numbers of the examinees involved, the types of abnormal behaviors (such as single abnormal actions, double-person collaborative cheating, etc.), and relevant evidence data (such as video clips when the abnormal behavior occurred, key point data change curves, etc.). The abnormal behavior report is sent to the invigilators in the examination room or the examination management center in a timely manner, so that the invigilators can quickly take corresponding measures for handling, such as warning and investigating the abnormal examinees, to maintain the order of the examination room and the fairness and justice of the examination. At the same time, the system can store and statistically analyze the abnormal behavior data, so as to study and summarize the laws of abnormal behaviors in the examination process subsequently, and provide data support for further optimizing the examination room supervision strategy

[0056] Embodiment Two

[0057] This embodiment discloses an examination room abnormal behavior detection system based on temporal key points;

[0058] As Figure 2 shown, an examination room abnormal behavior detection system based on temporal key points includes:

[0059] A key point data sequence information acquisition module, configured to: acquire the sequence of examinee images in the examination room video, and extract the sequence information of the human body temporal key points of the examinees in the examination room;

[0060] A single-person abnormal behavior prediction module, configured to: input the extracted sequence information of the human body temporal key points of the examinee into the trained Transformer model to obtain the single-person feature vector of each examinee; input the obtained single-person feature vector into a classifier for single-person abnormal behavior prediction;

[0061] A multiple-person abnormal behavior prediction module, configured to: according to the single-person abnormal behavior prediction result, construct a graph structure with the single-person feature vector as the vertex and the examination room seat chart as the topological structure; input the constructed graph structure into a graph neural network to determine whether there are abnormal association behaviors between the examinees with specific seat relationships.

[0062] Embodiment Three

[0063] The purpose of this embodiment is to provide a computer-readable storage medium.

[0064] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps in an examination room abnormal behavior detection method based on temporal key points as described in Embodiment 1.

[0065] Embodiment Four

[0066] The purpose of this embodiment is to provide an electronic device.

[0067] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in a method for detecting abnormal behaviors in an examination room based on timing key points as described in Embodiment 1 are implemented.

[0068] In the devices of the above Embodiments 2, 3, and 4, the steps involved correspond to those in Method Embodiment 1. For the specific implementation manners, reference may be made to the relevant description part of Embodiment 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0069] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple of them can be fabricated into a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.

[0070] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that, based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A method for detecting abnormal behavior in an examination room based on time sequence key points, characterized in that: include: Obtain the candidate image sequence in the examination room video, and extract the human body time sequence key point data sequence information of the candidate in the examination room; The extracted human body time series key point data sequence information of the examinee is input into the trained Transformer model to obtain the single-person feature vector of each examinee; the obtained single-person feature vector is input into the classifier to predict the single-person abnormal behavior; According to the prediction results of single-person abnormal behavior, a graph structure is constructed with the single-person feature vector as the vertex and the examination room seating chart as the topological structure; the constructed graph structure is input into the graph neural network to determine whether there is abnormal correlation behavior between candidates with specific seat relationships.

2. The method for detecting abnormal behavior in an examination room based on time sequence key points as claimed in claim 1, characterized in that: The process of acquiring the candidate image sequence in the examination room video also includes preprocessing the collected examination video data, and the preprocessing includes image denoising and color correction.

3. The method for detecting abnormal behavior in an examination room based on time sequence key points as claimed in claim 1, characterized in that: The process of extracting the human body time sequence key point data sequence information of the examinee in the examination room is as follows: The pre-processed examination room video frames are processed using a pre-trained human key point detection model to obtain the examinee's human body time sequence key point information; The human body timing key point information is sorted in chronological order to obtain the human body timing key point data sequence of each candidate.

4. The method for detecting abnormal behavior in an examination room based on time sequence key points as claimed in claim 3, characterized in that: The human body timing key point information of the examinee includes the position coordinates of the key parts of the examinee's head, neck, shoulders, elbows, wrists, waist, knees and ankles.

5. The method for detecting abnormal behavior in an examination room based on time sequence key points as claimed in claim 1, characterized in that: The trained Transformer model uses an encoder to positionally encode the input time series key point data sequence, retaining the order information of the key points in the time series; The multi-head attention mechanism is used to learn and fuse the relationship between different time steps and different key points to obtain a high-dimensional feature representation vector for each candidate; The decoder is used to convert the high-dimensional feature representation vector into a single-person feature vector suitable for abnormal behavior classification.

6. The method for detecting abnormal behavior in an examination room based on time sequence key points as claimed in claim 1, characterized in that: The classifier uses the cross entropy loss function as the optimization target and continuously adjusts the parameters of the Transformer model through the back propagation algorithm.

7. The method for detecting abnormal behavior in an examination room based on time sequence key points as claimed in claim 1, characterized in that: A method for detecting abnormal behavior in an examination room based on timing key points also includes immediately generating an abnormal behavior report when abnormal behavior of one or more persons is detected; the report content includes the time when the abnormal behavior occurred, the seat number of the examinee involved, the type of abnormal behavior and related evidence data.

8. A system for detecting abnormal behavior in an examination room based on time sequence key points, characterized in that: include: The key point data sequence information acquisition module is configured to: acquire the candidate image sequence in the examination room video, and extract the human body time sequence key point data sequence information of the candidate in the examination room; The single-person abnormal behavior prediction module is configured to: input the extracted human body time series key point data sequence information of the examinee into the trained Transformer model to obtain the single-person feature vector of each examinee; input the obtained single-person feature vector into the classifier to predict the single-person abnormal behavior; The multi-person abnormal behavior prediction module is configured as follows: based on the prediction results of single-person abnormal behavior, a graph structure is constructed with the single-person feature vector as the vertex and the examination room seating chart as the topological structure; the constructed graph structure is input into the graph neural network to determine whether there is abnormal correlation behavior between candidates with specific seat relationships.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for detecting abnormal behavior in an examination room based on timing key points as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for detecting abnormal behavior in an examination room based on timing key points as described in any one of claims 1-7 are implemented.

Images