Real-time end-to-end human behavior detection system and method based on relative position spatial coding

By using a real-time end-to-end human behavior detection system based on relative position spatial coding, combined with temporal feature caching and multi-scale feature fusion, the system solves the problems of visual blind spots and individual differences in proctoring during large-scale multi-person examinations. It achieves real-time and accurate monitoring of examinees and automatic identification of abnormal behaviors, ensuring the fairness and impartiality of the examination.

CN120853101BActive Publication Date: 2026-11-17HARBIN INST OF TECH +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510915841.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-07-03
Publication Date
2026-11-17
Estimated Expiration
2045-07-03

AI Technical Summary

Technical Problem

Existing offline examination proctoring methods rely on manual proctoring and post-exam monitoring, which have blind spots, invigilator fatigue leading to missed detections, individual differences, and monitoring gaps. They are difficult to achieve accurate monitoring at all times and in all aspects, and existing technologies cannot meet the real-time proctoring needs of large-scale, multi-person examinations.

Method used

A real-time end-to-end human behavior detection system based on relative position spatial coding is adopted. It combines temporal feature buffer and multi-scale feature fusion. The system uses image data collected by camera to perform feature extraction, target detection and abnormal behavior analysis, so as to realize real-time monitoring of candidates and automatic identification of abnormal behavior.

Benefits of technology

It improves the efficiency and accuracy of exam invigilation, reduces the burden on invigilators, ensures the fairness and impartiality of exams, enables real-time monitoring of multiple candidates simultaneously, identifies continuous violations, reduces misjudgments, and adapts to different exam room environments with camera layouts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120853101B_ABST
    Figure CN120853101B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of image processing technology, specifically relating to a real-time end-to-end human behavior detection system and method based on relative position spatial coding. The camera transmits the acquired image data to a feature extraction module, which then transmits the extracted multi-scale fused features to a target detection head and a temporal feature buffer. The features output by the target detection head, the features received by the temporal feature buffer, and the encoded information of the student's relative position and the camera are fused. Based on the fused features, the data is input to an abnormal behavior analysis module for abnormal behavior analysis, which then searches for students exhibiting abnormal behavior in the monitoring footage. This invention achieves accurate detection of human targets and effective analysis of abnormal behavior.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a real-time end-to-end human behavior detection system and method based on relative position spatial coding. Background Technology

[0002] The current large-scale offline examination invigilation system mainly relies on a combination of human invigilation and post-exam monitoring. This traditional model has the following technical shortcomings: each invigilator needs to monitor dozens of candidates simultaneously, resulting in blind spots and the risk of missed detections due to fatigue; existing monitoring equipment is only used for post-exam evidence collection and cannot achieve real-time blocking of abnormal behavior; the criteria for judging abnormal behavior by human invigilators vary from person to person, lacking standardized quantitative indicators; and invigilators patrolling across examination rooms can lead to monitoring gaps, making it difficult to detect cheating during these periods.

[0003] With the increasing prevalence of large-scale offline examinations, the fairness and security of these exams have become a focus of public attention. A fair examination environment not only affects the vital interests of examinees but also the credibility of the education evaluation system. Traditional examination invigilation methods mainly rely on manual patrols, where invigilators patrol the examination room to observe examinees' behavior and determine if any violations occur. However, due to limited human resources, relying solely on manual invigilation in large-scale examinations makes it difficult to achieve comprehensive and thorough supervision. Furthermore, the subjective judgment of invigilators may be influenced by their personal experience, introducing a degree of uncertainty. To compensate for the shortcomings of manual invigilation, many examination rooms are equipped with video surveillance equipment. By recording the examination process in real time, examination staff can remotely view the examination room and review the footage after the examination.

[0004] However, this traditional invigilation method has many shortcomings. Human invigilation is limited by the energy and attention of invigilators, making it difficult to achieve accurate monitoring throughout the entire process. In large-scale examinations, the number of candidates in a single examination room is large, while the number of invigilators is limited, often making it impossible to monitor the behavior of all candidates simultaneously, resulting in some violations going undetected. Furthermore, invigilators typically need to patrol between multiple examination rooms; this mobile invigilation model may create monitoring gaps at certain times, making cheating difficult to detect. Although video surveillance equipment can record the entire examination, post-examination playback and analysis not only consumes a significant amount of manpower and time but may also miss opportunities to stop violations in real time, thus affecting the fairness and effectiveness of the examination.

[0005] To address the aforementioned issues, intelligent proctoring systems based on computer vision and artificial intelligence have gained increasing attention in recent years. Among these, object detection technology has become a research hotspot, with researchers aiming to automatically detect abnormal behavior through deep learning algorithms, thereby achieving intelligent supervision of the entire examination process. Currently, object detection algorithms are widely used in various computer vision tasks, with the deep learning-based YOLO (You Only Look Once) algorithm achieving significant success in this field. The YOLO algorithm can detect objects in images in real time with extremely high speed and accuracy, and is widely used in security monitoring, autonomous driving, and other fields. In examination proctoring scenarios, the YOLO algorithm can be used to detect abnormal behavior by examinees, such as looking around, whispering, or using mobile phones, thereby improving the intelligence level of examination proctoring.

[0006] However, the YOLO algorithm has the following limitations in large-scale, multi-person examination scenarios, making it difficult to fully meet the needs of real-time examination proctoring. The YOLO algorithm is primarily optimized for detecting single or small numbers of targets. In large examination rooms, the monitoring screen often contains a large number of examinees, with numerous targets exhibiting varying postures and angles, easily leading to decreased detection accuracy and false positives or false negatives. YOLO is a single-frame target detection algorithm, relying solely on the information of the current frame for each detection, and cannot combine historical frame data for behavioral pattern analysis. Therefore, it struggles to accurately identify persistent abnormal behavior in examinees, such as prolonged sideways conversations or repeatedly turning to look at others' answer sheets. YOLO is mainly used for target recognition, while examination proctoring requires fine-grained classification of abnormal behavior, such as distinguishing between normal examination room actions and suspicious cheating. Furthermore, traditional YOLO detection algorithms lack a linkage mechanism with alarm systems, failing to provide timely and accurate alarm feedback, which may prevent proctors from responding quickly to violations.

[0007] Existing online examination systems have several significant shortcomings in practical applications. First, many systems can only monitor individual examinees in real-time, which is insufficient for offline examinations with multiple participants. For example, in a large examination room, dozens or even hundreds of examinees may be taking the exam simultaneously, and existing online examination systems cannot effectively monitor so many examinees at the same time. This limits their application value in real-world offline examination scenarios. Second, existing behavior analysis systems typically require multiple consecutive frames of images to detect and classify examinee behavior. While this method is theoretically feasible, it has a significant problem in practical application: it cannot meet the real-time feedback requirements of offline examinations. In other words, these systems cannot provide invigilators with timely analysis results of examinee behavior during the examination, thus failing to promptly detect and address potential cheating.

[0008] Besides YOLO detection technology, some online proctoring systems exist on the market, typically relying on the camera of a personal device (such as a computer or mobile phone) for single-person, single-computer proctoring. While these systems can detect abnormal behavior to some extent, such as faces moving away from the screen or window switching, they still have significant drawbacks. For example, they only support single-person, single-computer proctoring, making it difficult to meet the needs of large-scale examinations. Secondly, because these systems rely on a single device's camera, they can usually only capture a frontal image of the examinee, failing to comprehensively monitor the examinee's hands, surrounding environment, or off-screen movements, resulting in blind spots in cheating detection. Most importantly, they lack intelligent abnormal behavior analysis. Existing online proctoring systems mostly rely on fixed rules for violation detection, such as calculating the time the face leaves the screen or detecting mouse and keyboard activity, lacking accurate identification of complex cheating behaviors. Summary of the Invention

[0009] This invention provides a real-time end-to-end human behavior detection system and method based on relative position spatial coding, which can achieve accurate detection of human targets and effective analysis of abnormal behavior.

[0010] This invention is achieved through the following technical solution: A real-time end-to-end human behavior detection system based on relative position spatial coding, the system comprising a feature extraction module, a target detection head, a camera, a temporal feature buffer, and an abnormal behavior analysis module; The camera transmits the captured image data to the feature extraction module. The feature extraction module transmits the extracted multi-scale fusion features to the target detection head and the temporal feature buffer, respectively. The features output by the target detection head, the features received by the temporal feature buffer, and the encoding information of the student's relative position and the camera are fused together. Based on the fused features, the abnormal behavior analysis module is input to perform abnormal feature behavior analysis, and then students with abnormal behavior are found in the monitoring screen.

[0011] Furthermore, the feature extraction module uses the YOLO backbone network, which fuses the extracted multi-scale features through a feature pyramid network or a path aggregation network. The multi-scale features include low-level detailed features and high-level semantic features.

[0012] Furthermore, the encoding information is a relative position encoding, which assigns spatial position information to the grid cells in the feature map.

[0013] Furthermore, the target detection head performs a classification task based on the fused feature map; The temporal feature buffer is used to store feature maps or detection results of historical frames.

[0014] A detection method for a real-time end-to-end human behavior detection system based on relative positional spatial coding, wherein the detection method uses the aforementioned real-time end-to-end human behavior detection system based on relative positional spatial coding, and the detection method includes... The camera captures the examination monitoring footage and transmits it to the feature extraction module for multi-scale feature extraction based on the YOLO backbone network. Then, it is fused through a feature pyramid network or a path aggregation network to obtain multi-scale fused features. The multi-scale fusion features are divided into stored data and detection data. The detection data is input into the target detection head for target detection, and the stored data is stored in the temporal feature buffer. The relative position of the student and the encoded information of the camera are obtained. The encoded information is then fused with the target type features after target detection and the multi-scale fusion features stored in the temporal feature buffer. The abnormal behavior analysis module receives fused features from encoded information, target type features, and multi-scale fusion features at its input end. It then uses these fused features to perform abnormal behavior analysis and identify normal or abnormal behavior patterns.

[0015] Furthermore, the encoding information of the student's relative position specifically involves obtaining the resolution of the monitoring screen, taking the upper left corner of the screen as the origin of the coordinate system, then the rightward direction is the x-axis and the downward direction is the y-axis. At this point, each student has an independent coordinate. The spatial position of the student is determined by calculating the relative ratio between the monitoring screen resolution and the student's coordinates. Specifically, the encoding information of the camera is obtained by pre-setting the number of possible hanging positions of the camera, and then encoding each possible position.

[0016] Furthermore, the temporal feature buffer can track the target's motion path, calculate behavior patterns, and provide temporal contextual information for abnormal behavior analysis by analyzing the temporal correlation between consecutive frames. The motion path of the tracked target is specifically calculated by determining the position of the detection box for the same target in each image and obtaining the motion path of the target based on the change in the position of the detection box. The specific computational behavior pattern involves inputting the extracted detection box information and key point coordinate information into the abnormal behavior analysis network of the abnormal behavior analysis module to obtain the final behavior classification for each student.

[0017] Furthermore, the abnormal behavior analysis network consists of two information processing paths. One path extracts visual features from the input human pixel block through a two-dimensional convolutional layer and max pooling operation, and finally converts it into a feature map, i.e. a vector representation. Another approach applies multiple ST-GCN modules to the input keypoint time series to extract spatiotemporal features from dynamic behavior and output a vector representation.

[0018] Furthermore, the feature fusion involves taking the student's coordinates as additional input, jointly modeling them with the target type features after target detection and the multi-scale fusion features stored in the temporal feature cache, and then feeding them into a fully connected layer to complete the final behavior classification task.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0020] The beneficial effects of this invention are: The purpose of this invention is to provide a more efficient and reliable exam proctoring solution.

[0021] This invention integrates multiple core functions, including examinee identification, abnormal behavior detection, violation classification, and real-time alarms. It can analyze examinee behavior in real time and provide timely warnings during the examination. When the system detects suspicious behavior, it immediately sends an alarm to the invigilator, enabling them to take swift action and effectively reducing blind spots and loopholes in manual invigilation.

[0022] The application of this invention can not only improve the efficiency of examination invigilation and reduce the burden on invigilators, but also ensure the fairness and impartiality of the examination to the greatest extent, providing a safer and more reliable intelligent invigilation solution for various large-scale examinations.

[0023] This invention can simultaneously monitor a large number of examinees in the examination room in real time. Through optimized target detection and behavior analysis algorithms, it improves the accuracy and response speed of identifying abnormal examinee behavior.

[0024] This invention employs temporal feature caching technology, which can model the historical behavior of examinees, identify continuous violations, avoid misjudgment problems in single-frame detection, and thus improve the stability and accuracy of the system.

[0025] This invention, by introducing relative position coding and camera position coding, can adapt to camera layouts in different examination room environments, improving the recognition accuracy of candidates at a distance and enhancing cross-examination room generalization capabilities. Therefore, in scenarios requiring real-time monitoring of candidate behavior, such as examination invigilation, this invention can play a more effective role, helping invigilators to promptly detect and address abnormal candidate behavior, thereby ensuring the fairness and impartiality of the examination. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of the present invention.

[0027] Figure 2 This is a schematic diagram of the target detection head of the present invention.

[0028] Figure 3 This is a schematic diagram of the position encoding of the present invention. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0030] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] The following is in conjunction with the appendix to this application specification. Figure 1-3 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0034] Implementation Method 1 To address the issue that YOLO is unsuitable for continuous behavior detection, this invention introduces a temporal feature buffer (such as...). Figure 1As shown, multiple adjacent frames can be saved, and then all adjacent frames can be used together during recognition. This fully utilizes the temporal information in the video, enhances the model's performance, and overcomes the limitations of single-frame detection. Traditional YOLO algorithms rely solely on the information of the current frame for detection, which can easily lead to misjudgments of instantaneous actions, especially in accurately distinguishing between normal movement and abnormal behavior of examinees within a short period. Therefore, this invention designs a temporal feature caching mechanism that can store detection results from multiple past frames and fully utilize this historical information in the classification task of the current frame. This mechanism not only smooths out interference from short-term actions but also identifies persistent violations, such as prolonged whispering or peeking at other examinees' papers, through time series modeling.

[0035] Secondly, to optimize the detection performance of invigilator cameras captured from different locations, this invention introduces relative positional spatial coding. Existing invigilator cameras are typically mounted on the wall in front of the examination room. This top-down shooting method results in students closer to the camera occupying a larger portion of the frame, while students further away occupy a smaller portion, thus affecting the balance of the detection model. Furthermore, changes in the relative position of students can interfere with the detection results; for example, a student in the center of the frame may be more easily identified than a student at the edge of the frame even if they perform the same action. To address this issue, this invention introduces relative positional coding into the model, enabling the detection network to fully utilize the relative positional information of students and adaptively adjust the target scale during inference, thereby improving the detection accuracy of students at a distance. Simultaneously, due to differences in the installation height and horizontal position of cameras in different examination rooms, the image offset varies. Therefore, this invention further introduces camera positional coding to compensate for geometric transformation errors caused by different camera perspectives, ensuring the model maintains high generalization ability when deployed across different scenarios.

[0036] By combining temporal feature caching with positional encoding technology, this invention not only overcomes the shortcomings of YOLO in continuous image detection, but also significantly improves the detection stability and cross-examination adaptability in multi-person scenarios, providing more efficient and accurate technical support for intelligent proctoring of large-scale examinations.

[0037] This embodiment provides a real-time end-to-end human behavior detection system based on relative position spatial coding. The system includes a feature extraction module, a target detection head, a camera, a temporal feature buffer, and an abnormal behavior analysis module. The camera transmits the captured image data to the feature extraction module. The feature extraction module then transmits the extracted multi-scale fusion features to the target detection head and the temporal feature buffer, respectively. The features output by the target detection head, the features received by the temporal feature buffer, and the encoding information of the student's relative position and the camera are fused together. Based on the fused features, the abnormal behavior analysis module is input to perform abnormal behavior analysis and then search for students with abnormal behavior in the monitoring screen.

[0038] Furthermore, the feature extraction module uses the YOLO backbone network, which fuses the extracted multi-scale features through a Feature Pyramid Network (FPN) or a Path Aggregation Network (PANet). The multi-scale features include low-level detail features (such as edges and textures) and high-level semantic features (such as student actions).

[0039] Furthermore, the encoding information is a relative position encoding, which assigns spatial position information to the grid cells in the feature map.

[0040] Furthermore, the target detection head performs a classification task based on the fused feature map; The temporal feature buffer is used to store feature maps or detection results of historical frames.

[0041] The purpose of this invention is to provide a more efficient and reliable exam proctoring solution. First, by introducing advanced computer vision and multi-channel video processing technologies, this invention enables simultaneous monitoring of multiple examinees. Thus, in scenarios with multiple examinees, the proctoring system can simultaneously capture and analyze the behavior of all examinees, ensuring the fairness and impartiality of the exam. Second, this invention employs an innovative temporal feature buffer to store feature information extracted from multiple consecutive frames. By fusing the feature information from these consecutive frames with the features of the current frame, the algorithm can more accurately capture and analyze changes in examinee behavior. For example, it detects the entire process from looking down to looking up, rather than just a single instantaneous state. This method effectively utilizes temporal information from multiple frames of images, improving the accuracy and reliability of action detection. Therefore, in scenarios such as exam proctoring where real-time monitoring of examinee behavior is required, this improved algorithm can play a better role, helping proctors to promptly detect and address abnormal examinee behavior, thereby ensuring the fairness and impartiality of the exam.

[0042] Implementation Method 2 This embodiment provides a detection method for a real-time end-to-end human behavior detection system based on relative positional spatial coding. The detection method uses the real-time end-to-end human behavior detection system based on relative positional spatial coding as described in Embodiment 1. The detection method includes... The camera captures the examination monitoring screen and transmits the examination monitoring screen to the feature extraction module for multi-scale feature extraction based on the YOLO backbone network. Then, it is fused through the Feature Pyramid Network (FPN) or the Path Aggregation Network (PANet) to obtain multi-scale fused features. The multi-scale fusion features are divided into stored data and detection data. The detection data is input into the target detection head for target detection, and the stored data is stored in the temporal feature buffer. The relative position of the student and the encoded information of the camera are obtained. The encoded information is then fused with the target type features after target detection and the multi-scale fusion features stored in the temporal feature buffer. The abnormal behavior analysis module receives fused features from encoded information, target type features, and multi-scale fusion features at its input end. It then uses these fused features to perform abnormal behavior analysis and identify normal or abnormal behavior patterns.

[0043] Furthermore, the encoding information for the student's relative position is specifically as follows: assuming the monitoring screen resolution is 1920×1080, the top left corner of the screen is the origin, the x-axis is to the right, and the y-axis is downward. At this time, the center coordinates of a detected student frame are (480, 100).

[0044] Normalized to relative proportions: x = 480 / 1920 = 0.25, y = 100 / 1080 ≈ 0.09.

[0045] Meaning: y=0.09 indicates that the student is located in the upper part of the image (the back row of the lecture hall).

[0046] Feature fusion: (x, y) is used as an additional input and concatenated or weighted with image features (such as feature maps output by YOLO).

[0047] Example result: Without positional encoding: the model may ignore small targets at the top of the screen (students in the back row).

[0048] Using positional encoding: Based on the prior y≈0.09, the model strengthens its attention to the top region, which can improve the detection probability even if the target is blurry.

[0049] Specifically, the camera's encoding information is as follows: In a classroom, the camera is usually mounted on the wall directly in front. However, if the camera is mounted in different positions, the angle of the captured video frames will be different, which will affect the recognition results for the same student. Therefore, the number of possible camera mounting positions can be preset, such as 3, and then each possible position can be encoded as (0, 0, 1), (1, 0, 0), (0, 1, 0). This position information is then fused with image features to enhance the detection model's resistance to disturbances.

[0050] The target detection head performs a classification task based on the fused feature map. The output of the detection head is the core of the real-time target detection results, directly providing the target type (normal, standing, raising hands, etc.) and its detection confidence, providing a data foundation for subsequent time series analysis and anomaly judgment.

[0051] Furthermore, the temporal feature buffer can track the target's movement path and calculate behavioral patterns (such as continuous head turning or reaching out) by analyzing the temporal correlation between consecutive frames, and provide temporal contextual information for abnormal behavior analysis. This module enhances the model's ability to understand dynamic scenes, and is especially suitable for scenarios such as video surveillance that require long-term behavior analysis. The motion path of the tracked target is specifically calculated by determining the position of the detection box for the same target in each image and obtaining the motion path of the target based on the change in the position of the detection box. The specific computational behavior pattern is as follows: Figure 3 As shown, by Figure 2 The extracted detection box information and key point coordinate information are input into the abnormal behavior analysis network of the abnormal behavior analysis module to obtain the final behavior classification of each student (turning head, reaching out, etc.).

[0052] Furthermore, the abnormal behavior analysis module, based on target detection, further analyzes the behavior of the detected human targets. The behavior analysis unit analyzes the fused features to determine whether the human target's behavior conforms to expectations (e.g., ...). Figure 3 As shown in the diagram, actions such as standing or raising a hand are examples of behaviors. Through this process, the system can identify normal or abnormal behavioral patterns, providing technical support for the identification and early warning of abnormal behaviors.

[0053] Ultimately, this invention automates the precise detection and behavioral analysis of human targets through the collaborative work of these modules. This not only improves the efficiency and accuracy of invigilation and reduces examination costs, but also provides candidates with a fairer and more impartial examination environment. Simultaneously, it alleviates the invigilation pressure on examination organizers, allowing them to focus more on the organization and management of the examination.

[0054] The specific details of each part are explained below: 1) YOLO Backbone Network: The feature extraction part adopts the existing backbone structure in the YOLO model, consisting of convolutional layers, C3K2, C2PSA, and other modules, serving as the key feature extraction unit. After inputting a single frame image from the video, this module can effectively extract key information from the image, and extracting image features is an important step in the object detection process.

[0055] 2) Target detection head: This part of the network structure is as follows Figure 2 The demonstration features two main detection modules: one for object detection and the other focused on extracting key points of human pose. The entire architecture is designed around convolutional layers, possessing excellent multi-scale feature learning capabilities, enabling accurate identification and localization of targets of varying sizes within images.

[0056] The network first receives image features extracted from module 1) and generates two types of outputs in parallel based on these features: one is a bounding box for locating the examinee, i.e., location information; the other is key point information for describing the human skeletal structure. Considering that only the upper body of the examinee needs to be monitored in the invigilation scenario, the key point detection part focuses on seven key areas: the center of the head, the left and right shoulders, the left and right elbows, and the left and right wrists, thereby achieving accurate capture and analysis of the upper body posture.

[0057] 3) Temporal Feature Buffer: The temporal buffer stores the key points of human pose from the previous few frames obtained in step 2). Since human movements are continuous, it is difficult to accurately identify a complete action process based solely on a single frame. Therefore, it is necessary to introduce pose information from historical frames as an auxiliary basis to enhance the understanding and judgment of the current behavior. The buffer's contents are continuously updated with the arrival of new frames. Through a sliding window mechanism, the model can continuously acquire recent dynamic changes, improving the coherence and accuracy of action recognition.

[0058] 4) Feature Fusion: In the feature fusion process, location encoding and camera encoding are deeply fused with YOLO visual features through a concat method. For student location information, the image coordinates are transformed into spatial perception vectors, enhancing the visual feature weights of occluded areas and enabling the model to prioritize focusing on key areas such as the back rows of the lecture hall. For camera parameters, the camera position is encoded to adjust the sensitivity of the detection network to targets from different viewpoints. After fusing these two encodings with existing features, the discriminative power of visual features is maintained while incorporating prior physical knowledge of the scene and device, thus improving the robustness of recognition in complex occluded scenarios.

[0059] 5) Abnormal behavior analysis: The network structure is as follows Figure 3As shown, the main inputs are the target pixel region of the current frame and the corresponding temporal keypoint sequence, and the output is the behavior category information of the detected examinee. The overall architecture consists of two information processing paths: one path extracts visual features from the input pixel block of the person through a series of two-dimensional convolutional layers and max pooling operations, and finally converts it into a feature map; the other path applies multiple ST-GCN modules (Spatiotemporal Graph Convolutional Network) to the input keypoint time series to extract spatiotemporal features in dynamic behavior, and also outputs a vector representation.

[0060] Next, the two location encoding information mentioned above are fused with the features extracted from these two paths at the end through a concat operation. The pixel and time series information, as well as the location coordinates, are jointly modeled and then fed into a fully connected layer to complete the final behavior classification task.

[0061] Implementation Method 3 This invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any step in Embodiment 1 by running the computer program stored in the memory.

[0062] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0063] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0064] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods described above can also be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0065] The above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0066] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0067] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments for mutual reference, and will not be repeated here.

[0068] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0069] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A real-time end-to-end human action detection system based on relative position space coding, characterized in that, The system includes a feature extraction module, a target detection head, a camera, a temporal feature buffer, and an abnormal behavior analysis module; The camera transmits the captured image data to the feature extraction module. The feature extraction module uses the YOLO backbone network, which fuses the extracted multi-scale features through a feature pyramid network or a path aggregation network. The multi-scale features include low-level detail features and high-level semantic features. The feature extraction module transmits the extracted multi-scale fusion features to the target detection head and the temporal feature buffer respectively. The features output by the target detection head, the features received by the temporal feature buffer, and the encoding information of the student's relative position and the camera are fused together. Based on the fused features, the abnormal behavior analysis module is input to perform abnormal feature behavior analysis, and then students with abnormal behavior are found in the monitoring screen. The target detection head performs a classification task based on the fused feature map; The temporal feature buffer is used to store feature maps or detection results of historical frames.

2. The system of claim 1, wherein, The encoding information is a relative position encoding, which assigns spatial position information to the grid cells in the feature map.

3. A detection method of a real-time end-to-end human behavior detection system based on relative position space coding, characterized in that, The detection method uses a real-time end-to-end human behavior detection system based on relative position spatial coding as described in any one of claims 1-2, and the detection method includes, The camera captures the examination monitoring footage and transmits it to the feature extraction module for multi-scale feature extraction based on the YOLO backbone network. Then, it is fused through a feature pyramid network or a path aggregation network to obtain multi-scale fused features. The multi-scale fusion features are divided into stored data and detection data. The detection data is input into the target detection head for target detection, and the stored data is stored in the temporal feature buffer. The relative position of the student and the encoded information of the camera are obtained. The encoded information is then fused with the target type features after target detection and the multi-scale fusion features stored in the temporal feature buffer. The encoding information of the student's relative position is specifically as follows: obtain the resolution of the monitoring screen, take the upper left corner of the screen as the origin of the coordinate system, then the right is the x-axis and the down is the y-axis. At this time, each student has an independent coordinate. The spatial position of the student is determined by calculating the relative ratio between the resolution of the monitoring screen and the student's coordinate. Specifically, the encoding information of the camera is as follows: the number of possible hanging positions of the camera is preset, and then each possible position is encoded. The abnormal behavior analysis module receives fused features from encoded information, target type features, and multi-scale fusion features at its input end. It then uses these fused features to perform abnormal behavior analysis and identify normal or abnormal behavior patterns. The temporal feature buffer can track the target's motion path, calculate behavior patterns, and provide temporal contextual information for abnormal behavior analysis by analyzing the temporal correlation between consecutive frames. The computational behavior pattern specifically involves inputting the extracted detection box information and key point coordinate information into the abnormal behavior analysis network of the abnormal behavior analysis module to obtain the final behavior classification for each student. The abnormal behavior analysis network consists of two information processing paths. One path extracts visual features from the input human pixel block through a two-dimensional convolutional layer and max pooling operation, and finally converts it into a feature map, i.e. a vector representation. Another approach applies multiple ST-GCN modules to the input keypoint time series to extract spatiotemporal features from dynamic behavior and output a vector representation.

4. The detection method according to claim 3, characterized in that, The motion path of the tracked target is specifically calculated by determining the position of the detection box for the same target in each image and obtaining the motion path of the target based on the changes in the position of the detection box.

5. The detection method according to claim 3, characterized in that, The feature fusion involves taking the student's coordinates as additional input, jointly modeling them with the target type features after target detection and the multi-scale fusion features stored in the temporal feature buffer, and then feeding them into a fully connected layer to complete the final behavior classification task.

6. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 3-5.

Citation Information

Patent Citations

  • Monitoring video behavior analysis method and system based on time sequence characteristics, and terminal

    CN114120242A

  • Elevator car passenger abnormal behavior detection system and method

    CN115527271A