Intelligent analysis method and system for abnormal behavior based on multi-source data fusion in examination rooms

Abstract

This invention provides a method and system for intelligently analyzing abnormal behavior in examination rooms using multi-source data fusion. This system, which relates to the field of artificial intelligence technology, involves collecting video and audio data from the examination room, performing multi-view skeleton point fusion and sound source localization to extract features; constructing a hidden Markov model and kernel density estimation to identify abnormal behavior; calculating seat associations based on spatial weight coefficients and wavelet decomposition to identify coordinated cheating by multiple people; and delivering early warning information in real time. This invention improves the accuracy of cheating identification and effectively detects coordinated cheating by multiple people.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to an abnormal behavior intelligent analysis method and system based on multi-source data fusion in an examination room. Background Art

[0002] In recent years, the use of multi-source data fusion technology for examination room monitoring and abnormal behavior analysis has gradually become a research hotspot. By collecting and analyzing video and audio data, the behavior of examinees can be comprehensively monitored, thereby improving the safety and fairness of the examination room. However, existing technologies have some defects and shortcomings in identifying abnormal behavior in examination rooms. First, existing monitoring systems often rely only on a single data source and lack multi-dimensional information fusion, resulting in insufficient recognition capabilities for complex behaviors. Second, existing technologies have limitations in behavioral feature extraction and acoustic feature analysis, and are unable to effectively capture subtle abnormal behavior changes, affecting the accuracy of recognition. Finally, existing abnormal behavior recognition methods are mostly static analysis, which fails to consider the spatial relationship and dynamic changes between examinees, resulting in weak recognition capabilities for collaborative cheating behavior. Summary of the Invention

[0003] The embodiments of the present invention provide a method and system for intelligent analysis of abnormal behavior by fusing multi-source data in an examination room, which can solve the problems in the prior art.

[0004] According to a first aspect of the embodiments of the present invention,

[0005] Provides an intelligent analysis method for abnormal behavior by integrating multi-source data in the examination room, including:

[0006] Collect video and audio data from the examination room, perform multi-view skeleton point fusion and sound source localization on the detection area corresponding to each seat number, and extract behavioral feature sequences and acoustic features;

[0007] A hidden Markov model is constructed based on the behavioral feature sequence and acoustic features. Kernel density estimation is used to calculate the distribution difference between the answering state and the thinking state to obtain a dynamic adaptive weight. The consistency of the state behavior is evaluated by the matching degree between the state smoothed posterior probability and the personalized normal behavior model. The probability distribution entropy value is combined to identify abnormal behavior within a sliding time window.

[0008] A spatial weight coefficient is calculated based on the physical distance between the seat number identified as having abnormal behavior and the surrounding seats. The behavioral feature sequence and acoustic features of the relevant seat number are subjected to wavelet decomposition to obtain multi-scale coefficients. The correlation between the seat numbers is established based on the multi-scale coefficients. The behavioral-acoustic correlation between the seat numbers at different time scales is recursively calculated. The spatial weight coefficient is combined with the spatial weight coefficient to identify abnormal behaviors with correlation and identify multi-person coordinated cheating behavior.

[0009] The detected cheating behavior will be recorded and warning information will be pushed in real time.

[0010] In an optional embodiment,

[0011] Extracting behavioral feature sequences and acoustic features includes:

[0012] Multiple cameras are used to collect video data, the confidence of human skeleton nodes is calculated to obtain node visibility weights and the optimal viewing angle is selected. Time series prediction and structural constraints are performed on the human skeleton nodes under the optimal viewing angle, and the constrained skeleton nodes are weightedly fused with the corresponding skeleton nodes under other viewing angles to obtain the fusion position.

[0013] Based on the fusion position, a spatiotemporal graph structure is constructed and action features are extracted. The action features are temporally segmented and mapped to preset test scenario action primitives to obtain action semantic features. Based on the fusion position, the three-dimensional head angle and gaze center point are calculated, an attention distribution map is constructed, and temporal tracking is performed to obtain an attention transfer sequence. The action semantic features and the attention transfer sequence are input into a long short-term memory network to obtain an intention state probability distribution.

[0014] Calculating the similarity between the action semantic feature and the attention transfer sequence to obtain a modality consistency score, calculating the feature fusion weight based on the modality consistency score, performing weighted fusion on the action semantic feature, the attention transfer sequence, and the intention state probability distribution, and applying a temporal smoothing constraint to obtain a behavior feature sequence;

[0015] For the collected audio data, sound source localization is performed based on cross-correlation delay estimation and sound source enhancement algorithm, and the localization result is associated with the seat number; short-time energy calculation and double-threshold speech segment detection are performed on the associated audio data, and volume characteristics and duration characteristics are extracted as acoustic features.

[0016] In an optional embodiment,

[0017] A hidden Markov model is constructed based on the behavioral feature sequence and acoustic features. Kernel density estimation is used to calculate the distribution difference between the answering and thinking states to obtain dynamic adaptive weights. The consistency of state behavior is evaluated by the matching degree between the state smoothed posterior probability and the personalized normal behavior model. Abnormal behavior recognition is performed within a sliding time window in combination with the probability distribution entropy value, including:

[0018] Normalizing the behavioral feature sequence and the acoustic feature to obtain a normalized feature sequence; constructing a hidden Markov model based on the normalized feature sequence, using kernel density estimation and KL divergence to calculate the state distribution difference, and obtaining dynamic adaptive weights through Sigmoid mapping to construct a time-varying state transition probability matrix; in the hidden Markov model, the state smoothed posterior probability is calculated using the product of forward variables and backward variables, the forward variable is recursively obtained by multiplying the state emission probability and the time-varying state transition probability, and the backward variable is recursively obtained by multiplying the state emission probability at the next moment and the time-varying state transition probability;

[0019] Based on the normalized feature sequence collected during the first answering cycle after the start of the exam, a personalized normal behavior model of the examinee is constructed, and the matching degree between the state smoothed posterior probability and the state parameters of the personalized normal behavior model is calculated to obtain a state-behavior consistency score;

[0020] The probability distribution entropy value of the smoothed posterior probability of the state is calculated within a sliding window of multiple time scales, and the abnormal judgment is adaptively corrected in combination with the test time process and the behavioral baseline of the surrounding seats. When the abnormal scores of multiple consecutive time scales exceed the adaptive abnormal threshold, it is determined to be abnormal behavior.

[0021] In an optional embodiment,

[0022] A hidden Markov model is constructed based on the normalized feature sequence. The kernel density estimation and KL divergence are used to calculate the state distribution difference. The dynamic adaptive weight is obtained through Sigmoid mapping to construct the time-varying state transition probability matrix, including:

[0023] Based on the normalized feature sequence, a hidden Markov model including answering state and thinking state is constructed, and the state indicator function is used to mark the state of the examinee's answering behavior sequence. The feature samples marked as answering state are used to construct an answering state sample set, and the feature samples marked as thinking state are used to construct a thinking state sample set.

[0024] Using a kernel density estimation method to calculate the feature distribution density functions of the answering state sample set and the thinking state sample set respectively, calculating the KL divergence based on the feature distribution density function to obtain the feature distribution difference, and normalizing the feature distribution difference to obtain the normalized difference;

[0025] Calculating a dynamic adaptive weight based on the normalized difference, wherein the dynamic adaptive weight maps the normalized difference through a Sigmoid mapping function;

[0026] The dynamic adaptive weights are used to construct a time-varying state transition probability matrix, wherein the state transition probability is obtained by weighted calculation of the basic transition probability and the conditional transition probability based on the current observation by the dynamic adaptive weights, and the time-varying state transition probability matrix is used for subsequent state sequence estimation.

[0027] In an optional embodiment,

[0028] The probability distribution entropy of the smoothed posterior probability of the state is calculated within a sliding window of multiple time scales. Adaptive correction is performed on the abnormality judgment based on the test time process and the surrounding seat behavior baseline. When the abnormality scores of multiple consecutive time scales exceed the adaptive abnormality threshold, it is determined to be abnormal behavior including:

[0029] The probability distribution entropy of the smoothed posterior probability of the state is calculated in sliding windows of three time scales: short-term, medium-term and long-term. The short-term window is used to capture sudden anomalies, the medium-term window is used to detect persistent anomalies, and the long-term window is used to evaluate the overall anomaly trend.

[0030] The current exam time process is calculated based on the exam start time, and the exam process is divided into three stages: the beginning, the middle, and the end. Different abnormal scoring weights are set for the probability distribution entropy values of different stages.

[0031] Obtain the state smoothed posterior probability of other seats within a preset range around the determined seat number, calculate the mean of the probability distribution entropy value of the local area as the behavior baseline, and use the behavior baseline to correct the state-behavior consistency score;

[0032] The fusion anomaly score is calculated based on the probability distribution entropy value of different time scales, the test stage weight and the corrected state-behavior consistency score; the adaptive anomaly threshold is calculated based on the fusion anomaly score in the first answering cycle. When the fusion anomaly scores of three consecutive time scales exceed the corresponding adaptive anomaly threshold, it is judged as abnormal behavior.

[0033] In an optional embodiment,

[0034] A spatial weight coefficient is calculated based on the physical distance between the seat number determined to have abnormal behavior and the surrounding seats. The behavioral feature sequence and acoustic features of the relevant seat number are subjected to wavelet decomposition to obtain multi-scale coefficients. The correlation between the seat numbers is established based on the multi-scale coefficients. The behavioral-acoustic correlation between the seat numbers at different time scales is recursively calculated. The spatial weight coefficient is used to identify abnormal behaviors with correlation. Identifying multi-person coordinated cheating behavior includes:

[0035] Calculating a spatial weight coefficient based on the physical distance between the seat number determined to be abnormal behavior and the surrounding seats, wherein the spatial weight coefficient is obtained by mapping the physical distance between seats using a Gaussian kernel function and normalizing it;

[0036] Performing wavelet decomposition on the behavioral feature sequence and the acoustic feature sequence of the relevant seat numbers to obtain multi-scale coefficients, wherein the multi-scale coefficients include the approximate coefficient and detail coefficient of the behavioral feature sequence and the approximate coefficient and detail coefficient of the acoustic feature sequence; calculating the mutual correlation coefficients between the seat numbers at different time scales based on the multi-scale coefficients, wherein the mutual correlation coefficients are obtained by dividing the covariance of the multi-scale coefficients of the behavioral feature sequence and the multi-scale coefficients of the acoustic feature sequence by the product of the standard deviations; performing weighted fusion on the mutual correlation coefficients to obtain a fused correlation degree, and using the fused correlation degree as the initial correlation degree;

[0037] An initial correlation relationship between seat numbers is constructed based on the initial correlation, and the product of the spatial weight coefficient and the initial correlation is used as the weighted correlation between seat numbers. A correlation threshold is determined based on the fluctuation range of the weighted correlation in the first answering cycle after the start of the exam. When the weighted correlation exceeds the correlation threshold and the duration exceeds a preset time window, it is determined that preliminary collaborative cheating behavior exists between the relevant seat numbers.

[0038] In an optional embodiment,

[0039] Further identification of the preliminary coordinated cheating behavior includes:

[0040] A time-varying graph structure with seat numbers as nodes is constructed. The behavioral feature sequences and acoustic features corresponding to the seat numbers are used as node features. The edge weights are obtained by adaptively weighting the physical distance, behavioral similarity, and acoustic correlation between seat numbers.

[0041] For each seat number, a state vector including behavioral state parameters and acoustic state parameters is established. A state transfer equation is constructed based on the state vector and the edge weights. The state transfer equation includes an individual behavior evolution term and an interaction influence term. The individual behavior evolution term is used to calculate the state transfer probability distribution. The interaction influence term is obtained by calculating the difference of the state vector and performing nonlinear mapping with the edge weights. The group synchronization index is obtained based on the state influence.

[0042] The correlation between the behavior switching sequences of seat numbers is calculated as the temporal collaborative feature, the consistency of the state transition probability distribution is calculated as the state collaborative feature, the density of neighborhoods with similar behaviors is calculated as the group aggregation feature, and the behavior triggering relationship is calculated as the causal collaborative feature.

[0043] The temporal collaborative features, state collaborative features, group aggregation features and causal collaborative features are multi-level integrated to obtain collaborative behavior features, and collaborative cheating behavior is determined based on the collaborative behavior features and the group synchronization index.

[0044] According to a second aspect of the embodiments of the present invention,

[0045] Provides an intelligent analysis system for abnormal behavior by integrating multi-source data in the examination room, including:

[0046] The first unit is used to collect video and audio data in the examination room, perform multi-view skeleton point fusion and sound source localization on the detection area corresponding to each seat number, and extract behavioral feature sequences and acoustic features;

[0047] The second unit is used to construct a hidden Markov model based on the behavioral feature sequence and acoustic features, use kernel density estimation to calculate the distribution difference between answering and thinking states to obtain dynamic adaptive weights, evaluate state-behavior consistency by matching the state smoothed posterior probability with the personalized normal behavior model, and identify abnormal behavior within a sliding time window in combination with the probability distribution entropy value;

[0048] The third unit is used to calculate a spatial weight coefficient based on the physical distance between the seat number determined to have abnormal behavior and the surrounding seats, perform wavelet decomposition on the behavioral feature sequence and acoustic features of the relevant seat number to obtain multi-scale coefficients, establish a correlation between the seat numbers based on the multi-scale coefficients, recursively calculate the behavioral-acoustic correlation between the seat numbers at different time scales, and use the spatial weight coefficients to identify correlated abnormal behaviors and identify multi-person coordinated cheating behavior;

[0049] The fourth unit is used to record detected cheating behaviors and push warning information in real time.

[0050] According to a third aspect of the embodiments of the present invention,

[0051] An electronic device is provided, comprising:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0055] According to a fourth aspect of the embodiments of the present invention,

[0056] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0057] Beneficial effects of this application:

[0058] Improve the comprehensiveness and accuracy of examination room monitoring. Through multi-source data fusion technology, it is possible to comprehensively collect behavioral and acoustic characteristics in the examination room, providing a richer data basis for the identification of abnormal behavior.

[0059] Enhance the intelligence and dynamic adaptability of abnormal behavior recognition. By using the hidden Markov model and kernel density estimation method, it is possible to analyze the examinee's answer and thinking state in real time, and dynamically adjust the recognition weight, thereby improving the accuracy and sensitivity of recognition.

[0060] It promotes the timely discovery and early warning of cheating behaviors. By calculating the spatial weight coefficient and behavior-acoustic correlation between seats, it can effectively identify the coordinated cheating behaviors of multiple people and push early warning information in real time to ensure the fairness and impartiality of the examination room. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the process of the abnormal behavior intelligent analysis method based on multi-source data fusion in the examination room according to an embodiment of the present invention;

[0062] Figure 2 It is a multi-time-scale sliding window entropy fusion detection map;

[0063] Figure 3 This is an analysis chart of group synchronization index changes and multi-dimensional collaborative characteristics. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0065] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of an abnormal behavior intelligent analysis method based on multi-source data fusion in an examination room according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] Collect video and audio data from the examination room, perform multi-view skeleton point fusion and sound source localization on the detection area corresponding to each seat number, and extract behavioral feature sequences and acoustic features;

[0068] A hidden Markov model is constructed based on the behavioral feature sequence and acoustic features. Kernel density estimation is used to calculate the distribution difference between the answering state and the thinking state to obtain a dynamic adaptive weight. The consistency of the state behavior is evaluated by the matching degree between the state smoothed posterior probability and the personalized normal behavior model. The probability distribution entropy value is combined to identify abnormal behavior within a sliding time window.

[0069] A spatial weight coefficient is calculated based on the physical distance between the seat number identified as having abnormal behavior and the surrounding seats. The behavioral feature sequence and acoustic features of the relevant seat number are subjected to wavelet decomposition to obtain multi-scale coefficients. The correlation between the seat numbers is established based on the multi-scale coefficients. The behavioral-acoustic correlation between the seat numbers at different time scales is recursively calculated. The spatial weight coefficient is combined with the spatial weight coefficient to identify abnormal behaviors with correlation and identify multi-person coordinated cheating behavior.

[0070] The detected cheating behavior will be recorded and warning information will be pushed in real time.

[0071] In an optional embodiment, extracting the behavioral feature sequence and the acoustic feature includes:

[0072] Multiple cameras are used to collect video data, the confidence of human skeleton nodes is calculated to obtain node visibility weights and the optimal viewing angle is selected. Time series prediction and structural constraints are performed on the human skeleton nodes under the optimal viewing angle, and the constrained skeleton nodes are weightedly fused with the corresponding skeleton nodes under other viewing angles to obtain the fusion position.

[0073] Based on the fusion position, a spatiotemporal graph structure is constructed and action features are extracted. The action features are temporally segmented and mapped to preset test scenario action primitives to obtain action semantic features. Based on the fusion position, the three-dimensional head angle and gaze center point are calculated, an attention distribution map is constructed, and temporal tracking is performed to obtain an attention transfer sequence. The action semantic features and the attention transfer sequence are input into a long short-term memory network to obtain an intention state probability distribution.

[0074] Calculating the similarity between the action semantic feature and the attention transfer sequence to obtain a modality consistency score, calculating the feature fusion weight based on the modality consistency score, performing weighted fusion on the action semantic feature, the attention transfer sequence, and the intention state probability distribution, and applying a temporal smoothing constraint to obtain a behavior feature sequence;

[0075] For the collected audio data, sound source localization is performed based on cross-correlation delay estimation and sound source enhancement algorithm, and the localization result is associated with the seat number; short-time energy calculation and double-threshold speech segment detection are performed on the associated audio data, and volume characteristics and duration characteristics are extracted as acoustic features.

[0076] For example, video data is collected by multiple cameras, each of which is placed in a different position in the examination room, such as the four corners and the center top of the classroom, to ensure comprehensive coverage of the examinee's behavior. The human skeleton detection algorithm is applied to the video data from each perspective to obtain human skeleton nodes, including 25 key points such as the head, neck, shoulder, elbow, and wrist. To determine the optimal perspective, the confidence of each skeleton node is calculated. The confidence calculation takes into account the node detection score and its stability, and the value range is 0-1. For example, the confidence of the wrist node of a candidate's right hand is 0.92 under perspective A, 0.78 under perspective B, and 0.45 under perspective C. Then perspective A is selected as the optimal perspective for this node. The skeleton nodes under the optimal perspective are predicted in time series, and the Kalman filter method is used to predict the possible position of the node in the next frame. If the deviation between the predicted position and the actual detected position exceeds a threshold (such as 15 pixels), the structural constraint correction is initiated. The structural constraint is based on the principle of constant length of the human skeleton and limits the distance change between adjacent skeleton nodes to no more than 5%. For example, the length of the upper arm should remain relatively stable in the sequence. If abnormal elongation occurs, it is corrected according to the historical average value. The constrained skeleton nodes are weightedly fused with the corresponding nodes from other perspectives. The fusion weight is proportional to the node visibility weight. For example, if the visibility weights of a key point from three perspectives are 0.9, 0.7, and 0.4, respectively, the corresponding weight ratios when calculating the fusion position are 45%, 35%, and 20%.

[0077] A spatiotemporal graph structure is constructed based on the fusion positions. The graph nodes are skeletal keypoints, and the edges include spatial connections (skeletal connections) and temporal connections (connections between the same keypoint in adjacent frames). A graph convolutional network is applied to this graph structure to extract action features with a feature dimension of 128. A sliding window (window size of 30 frames, step size of 15 frames) is applied to the action feature sequence for temporal segmentation. A classifier is then used to map each segment to a preset test scenario action primitive, such as "writing," "raising the head," "turning the head," "raising the hand," and other 12 basic actions. The mapping results form an action semantic feature with a dimension of 12, where each dimension represents the confidence level of the corresponding action primitive.

[0078] Based on the fused position, the three-dimensional head angle is calculated, including pitch (nodding up and down), yaw (turning the head left and right), and roll (head tilt). Head orientation is determined by identifying facial landmarks, and then the gaze center is predicted using a field of view model. The gaze center is represented as (x, y, z) in the exam room coordinate system. For example, (150, 200, 50) centimeters represents the position on the table where the examinee is likely to be looking.

[0079] When constructing the attention distribution map, a Gaussian distribution is constructed with the center of gaze as the center and the radius as the visual cone range (approximately 30 degrees of visual angle). The intensity decays as the distance from the center is approached. The attention distribution map is tracked in time, and the movement trajectory of the center point within 5 seconds is recorded to generate an attention transfer sequence. For example, the scanning trajectory from the upper left corner to the upper right corner and then to the lower left corner of the test paper can be represented as a continuous sequence of coordinate points. The action semantic features and attention transfer sequence are input into the long short-term memory network (the network contains 64 hidden units) to output the intention state probability distribution. The intention state includes 8 types of test behavior intentions such as "focusing on answering questions", "thinking", "checking references", and "communicating with others". The probability distribution represents the possibility of each intention.

[0080] The modal consistency score is calculated by calculating the similarity between the semantic features of the action and the sequence of attention shifts. This similarity is calculated using the cosine similarity method, which maps the two features to the same dimensional space and then calculates the cosine of the angle between the vectors, with a value ranging from -1 to 1. For example, the attention distribution of the "head down to write" action is highly consistent with the attention distribution of the test paper, with a similarity of over 0.85. However, the attention distribution of the "head up" action is inconsistent with the attention distribution of the test paper, with a similarity as low as 0.3.

[0081] Feature fusion weights are calculated based on the modal consistency score. Higher consistency scores correspond to larger feature weights. For example, when the consistency score is 0.9, the action feature weight can be set to 0.5, the attention feature weight to 0.3, and the intention state weight to 0.2. When the consistency score is 0.4, the weights can be adjusted to 0.3, 0.3, and 0.4. The action semantic features, attention transfer sequence, and intention state probability distribution are weighted and fused to obtain the behavior feature vector. A temporal smoothing constraint is imposed on the behavior feature sequence, using an exponentially weighted moving average method. The feature value calculation of the current frame takes into account the influence of historical frames to avoid drastic feature fluctuations. The smoothing parameter is set to 0.8, which means that the current frame retains 20% of the original features and integrates 80% of the historical features.

[0082] In terms of acoustic feature extraction, classroom audio data is collected using multiple microphone arrays. These arrays can be placed at the front, back, and sides of the classroom, with each array containing 4-8 microphones. The sound source location is calculated using a cross-correlation delay estimation algorithm. The cross-correlation function is calculated for the signals received by the two microphones, and the time difference corresponding to the cross-correlation peak is found. The sound source direction is then calculated based on the speed of sound (approximately 340 m / s) and the microphone positions. A sound source enhancement algorithm is used to further improve localization accuracy. A beamformer is constructed to enhance the signal from sound sources in specific directions and suppress interference from other directions. The localization results are associated with seat numbers. For example, if the sound source is located at coordinates (250, 180) cm, the seating distribution table is searched to determine the corresponding seat number for row 3 and column 5. The associated audio data is preprocessed, including denoising and reverberation reduction. Using short-time energy calculation, the audio signal is framed (25 milliseconds in length, with a 10 millisecond frame shift). The energy mean of each frame is calculated and normalized to obtain a volume feature with a value range of 0-1. For example, the volume characteristic value of a normal answer to a question is approximately 0.4-0.7, while that of a whispered conversation is approximately 0.1-0.3. A dual-threshold speech segment detection algorithm is used to identify valid speech segments, setting a high threshold (e.g., energy value 0.4) and a low threshold (e.g., energy value 0.2). When the energy value exceeds the high threshold, it is marked as the start of speech; when it remains below the low threshold for more than 100 milliseconds, it is marked as the end of speech. The duration of each speech segment is calculated. For example, a continuous explanation of a question may last 20-60 seconds, while a brief response may only last 2-5 seconds.

[0083] The present invention improves the accuracy and robustness of human posture recognition through multi-view skeleton point fusion and adaptive confidence weighting, and solves the problems of occlusion and perspective change. The multimodal feature fusion mechanism combining action semantic features, attention transfer sequence and intention state achieves a fine-grained understanding of the examinee's behavior. The rationality of feature fusion and the stability of behavior recognition are guaranteed through modal consistency evaluation and temporal smoothing constraints. At the same time, multi-microphone array collaborative positioning and dual-threshold voice detection provide reliable acoustic features, which overall improve the accuracy and real-time performance of abnormal behavior detection in the examination room.

[0084] In an optional embodiment, a hidden Markov model is constructed based on the behavioral feature sequence and acoustic features, kernel density estimation is used to calculate the distribution difference between the answering state and the thinking state to obtain a dynamic adaptive weight, the state behavior consistency is evaluated by the matching degree between the state smoothed posterior probability and the personalized normal behavior model, and abnormal behavior recognition is performed within a sliding time window in combination with the probability distribution entropy value, including:

[0085] Normalizing the behavioral feature sequence and the acoustic feature to obtain a normalized feature sequence; constructing a hidden Markov model based on the normalized feature sequence, using kernel density estimation and KL divergence to calculate the state distribution difference, and obtaining dynamic adaptive weights through Sigmoid mapping to construct a time-varying state transition probability matrix; in the hidden Markov model, the state smoothed posterior probability is calculated using the product of forward variables and backward variables, the forward variable is recursively obtained by multiplying the state emission probability and the time-varying state transition probability, and the backward variable is recursively obtained by multiplying the state emission probability at the next moment and the time-varying state transition probability;

[0086] Based on the normalized feature sequence collected during the first answering cycle after the start of the exam, a personalized normal behavior model of the examinee is constructed, and the matching degree between the state smoothed posterior probability and the state parameters of the personalized normal behavior model is calculated to obtain a state-behavior consistency score;

[0087] The probability distribution entropy value of the smoothed posterior probability of the state is calculated within a sliding window of multiple time scales, and the abnormal judgment is adaptively corrected in combination with the test time process and the behavioral baseline of the surrounding seats. When the abnormal scores of multiple consecutive time scales exceed the adaptive abnormal threshold, it is determined to be abnormal behavior.

[0088] For example, the behavioral feature sequence and acoustic feature are normalized by collecting behavioral feature data and acoustic feature data over a period of time, calculating the minimum and maximum values of each feature, and then using these minimum and maximum values to convert each feature value to a value between 0 and 1, thereby obtaining a normalized feature sequence.

[0089] After obtaining the normalized feature sequences, a hidden Markov model is constructed based on these sequences. A hidden Markov model consists of states, observations, and transition probabilities. The model's state and observation sets are defined, and the states are initialized using the normalized feature sequences. Kernel density estimation is used to calculate the observation probability distribution for each state, thereby obtaining the state's emission probability. KL divergence is also used to assess the distribution differences between different states. Dynamically adaptive weights are obtained by calculating the state distribution differences. These weights reflect the importance of different states in the model. In specific implementation, the Sigmoid mapping method can be used to convert state distribution differences into weights, ensuring that the weights vary between 0 and 1. Based on these dynamic adaptive weights, a time-varying state transition probability matrix is constructed to reflect the transition relationships between states.

[0090] In a hidden Markov model, the product of the forward and backward variables is used to calculate the smoothed posterior probability of the state. The forward variable represents the probability that the system is in a certain state given a given observation sequence. It is obtained recursively by multiplying the state emission probability and the time-varying state transition probability. The backward variable represents the probability that the system is in a certain state given a future observation sequence. It is obtained recursively by multiplying the state emission probability at the next moment and the time-varying state transition probability. Multiplying the forward and backward variables yields the smoothed posterior probability of the state, which represents the probability that the system is in a certain state given the entire observation sequence.

[0091] To build a personalized normal behavior model for a test-taker, we need to collect behavioral characteristic data from multiple test-takers in similar environments to form a baseline representing normal behavior. By analyzing this data, we extract state parameters and calculate their matching with the smoothed posterior probability of the state. This matching is calculated by comparing the similarity between the smoothed posterior probability of the state and the state parameters of the personalized normal behavior model to obtain a state-behavior consistency score.

[0092] The entropy of the probability distribution of the smoothed posterior probability of the state is calculated within a sliding window at multiple time scales. The entropy reflects the uncertainty of the probability distribution; higher entropy indicates greater uncertainty about the state. Adaptive correction of anomaly determinations is performed based on the exam timeline and the behavioral baseline of surrounding test takers. Specifically, the anomaly determination threshold can be dynamically adjusted based on the exam progress and the behavioral patterns of surrounding test takers. When the anomaly score exceeds the adaptive anomaly threshold for multiple consecutive time scales, abnormal behavior is determined. This process involves monitoring and analyzing the smoothed posterior probability of the state, ensuring accurate identification of abnormal behavior at different time scales.

[0093] Existing rule-based anomaly detection methods use fixed thresholds and fail to account for individual differences among examinees. This paper introduces kernel density estimation and KL divergence to calculate state distribution variability. It constructs dynamic adaptive weights through sigmoid mapping to dynamically adjust state transition probabilities, improving the adaptability of state modeling. It also constructs a personalized model based on the normal sequence of test-taking behaviors at the initial exam stage, providing a personalized benchmark for anomaly detection. It also uses a multi-timescale sliding window to analyze the entropy of state probability distributions, and adaptively corrects them based on the exam progress and surrounding behavioral baselines, enhancing the reliability of anomaly detection.

[0094] In an optional embodiment, a hidden Markov model is constructed based on the normalized feature sequence, kernel density estimation and KL divergence are used to calculate the state distribution difference, and a dynamic adaptive weight is obtained through Sigmoid mapping to construct a time-varying state transition probability matrix, including:

[0095] Based on the normalized feature sequence, a hidden Markov model including answering state and thinking state is constructed, and the state indicator function is used to mark the state of the examinee's answering behavior sequence. The feature samples marked as answering state are used to construct an answering state sample set, and the feature samples marked as thinking state are used to construct a thinking state sample set.

[0096] Using a kernel density estimation method to calculate the feature distribution density functions of the answering state sample set and the thinking state sample set respectively, calculating the KL divergence based on the feature distribution density function to obtain the feature distribution difference, and normalizing the feature distribution difference to obtain the normalized difference;

[0097] Calculating a dynamic adaptive weight based on the normalized difference, wherein the dynamic adaptive weight maps the normalized difference through a Sigmoid mapping function;

[0098] The dynamic adaptive weights are used to construct a time-varying state transition probability matrix, wherein the state transition probability is obtained by weighted calculation of the basic transition probability and the conditional transition probability based on the current observation by the dynamic adaptive weights, and the time-varying state transition probability matrix is used for subsequent state sequence estimation.

[0099] Exemplarily, a hidden Markov model including the answering state and the thinking state is constructed based on the normalized feature sequence. First, a state indicator function is defined, which determines the state of the examinee based on a combination of behavioral features and acoustic features: when a continuous writing action is detected and the head is looking at the test paper area, it is determined to be the answering state; when the head is raised, the pen is stopped, and there are thinking expression features, it is determined to be the thinking state. The feature sequence is traversed and marked by the state indicator function to obtain a sample sequence with state labels. The feature samples marked as the answering state are then classified into the answering state sample set, and the feature samples marked as the thinking state are classified into the thinking state sample set. These two sample sets are used for subsequent distribution feature calculations.

[0100] The kernel density estimation method was used to calculate the feature distribution density functions for the answering state sample set and the thinking state sample set. For each feature dimension, a Gaussian kernel function was selected as the kernel function, and kernel density estimation was performed on the sample points. The kernel bandwidth parameter was determined through cross-validation and was set to 0.15 in this example. The feature distribution density function was obtained by applying the kernel function to each sample point in the sample set and accumulating the results. Similarly, the same method was applied to the thinking state sample set to obtain the feature distribution density function of the thinking state.

[0101] The KL divergence is calculated based on the feature distribution density function to obtain the feature distribution difference. The discretization method is used to divide the feature space into a finite number of grid points, and the density function value is calculated at each grid point, and then the KL divergence is calculated. Taking the data of a certain candidate as an example, the feature dimension is 5, and the calculated KL divergence value is 3.27. The feature distribution difference is normalized, and by analyzing historical data, it is determined that the theoretical maximum value of the KL divergence is 10 and the minimum value is 0. The KL divergence value is mapped to the range of 0 to 1 to obtain the normalized difference.

[0102] The dynamic adaptive weight is calculated based on the normalized difference. A sigmoid mapping function is used to map the normalized difference, and parameters are introduced to adjust the function shape. These parameters were determined through experimental analysis so that when the normalized difference is 0.5, the dynamic adaptive weight is approximately 0.5. For the normalized difference of 0.327 in this example, the calculated dynamic adaptive weight is 0.281.

[0103] A time-varying state transition probability matrix is constructed using dynamic adaptive weights. The state transition probability is calculated by weighting two probabilities using dynamic adaptive weights: a basic transition probability, calculated based on historical data statistics, representing the transition probability without considering the current observation; and a conditional transition probability based on the current observation, calculated using a classifier, representing the transition probability under the current observation. By weighting these two probabilities, the final time-varying state transition probability matrix is obtained. This matrix is used for subsequent state sequence estimation to accurately identify and predict the test-taker's behavioral state.

[0104] This method uses kernel density estimation to calculate state distribution characteristics, overcoming the limitations of parametric models in describing complex behavioral distributions. By dynamically evaluating state differences through KL divergence and performing normalization, it enables adaptive tracking of changes in individual test-taker behavior patterns.

[0105] In an optional embodiment, the probability distribution entropy value of the state smoothed posterior probability is calculated within a sliding window of multiple time scales, and the abnormality determination is adaptively corrected in combination with the test time process and the surrounding seat behavior baseline. When the abnormality scores of multiple consecutive time scales exceed the adaptive abnormality threshold, it is determined that the behavior is abnormal, including:

[0106] The probability distribution entropy of the smoothed posterior probability of the state is calculated in sliding windows of three time scales: short-term, medium-term and long-term. The short-term window is used to capture sudden anomalies, the medium-term window is used to detect persistent anomalies, and the long-term window is used to evaluate the overall anomaly trend.

[0107] The current exam time process is calculated based on the exam start time, and the exam process is divided into three stages: the beginning, the middle, and the end. Different abnormal scoring weights are set for the probability distribution entropy values of different stages.

[0108] Obtain the state smoothed posterior probability of other seats within a preset range around the determined seat number, calculate the mean of the probability distribution entropy value of the local area as the behavior baseline, and use the behavior baseline to correct the state-behavior consistency score;

[0109] The fusion anomaly score is calculated based on the probability distribution entropy value of different time scales, the test stage weight and the corrected state-behavior consistency score; the adaptive anomaly threshold is calculated based on the fusion anomaly score in the first answering cycle. When the fusion anomaly scores of three consecutive time scales exceed the corresponding adaptive anomaly threshold, it is judged as abnormal behavior.

[0110] For example, sliding windows with three time scales are defined: short-term, medium-term, and long-term. The short-term window covers the first ten minutes after the exam begins, the medium-term window covers the first hour, and the long-term window covers the entire duration of the exam. The probability distribution entropy of the smoothed posterior probability of the state within each window is derived by analyzing the state data within each time window. Specifically, the state data within each time window is collected and smoothed to eliminate the influence of noise and outliers. Within the short-term window, sudden anomalies are primarily detected. By monitoring the examinee's behavioral data in real time, such as answering speed and frequent hand-raising, the smoothed posterior probability of the state within that window is calculated, and the probability distribution entropy of the window is calculated based on this data. If the entropy value is significantly higher than the average level of historical data, it can be identified as a potential sudden anomaly. Within the medium-term window, persistent abnormal behavior is detected. By analyzing the examinee's behavioral patterns within that window, such as changes in error rates and increases in pause time, the smoothed posterior probability of the state is calculated, and the probability distribution entropy value of the window is derived. If the entropy value remains consistently above a preset baseline, it can be identified as a persistent anomaly. The long-term window is used to assess overall abnormal trends. By aggregating behavioral data from the entire exam process, the smoothed posterior probability of the state within the long-term window and its probability distribution entropy are calculated. If this entropy shows a clear upward trend, it can be considered that there is a risk of overall abnormal behavior.

[0111] The current exam timeline is calculated based on the exam start time, and the exam process is divided into three phases: the beginning, middle, and end. For example, the beginning phase is the first fifteen minutes after the exam begins, the end phase is the last fifteen minutes, and the middle phase is the rest of the time. Different anomaly scoring weights are assigned based on the probability distribution entropy values at different phases. For example, in the beginning phase, candidates are less adaptable and may experience high fluctuations, so the weight for this phase is set higher; in the end phase, candidates have reached a stable state, so the weight is set lower.

[0112] Obtain the smoothed posterior probabilities of the states of other seats within a preset range around the determined seat number. By analyzing the behavioral data of the surrounding seats, the mean entropy value of the probability distribution in the local area is calculated as the behavioral baseline. This baseline is used to calibrate the state-behavior consistency score. Specifically, if a candidate's behavior differs significantly from that of the surrounding candidates, their state-behavior consistency score will be adjusted to reflect the candidate's unusual behavior.

[0113] A fused anomaly score is calculated based on the probability distribution entropy values at different time scales, the exam stage weights, and the corrected state-behavior consistency score. This fused anomaly score comprehensively considers performance in short-term, medium-term, and long-term windows, providing a more comprehensive reflection of the examinee's behavior. An adaptive anomaly threshold is calculated based on the fused anomaly score during the first test-taking cycle. If the fused anomaly scores for three consecutive time scales exceed the corresponding adaptive anomaly threshold, the behavior is considered abnormal.

[0114] like Figure 2The figure below shows the multi-timescale sliding window entropy fusion detection. The upper part shows the entropy changes of the smoothed posterior probability of the state within the short-term (10-minute), medium-term (30-minute), and long-term (60-minute) windows. It can be seen that between the 25th and 40th minutes, the entropy values of all three timescales increase significantly and exceed the anomaly threshold, but exhibit different response characteristics: the short-term window (solid line) has the fastest response to anomalies and the largest fluctuations, with the entropy value reaching its highest point within the anomaly region; the medium-term window (dashed line) has a response speed and fluctuation amplitude between the short-term and long-term windows; and the long-term window (dash-dotted line) has a more gradual response and a smoother entropy curve. The anomaly threshold line marked in the figure clearly demarcates the boundary between normal and anomalous behavior. All three curves exceed this threshold within the anomaly region, indicating that the system detects anomalous behavior at different timescales. The lower part shows the anomaly score and detection results after fusing the entropy values of the three timescales. The system successfully detected the onset of abnormal behavior around the 25th minute and confirmed its end around the 43rd minute. The anomaly score curve is significantly lower than the adaptive anomaly threshold within the abnormal region, demonstrating the system's effective identification of abnormal conditions. Black dots indicate the detected abnormal event points, while the vertical dashed lines indicate the start and end times of the anomalies. The figure shows that the fusion method effectively reduces false positives while maintaining rapid response. Compared to a single time scale, the multi-scale fusion method achieves higher detection rates for both sudden and persistent anomalies (92.8% and 95.3%, respectively), while keeping the false positive rate to 2.4% and the detection latency to less than 3 minutes. By integrating the entropy characteristics of three time scales, the system can simultaneously achieve rapid response to sudden cheating and reliable detection of persistent anomalies, significantly reducing the false positive rate while maintaining detection sensitivity, providing a more reliable abnormal behavior detection mechanism for exam room monitoring.

[0115] In an optional embodiment, a spatial weight coefficient is calculated based on the physical distance between the seat number determined to have abnormal behavior and the surrounding seats. Wavelet decomposition is performed on the behavioral feature sequence and acoustic features of the relevant seat number to obtain multi-scale coefficients. A correlation relationship between the seat numbers is established based on the multi-scale coefficients. The behavioral-acoustic correlation between the seat numbers at different time scales is recursively calculated. The spatial weight coefficient is used to identify abnormal behaviors with correlation. Identifying multi-person collaborative cheating behavior includes:

[0116] Calculating a spatial weight coefficient based on the physical distance between the seat number determined to be abnormal behavior and the surrounding seats, wherein the spatial weight coefficient is obtained by mapping the physical distance between seats using a Gaussian kernel function and normalizing it;

[0117] Performing wavelet decomposition on the behavioral feature sequence and the acoustic feature sequence of the relevant seat numbers to obtain multi-scale coefficients, wherein the multi-scale coefficients include the approximate coefficient and detail coefficient of the behavioral feature sequence and the approximate coefficient and detail coefficient of the acoustic feature sequence; calculating the mutual correlation coefficients between the seat numbers at different time scales based on the multi-scale coefficients, wherein the mutual correlation coefficients are obtained by dividing the covariance of the multi-scale coefficients of the behavioral feature sequence and the multi-scale coefficients of the acoustic feature sequence by the product of the standard deviations; performing weighted fusion on the mutual correlation coefficients to obtain a fused correlation degree, and using the fused correlation degree as the initial correlation degree;

[0118] An initial correlation relationship between seat numbers is constructed based on the initial correlation, and the product of the spatial weight coefficient and the initial correlation is used as the weighted correlation between seat numbers. A correlation threshold is determined based on the fluctuation range of the weighted correlation in the first answering cycle after the start of the exam. When the weighted correlation exceeds the correlation threshold and the duration exceeds a preset time window, it is determined that preliminary collaborative cheating behavior exists between the relevant seat numbers.

[0119] For example, the physical location coordinates of all seats are obtained based on the examination room seating layout information. When the system detects that a seat number exhibits abnormal behavior, the physical distance between that seat and the surrounding seats is calculated. For example, in an examination room, if seat number 5 is detected to be behaving abnormally, the physical distances between seat number 5 and the surrounding seats are calculated as follows: d(5, 1) = 3.2 meters, d(5, 2) = 2.8 meters, d(5, 6) = 1.5 meters, and d(5, 9) = 2.1 meters.

[0120] Physical distances are mapped to spatial weight coefficients using a Gaussian kernel function. For the physical distance d(i, j) between seat i and seat j, the spatial weight coefficient w(i, j) is calculated using the following steps: First, the Gaussian kernel mapping values are calculated. For the example above, the bandwidth parameter σ = 2.0 is selected, resulting in the following mapping results: w'(5, 1) = 0.60, w'(5, 2) = 0.67, w'(5, 6) = 0.89, and w'(5, 9) = 0.78. Then, normalization is performed to obtain the final spatial weight coefficients: w(5, 1) = 0.20, w(5, 2) = 0.23, w(5, 6) = 0.30, and w(5, 9) = 0.27.

[0121] The behavioral features and acoustic features collected for each relevant seat are preprocessed. Behavioral features mainly include the frequency of students' head movements, the angle of sight deviation, the frequency of page turning, etc. Acoustic features include sound energy, spectral features, etc. For example, in a certain exam, the behavioral feature sequences X(5), X(6) and acoustic feature sequences Y(5), Y(6) of seats 5 and 6 are sampled with a sampling rate of 2Hz and an observation window of 120 seconds, resulting in a feature sequence of length 240.

[0122] The acquired feature sequences were subjected to wavelet decomposition. Daubechies wavelet (db4) was used to perform a three-layer decomposition of the behavioral and acoustic feature sequences, respectively, to obtain multi-scale coefficients. Taking seat 5 as an example, the behavioral feature sequence decomposition yielded the approximate coefficient A3(X5) and detail coefficients D1(X5), D2(X5), and D3(X5); the acoustic feature sequence decomposition yielded the approximate coefficient A3(Y5) and detail coefficients D1(Y5), D2(Y5), and D3(Y5).

[0123] The cross-correlation coefficients between seat numbers at different time scales are calculated based on multi-scale coefficients. For seats 5 and 6, the cross-correlation coefficients between the behavioral and acoustic features at each scale are calculated separately: first, the covariance between the coefficients is calculated, and then divided by the product of the corresponding standard deviations. For example, at the third decomposition scale, the cross-correlation coefficient between the behavioral feature approximation coefficient A3(X5) and the acoustic feature approximation coefficient A3(Y6) is calculated as ρA3(5,6)=0.76. Similarly, the cross-correlation coefficients between the detail coefficients are: ρD3(5,6)=0.68, ρD2(5,6)=0.45, and ρD1(5,6)=0.32.

[0124] Perform a weighted fusion of the cross-correlation coefficients. Cross-correlation coefficients at different decomposition levels have different importance and are assigned different weights: wA3 = 0.4, wD3 = 0.3, wD2 = 0.2, and wD1 = 0.1. Calculate the fused correlation R(5, 6) = 0.4 × 0.76 + 0.3 × 0.68 + 0.2 × 0.45 + 0.1 × 0.32 = 0.63. Use this fused correlation as the initial correlation between seats 5 and 6.

[0125] Construct an initial correlation between seat numbers. For all relevant seat pairs, calculate the initial correlation and build a correlation matrix. For example, the initial correlations between seat 5 and its surrounding seats 1, 2, 6, and 9 are: R(5, 1) = 0.25, R(5, 2) = 0.31, R(5, 6) = 0.63, and R(5, 9) = 0.29.

[0126] Multiply the spatial weight coefficient by the initial correlation to obtain the weighted correlation. For the above example, the weighted correlations are: R'(5, 1) = 0.20 × 0.25 = 0.05, R'(5, 2) = 0.23 × 0.31 = 0.07, R'(5, 6) = 0.30 × 0.63 = 0.19, and R'(5, 9) = 0.27 × 0.29 = 0.08.

[0127] The correlation threshold is determined based on the fluctuation of the weighted correlation during the first test-taking period (typically the first 30 minutes). Analysis of historical data shows that under normal circumstances, the standard deviation of the weighted correlation is approximately 0.04, with 3 times the standard deviation being used as the upper limit of the fluctuation range. The correlation threshold is calculated as τ = μ + 3σ, where μ is the mean of the weighted correlation during the first test-taking period. For example, for seats 5 and 6, the mean of the weighted correlation during the first test-taking period is 0.07, and the standard deviation is 0.04. Therefore, the correlation threshold is set as τ(5, 6) = 0.07 + 3 × 0.04 = 0.19.

[0128] Continuously monitor the weighted correlation of each seat pair: When the weighted correlation exceeds a threshold and persists for longer than a preset time window (e.g., three consecutive minutes), the system determines that preliminary coordinated cheating has occurred between the relevant seats. For example, starting at the 45th minute of an exam, the weighted correlation between seats 5 and 6 remained above 0.22, exceeding the preset threshold of 0.19, and lasted for 4 minutes and 30 seconds, exceeding the preset time window of 3 minutes. The system then determined that preliminary coordinated cheating had occurred between seats 5 and 6.

[0129] In an optional implementation, further identifying the preliminary collaborative cheating behavior includes:

[0130] A time-varying graph structure with seat numbers as nodes is constructed. The behavioral feature sequences and acoustic features corresponding to the seat numbers are used as node features. The edge weights are obtained by adaptively weighting the physical distance, behavioral similarity, and acoustic correlation between seat numbers.

[0131] For each seat number, a state vector including behavioral state parameters and acoustic state parameters is established. A state transfer equation is constructed based on the state vector and the edge weights. The state transfer equation includes an individual behavior evolution term and an interaction influence term. The individual behavior evolution term is used to calculate the state transfer probability distribution. The interaction influence term is obtained by calculating the difference of the state vector and performing nonlinear mapping with the edge weights. The group synchronization index is obtained based on the state influence.

[0132] The correlation between the behavior switching sequences of seat numbers is calculated as the temporal collaborative feature, the consistency of the state transition probability distribution is calculated as the state collaborative feature, the density of neighborhoods with similar behaviors is calculated as the group aggregation feature, and the behavior triggering relationship is calculated as the causal collaborative feature.

[0133] The temporal collaborative features, state collaborative features, group aggregation features and causal collaborative features are multi-level integrated to obtain collaborative behavior features, and collaborative cheating behavior is determined based on the collaborative behavior features and the group synchronization index.

[0134] For example, a time-varying graph structure with seat numbers as nodes is constructed: each seat number in the examination room is treated as a node in the graph, and each node has two types of features: behavioral feature sequences and acoustic features. The edge weights of the graph are determined using an adaptive weighting mechanism, which comprehensively considers three factors: the physical distance between seat numbers, behavioral similarity, and acoustic correlation. Physical distance is calculated as the Euclidean distance based on the seating arrangement. Behavioral similarity is calculated by comparing the behavioral sequences of two seat numbers. Specifically, this is achieved by counting the frequency of the same behavior occurring within the same time window. For example, within 10 minutes, seat A and seat B have the behavioral sequences [lower head, lower head, raise head, lean sideways, lower head] and [lower head, raise head, raise head, lean sideways, lower head], respectively. The similarity is 3 / 5 = 0.6. Acoustic correlation is calculated by calculating the cross-correlation coefficient of the sound signals at the two seat positions. The value range is [-1, 1], where 1 indicates perfect positive correlation, -1 indicates perfect negative correlation, and 0 indicates no correlation. The final edge weight is the weighted sum of the three: W = α × (1 / physical distance) + β × behavioral similarity + γ × acoustic correlation, where α, β, and γ are adaptive weights that are dynamically adjusted according to the exam stage. For example, at the beginning of the exam, α = 0.5, β = 0.3, and γ = 0.2; in the middle of the exam, α = 0.3, β = 0.4, and γ = 0.3; and at the end of the exam, α = 0.2, β = 0.5, and γ = 0.3.

[0135] For each seat number, a state vector is established and a state transition equation is constructed. The state vector contains behavioral state parameters and acoustic state parameters. Behavioral state parameters include the current behavior type, duration, and frequency, while acoustic state parameters include volume and voice activity detection results. The state transition equation consists of an individual behavioral evolution term and an interaction influence term. The individual behavioral evolution term calculates the state transition probability distribution by analyzing historical behavior sequences. For example, the state transition matrix is formed by calculating the probability of a particular student changing from "looking down" to "looking up" and from "looking up" to "looking around." For example, based on historical data, the probability of a student with this seat number changing from "looking down to write" to "looking up at the blackboard" is 0.7, the probability of changing to "leaning sideways" is 0.2, and the probability of changing to "looking down to flipping through a book" is 0.1. The interaction influence term calculates the difference between the state vectors of two seat numbers and applies a nonlinear mapping to the edge weights to obtain the state influence. The difference is calculated using the cosine distance; the difference between the two state vectors S1 and S2 is 1-cos(S1, S2). Nonlinear mapping uses a sigmoid function to convert differences and edge weights into influence values. This is used to calculate the group synchronization index, which is the average similarity of all seat number state vectors within a specific time window. For example, for a 30-minute window, a calculated group synchronization index of 0.75 indicates high behavioral synchronization.

[0136] Extracting Multidimensional Collaborative Features: Time-series collaborative features are obtained by calculating the correlation between behavioral switching sequences between seat numbers. For two seat numbers, their behavioral switching sequences are extracted (e.g., "lowering → raising head" is a single switch), and the correlation coefficient of these sequences is then calculated. State collaborative features are obtained by calculating the consistency of the state transition probability distribution, using the inverse of the KL divergence as the consistency metric. Group clustering features are obtained by calculating the density of neighborhoods with similar behaviors. Centered on seat number i, the number of neighbors whose behavioral similarity exceeds a threshold (e.g., 0.6) is counted and then divided by the total number of neighbors to obtain the density value. Causal collaborative features are obtained by calculating the behavioral trigger relationship. Specifically, after seat number i changes its behavior, the probability that seat number j will also change its behavior within a specific time window (e.g., 5 seconds) is calculated. Taking a specific examination room as an example, the correlation coefficient of the behavior switching sequences of seats (2, 3) and (2, 4) is 0.82, the inverse of the KL divergence of the state transition probability distribution is 0.75, the neighborhood density of seat (2, 3) is 0.65, and the probability that the behavior change of seat (2, 3) triggers the behavior change of seat (2, 4) is 0.58.

[0137] Multidimensional collaborative features are fused at multiple levels to generate collaborative behavior features. First, feature normalization is performed, followed by a hierarchical fusion strategy. The first level fuses temporal and state collaborative features to generate behavioral synergy. The second level fuses group aggregation features and causal collaborative features to generate spatial correlation. Finally, the third level fuses behavioral synergy and spatial correlation to generate the final collaborative behavior feature. For example, for a pair of seats, the calculated behavioral synergy is 0.78, the spatial correlation is 0.62, and the final collaborative behavior feature is 0.71.

[0138] Collaborative cheating is determined based on collaborative behavior characteristics and group synchronization indicators: the collaborative behavior characteristic threshold is set to 0.65, and the group synchronization indicator threshold is set to 0.7. When the collaborative behavior characteristic of a pair of seats exceeds 0.65 and the group synchronization indicator exceeds 0.7, collaborative cheating is determined to have occurred.

[0139] Existing collaborative cheating detection suffers from problems such as insufficient detection accuracy, poor adaptability, insufficient analysis depth, and high false alarm rate. These problems are mainly manifested in the following aspects: a single feature cannot accurately identify collaborative behavior, fixed rules cannot cope with diverse cheating methods, there is a lack of modeling of group behavior dynamics, and spatiotemporal correlation information is not fully utilized.

[0140] like Figure 3The graph shows changes in group synchronization indicators and multi-dimensional collaborative features. The top half displays the "Group Synchronicity Index Changes," with time (minutes) on the horizontal axis and synchronization index values (0-1.0) on the vertical axis. The graph contains two curves: the solid line represents the coordinated cheating group, and the dashed line represents the non-coordinated group. The synchronization index for the coordinated cheating group gradually increases over time, beginning to increase significantly after 30 minutes and reaching a stable high value (approximately 0.8-0.9) around 60 minutes. The synchronization index for the non-coordinated group remains at a low level (approximately 0.1-0.15). The graph also indicates the "synchronicity threshold" and "collaborative behavior region." When the synchronization index exceeds the threshold and enters the collaborative behavior region, coordinated cheating is detected. The bottom half is a "Multi-Dimensional Collaborative Feature Radar Chart," showing five dimensions of collaborative features: temporal coordination, causal coordination, state coordination, group clustering, and behavioral clustering. The solid polygons in the graph represent the characteristic distribution of the coordinated cheating group, while the dashed polygons represent the characteristic distribution of the non-coordinated group. The radar chart shows that the collaborative cheating group has significantly higher eigenvalues across all five dimensions than the non-collaborative group, with the most significant differences in the temporal and state coordination dimensions. This indicates that collaborative cheating behavior exhibits distinct characteristics in terms of temporal synchronization and state consistency. This chart intuitively demonstrates the temporal evolution and multi-dimensional collaborative characteristics of collaborative cheating behavior, providing an important visual analysis basis for collaborative cheating detection.

[0141] The multi-scale feature fusion framework proposed in this application: combines physical distance, behavioral characteristics and acoustic characteristics for comprehensive analysis, realizes feature extraction of multiple time scales through wavelet decomposition, and adopts an adaptive weight mechanism for feature fusion. This multi-dimensional, multi-scale fusion method significantly improves the feature expression capability, enabling the system to capture more subtle collaborative behavior characteristics. This application innovatively introduces group behavior dynamics modeling: constructs a time-varying graph structure to describe the dynamic relationship between candidates, designs state transition equations that include individual evolution and interactive influences, and extracts multi-dimensional collaborative features to characterize group behavior. At the same time, a multi-level judgment mechanism is established, from preliminary anomaly detection to in-depth collaborative analysis, combining temporal features and spatial features, introducing group synchronization indicators, and realizing accurate identification of collaborative cheating behavior.

[0142] According to a second aspect of the embodiments of the present invention,

[0143] Provides an intelligent analysis system for abnormal behavior by integrating multi-source data in the examination room, including:

[0144] The first unit is used to collect video and audio data in the examination room, perform multi-view skeleton point fusion and sound source localization on the detection area corresponding to each seat number, and extract behavioral feature sequences and acoustic features;

[0145] The second unit is used to construct a hidden Markov model based on the behavioral feature sequence and acoustic features, use kernel density estimation to calculate the distribution difference between answering and thinking states to obtain dynamic adaptive weights, evaluate state-behavior consistency by matching the state smoothed posterior probability with the personalized normal behavior model, and identify abnormal behavior within a sliding time window in combination with the probability distribution entropy value;

[0146] The third unit is used to calculate a spatial weight coefficient based on the physical distance between the seat number determined to have abnormal behavior and the surrounding seats, perform wavelet decomposition on the behavioral feature sequence and acoustic features of the relevant seat number to obtain multi-scale coefficients, establish a correlation between the seat numbers based on the multi-scale coefficients, recursively calculate the behavioral-acoustic correlation between the seat numbers at different time scales, and use the spatial weight coefficients to identify correlated abnormal behaviors and identify multi-person coordinated cheating behavior;

[0147] The fourth unit is used to record detected cheating behaviors and push warning information in real time.

[0148] According to a third aspect of the embodiments of the present invention,

[0149] An electronic device is provided, comprising:

[0150] processor;

[0151] a memory for storing processor-executable instructions;

[0152] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0153] According to a fourth aspect of the embodiments of the present invention,

[0154] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0155] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent analysis method for abnormal behavior based on multi-source data fusion in examination rooms, characterized by: include: Collect video and audio data from the examination room, perform multi-view skeleton point fusion and sound source localization on the detection area corresponding to each seat number, and extract behavioral feature sequences and acoustic features; A hidden Markov model is constructed based on the behavioral feature sequence and acoustic features. Kernel density estimation is used to calculate the distribution difference between the answering state and the thinking state to obtain a dynamic adaptive weight. The consistency of the state behavior is evaluated by the matching degree between the state smoothed posterior probability and the personalized normal behavior model. The probability distribution entropy value is combined to identify abnormal behavior within a sliding time window. A spatial weight coefficient is calculated based on the physical distance between the seat number identified as having abnormal behavior and the surrounding seats. The behavioral feature sequence and acoustic features of the relevant seat number are subjected to wavelet decomposition to obtain multi-scale coefficients. The correlation between the seat numbers is established based on the multi-scale coefficients. The behavioral-acoustic correlation between the seat numbers at different time scales is recursively calculated. The spatial weight coefficient is combined with the spatial weight coefficient to identify abnormal behaviors with correlation and identify multi-person coordinated cheating behavior. The detected cheating behavior will be recorded and warning information will be pushed in real time.

2. The method for intelligent analysis of abnormal behavior based on multi-source data fusion in an examination room according to claim 1 is characterized in that: Extracting behavioral feature sequences and acoustic features includes: Multiple cameras are used to collect video data, the confidence of human skeleton nodes is calculated to obtain node visibility weights and the optimal viewing angle is selected. Time series prediction and structural constraints are performed on the human skeleton nodes under the optimal viewing angle, and the constrained skeleton nodes are weightedly fused with the corresponding skeleton nodes under other viewing angles to obtain the fusion position. Based on the fusion position, a spatiotemporal graph structure is constructed and action features are extracted. The action features are temporally segmented and mapped to preset test scenario action primitives to obtain action semantic features. Based on the fusion position, the three-dimensional head angle and gaze center point are calculated, an attention distribution map is constructed, and temporal tracking is performed to obtain an attention transfer sequence. The action semantic features and the attention transfer sequence are input into a long short-term memory network to obtain an intention state probability distribution. Calculating the similarity between the action semantic feature and the attention transfer sequence to obtain a modality consistency score, calculating the feature fusion weight based on the modality consistency score, performing weighted fusion on the action semantic feature, the attention transfer sequence, and the intention state probability distribution, and applying a temporal smoothing constraint to obtain a behavior feature sequence; For the collected audio data, sound source localization is performed based on cross-correlation delay estimation and sound source enhancement algorithm, and the localization result is associated with the seat number; short-time energy calculation and double-threshold speech segment detection are performed on the associated audio data, and volume characteristics and duration characteristics are extracted as acoustic features.

3. The method for intelligent analysis of abnormal behavior based on multi-source data fusion in an examination room according to claim 1 is characterized in that: A hidden Markov model is constructed based on the behavioral feature sequence and acoustic features. Kernel density estimation is used to calculate the distribution difference between the answering and thinking states to obtain dynamic adaptive weights. The consistency of state behavior is evaluated by the matching degree between the state smoothed posterior probability and the personalized normal behavior model. Abnormal behavior recognition is performed within a sliding time window in combination with the probability distribution entropy value, including: Normalizing the behavioral feature sequence and the acoustic feature to obtain a normalized feature sequence; constructing a hidden Markov model based on the normalized feature sequence, using kernel density estimation and KL divergence to calculate the state distribution difference, and obtaining dynamic adaptive weights through Sigmoid mapping to construct a time-varying state transition probability matrix; in the hidden Markov model, the state smoothed posterior probability is calculated using the product of forward variables and backward variables, the forward variable is recursively obtained by multiplying the state emission probability and the time-varying state transition probability, and the backward variable is recursively obtained by multiplying the state emission probability at the next moment and the time-varying state transition probability; Based on the normalized feature sequence collected during the first answering cycle after the start of the exam, a personalized normal behavior model of the examinee is constructed, and the matching degree between the state smoothed posterior probability and the state parameters of the personalized normal behavior model is calculated to obtain a state-behavior consistency score; The probability distribution entropy value of the smoothed posterior probability of the state is calculated within a sliding window of multiple time scales, and the abnormal judgment is adaptively corrected in combination with the test time process and the behavioral baseline of the surrounding seats. When the abnormal scores of multiple consecutive time scales exceed the adaptive abnormal threshold, it is determined to be abnormal behavior.

4. The method for intelligent analysis of abnormal behavior based on multi-source data fusion in an examination room according to claim 3 is characterized in that: A hidden Markov model is constructed based on the normalized feature sequence. The kernel density estimation and KL divergence are used to calculate the state distribution difference. The dynamic adaptive weight is obtained through Sigmoid mapping to construct the time-varying state transition probability matrix, including: Based on the normalized feature sequence, a hidden Markov model including answering state and thinking state is constructed, and the state indicator function is used to mark the state of the examinee's answering behavior sequence. The feature samples marked as answering state are used to construct an answering state sample set, and the feature samples marked as thinking state are used to construct a thinking state sample set. Using a kernel density estimation method to calculate the feature distribution density functions of the answering state sample set and the thinking state sample set respectively, calculating the KL divergence based on the feature distribution density function to obtain the feature distribution difference, and normalizing the feature distribution difference to obtain the normalized difference; Calculating a dynamic adaptive weight based on the normalized difference, wherein the dynamic adaptive weight maps the normalized difference through a Sigmoid mapping function; The dynamic adaptive weights are used to construct a time-varying state transition probability matrix, wherein the state transition probability is obtained by weighted calculation of the basic transition probability and the conditional transition probability based on the current observation by the dynamic adaptive weights, and the time-varying state transition probability matrix is used for subsequent state sequence estimation.

5. The method for intelligent analysis of abnormal behavior based on multi-source data fusion in an examination room according to claim 3 is characterized in that: The probability distribution entropy of the smoothed posterior probability of the state is calculated within a sliding window of multiple time scales. Adaptive correction is performed on the abnormality judgment based on the test time process and the surrounding seat behavior baseline. When the abnormality scores of multiple consecutive time scales exceed the adaptive abnormality threshold, it is determined to be abnormal behavior including: The probability distribution entropy of the smoothed posterior probability of the state is calculated in sliding windows of three time scales: short-term, medium-term and long-term. The short-term window is used to capture sudden anomalies, the medium-term window is used to detect persistent anomalies, and the long-term window is used to evaluate the overall anomaly trend. The current exam time process is calculated based on the exam start time, and the exam process is divided into three stages: the beginning, the middle, and the end. Different abnormal scoring weights are set for the probability distribution entropy values of different stages; Obtain the state smoothed posterior probability of other seats within a preset range around the determined seat number, calculate the mean of the probability distribution entropy value of the local area as the behavior baseline, and use the behavior baseline to correct the state-behavior consistency score; The fusion anomaly score is calculated based on the probability distribution entropy value of different time scales, the test stage weight and the corrected state-behavior consistency score; the adaptive anomaly threshold is calculated based on the fusion anomaly score in the first answering cycle. When the fusion anomaly scores of three consecutive time scales exceed the corresponding adaptive anomaly threshold, it is judged as abnormal behavior.

6. The method for intelligent analysis of abnormal behavior based on multi-source data fusion in an examination room according to claim 1 is characterized in that: A spatial weight coefficient is calculated based on the physical distance between the seat number determined to have abnormal behavior and the surrounding seats. The behavioral feature sequence and acoustic features of the relevant seat number are subjected to wavelet decomposition to obtain multi-scale coefficients. The correlation between the seat numbers is established based on the multi-scale coefficients. The behavioral-acoustic correlation between the seat numbers at different time scales is recursively calculated. The spatial weight coefficient is used to identify abnormal behaviors with correlation. Identifying multi-person coordinated cheating behavior includes: Calculating a spatial weight coefficient based on the physical distance between the seat number determined to be abnormal behavior and the surrounding seats, wherein the spatial weight coefficient is obtained by mapping the physical distance between seats using a Gaussian kernel function and normalizing it; Performing wavelet decomposition on the behavioral feature sequence and the acoustic feature sequence of the relevant seat numbers to obtain multi-scale coefficients, wherein the multi-scale coefficients include the approximate coefficient and detail coefficient of the behavioral feature sequence and the approximate coefficient and detail coefficient of the acoustic feature sequence; calculating the mutual correlation coefficients between the seat numbers at different time scales based on the multi-scale coefficients, wherein the mutual correlation coefficients are obtained by dividing the covariance of the multi-scale coefficients of the behavioral feature sequence and the multi-scale coefficients of the acoustic feature sequence by the product of the standard deviations; performing weighted fusion on the mutual correlation coefficients to obtain a fused correlation degree, and using the fused correlation degree as the initial correlation degree; An initial correlation relationship between seat numbers is constructed based on the initial correlation, and the product of the spatial weight coefficient and the initial correlation is used as the weighted correlation between seat numbers. A correlation threshold is determined based on the fluctuation range of the weighted correlation in the first answering cycle after the start of the exam. When the weighted correlation exceeds the correlation threshold and the duration exceeds a preset time window, it is determined that preliminary collaborative cheating behavior exists between the relevant seat numbers.

7. The method for intelligent analysis of abnormal behavior based on multi-source data fusion in an examination room according to claim 6 is characterized in that: Further identification of the preliminary coordinated cheating behavior includes: A time-varying graph structure with seat numbers as nodes is constructed. The behavioral feature sequences and acoustic features corresponding to the seat numbers are used as node features. The edge weights are obtained by adaptively weighting the physical distance, behavioral similarity, and acoustic correlation between seat numbers. For each seat number, a state vector including behavioral state parameters and acoustic state parameters is established. A state transfer equation is constructed based on the state vector and the edge weights. The state transfer equation includes an individual behavior evolution term and an interaction influence term. The individual behavior evolution term is used to calculate the state transfer probability distribution. The interaction influence term is obtained by calculating the difference of the state vector and performing nonlinear mapping with the edge weights. The group synchronization index is obtained based on the state influence. The correlation between the behavior switching sequences of seat numbers is calculated as the temporal collaborative feature, the consistency of the state transition probability distribution is calculated as the state collaborative feature, the density of neighborhoods with similar behaviors is calculated as the group aggregation feature, and the behavior triggering relationship is calculated as the causal collaborative feature. The temporal collaborative features, state collaborative features, group aggregation features and causal collaborative features are multi-level integrated to obtain collaborative behavior features, and collaborative cheating behavior is determined based on the collaborative behavior features and the group synchronization index.

8. An intelligent analysis system for abnormal behavior based on multi-source data fusion in an examination room, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to collect video and audio data in the examination room, perform multi-view skeleton point fusion and sound source localization on the detection area corresponding to each seat number, and extract behavioral feature sequences and acoustic features; The second unit is used to construct a hidden Markov model based on the behavioral feature sequence and acoustic features, use kernel density estimation to calculate the distribution difference between answering and thinking states to obtain dynamic adaptive weights, evaluate state-behavior consistency by matching the state smoothed posterior probability with the personalized normal behavior model, and identify abnormal behavior within a sliding time window in combination with the probability distribution entropy value; The third unit is used to calculate a spatial weight coefficient based on the physical distance between the seat number determined to have abnormal behavior and the surrounding seats, perform wavelet decomposition on the behavioral feature sequence and acoustic features of the relevant seat number to obtain multi-scale coefficients, establish a correlation between the seat numbers based on the multi-scale coefficients, recursively calculate the behavioral-acoustic correlation between the seat numbers at different time scales, and use the spatial weight coefficients to identify correlated abnormal behaviors and identify multi-person coordinated cheating behavior; The fourth unit is used to record detected cheating behaviors and push warning information in real time.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images