Analysis method, device and system suitable for judge abnormal behaviors

Abstract

The invention discloses an analysis method, device and system suitable for an abnormal behavior of a judge. The method comprises the following steps: collecting a video stream of an area where the judge is located in real time through a plurality of cameras; identifying and analyzing the acquired video stream; acquiring a face image of a character object in the video image and joint point coordinate information of a body; obtaining an attitude feature information matrix S1; extracting a human face region by using MTCNN, and representing the human face region as K1; fusing the human body expression information K1 and the human body posture information S1, and representing a feature matrix S2; enabling the fused feature matrix to pass through an abnormal behavior recognition model of a residual network with noise robustness to obtain a final behavior recognition result of the human body posture; the method comprises the steps of extracting audio information in a video stream, performing text conversion and keyword extraction on the audio information to obtain an audio keyword set, and predicting an abnormal behavior to obtain an abnormal behavior prediction result. The method is accurate in monitoring and can deal with various scenes.

Description

Technical Field

[0001] The present invention relates to a behavior analysis method, in particular to an analysis method, device and system suitable for abnormal behavior of judges. Background Art

[0002] Artificial intelligence technology simulates human cognitive abilities through algorithms and data analysis, enabling the construction of intelligent systems with perception, reasoning, and decision-making capabilities. The multimodal fusion-based evaluation room judge abnormal behavior analysis system is an AI application that integrates computer vision, speech recognition, and sensor technologies. By monitoring judges' behavioral characteristics and environmental parameters in real time, it automatically identifies scenarios such as abnormal number of people, personnel entering and exiting, sensitive speech, mobile phone use, gaming, and smoking. Its purpose is to enhance the objectivity and real-time nature of bid evaluation process supervision, reduce the risk of human oversight omissions, and provide a traceable basis for decision-making to ensure the fairness of bid evaluation through data-driven analysis.

[0003] Existing methods usually rely on manual inspections and single-sensor monitoring, which have problems such as low efficiency, limited coverage, and lack of real-time performance. Due to the lack of support for computer vision and deep learning algorithms, traditional monitoring systems find it difficult to accurately distinguish between normal actions and violations during the behavior recognition stage. In addition, without the introduction of real-time detection models for voice-sensitive words and dynamic analysis technology for smoke concentration, existing systems find it difficult to accurately capture hidden violations (such as suggestive language). In addition, existing systems usually do not integrate continuous learning mechanisms, cannot adapt to the dynamic changes in bidding scenarios, and lack intelligent processing capabilities in data correlation analysis, multi-device collaborative decision-making, etc. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a method, device and system for analyzing abnormal behavior of judges.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for analyzing abnormal behavior of judges, comprising the following steps:

[0007] Use multiple cameras to collect real-time video streams of the area where the judges are located;

[0008] Performing recognition and analysis on the acquired video stream, performing frame processing on the video stream to obtain N frames of video images, and extracting human objects and environmental objects in each frame of the video image, where the environmental objects include other tools except human objects;

[0009] Obtain the facial image and body joint coordinate information of the person in the video image, and obtain the 18 joint coordinate sequences {(x11, y11), (x12, y12), …, (x1N, y1N)} of the current frame image;

[0010] The vector angle θ of the same joint point in adjacent frames 1N The calculation of the joint point dynamic information {θ 11 ,θ 12 ,…,θ 1N};

[0011] The position and dynamic information of the joint points are calculated according to the weight f t Allocation, feature fusion is performed, expressed as 18*T*4 posture feature information matrix S1;

[0012] Use MTCNN to extract the face area, and use MobileNet to extract the expression features of the detected face image, expressed as K1 of 16*T*1;

[0013] The human expression information K1 and the human posture information S1 are fused and expressed as a 18*T*4 feature matrix S2;

[0014] The fused feature matrix is ​​passed through the abnormal behavior recognition model of the residual network with noise robustness to obtain the final human posture behavior recognition result;

[0015] Extract audio information from the video stream, convert the audio information into text and extract keywords to obtain an audio keyword set;

[0016] According to the human body posture and audio keyword set, abnormal behavior is predicted to obtain abnormal behavior prediction results.

[0017] As a further improvement, the angle of the joint point is calculated using the following formula:

[0018]

[0019] The weights f are assigned as follows t Calculation:

[0020] According to the intensity of the change in the direction and speed of the joint point, the weight f is assigned to the T frame image. T , represents the predicted probability value of a single frame, and calculates the final weight f of the T-th key frame T The formula is as follows

[0021]

[0022] where θ t Indicates the maximum angle change of the joint point in the T-th image frame, vt represents the maximum velocity change of the joint point in the Tth image frame, k1 and k2 are the weight values ​​of angle and velocity.

[0023] As a further improvement, the step of converting the audio into text and extracting keywords to obtain an audio keyword set includes:

[0024] Based on ASR technology, the audio is converted into text to obtain the text to be analyzed;

[0025] Segmenting the text to be analyzed to obtain a phrase set;

[0026] Acquire a keyword set corresponding to the video stream from a preset keyword library as a target keyword set;

[0027] Match each phrase in the phrase set with the test keywords in the target keyword set to obtain a matching result;

[0028] If there is a successful matching result, each of the phrases corresponding to the successful matching result is used as the audio keyword set.

[0029] As a further improvement, the abnormal behavior detection model adopts a noise-robust residual network:

[0030] The residual network with INSP activation function is used as the abnormal behavior recognition model. The network adds residual units on the basis of VGG-19 and replaces the ReLU function with the NSP function for improvement.

[0031]

[0032] Where k represents the control of noise scale, S represents the scale parameter, x represents the input value of the previous convolution layer, and w i Represents the weight of the previous convolution layer, i represents the i-th neuron, τ syn represents the synaptic time constant.

[0033] As a further improvement, if it is detected that the total number of people in the video stream exceeds the set number and the duration exceeds the set time, an alarm will be issued and a preliminary determination will be made that abnormal behavior has occurred; manual intervention will be required for verification.

[0034] As a further improvement, when the facial image is detected, facial expression samples are obtained after face detection, and the facial expression samples include various expression sequence samples, and the various expression sequences include neutral expression sequences, weak expression sequences, and exaggerated expression sequences;

[0035] For the facial expression sample, extracting features of the optical flow motion field by fusing the binary image and the grayscale image sequence to obtain a feature sequence;

[0036] Using the feature sequence as an observation sequence, training the neutral expression sequence and the weak expression sequence in the facial expression sample, and establishing corresponding hidden Markov models for various expressions;

[0037] Combine hidden Markov models of various expressions to form a facial expression classifier;

[0038] If the detected facial expression shows any changes beyond the preset expression compared to the preset facial surface, it is determined that abnormal behavior has occurred.

[0039] As a further improvement, the environmental objects include mobile phones, cigarettes, pens and paper. If smoke is detected, an alarm is issued. If the audio volume exceeds a set value, abnormal behavior is determined.

[0040] A device for analyzing abnormal behavior of judges based on multimodal data, comprising an analysis method suitable for abnormal behavior of judges according to any one of claims 1 to 8.

[0041] A system for analyzing abnormal behavior of judges based on multimodal data, comprising:

[0042] A data acquisition unit, used for capturing and obtaining a video stream;

[0043] A conversion module, used for identifying and analyzing the unit, and for performing frame processing on the video stream;

[0044] A determination unit, used for determining abnormal behavior;

[0045] Alarm unit, used to issue warnings.

[0046] The present invention has the following beneficial technical effects:

[0047] By collecting video streams in real time through multiple cameras and fusing multimodal data such as human posture, expressions, and audio keywords, and combining them with an improved residual network model for abnormal behavior recognition, the system can fully capture the judges' micro-expressions, abnormal body movements, and sensitive voices, effectively solving the problems of insufficient multimodal data fusion and low recognition accuracy in traditional methods. It has the advantages of improving the real-time nature of supervision and enhancing the comprehensiveness of abnormal behavior recognition. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The following reference embodiments are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0049] In the description of the present invention, it should be understood that if the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like are used to indicate an orientation or positional relationship based on the orientation or positional relationship shown, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0050] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed connections, removable connections, or integral connections. They may refer to mechanical connections or electrical connections. They may refer to direct connections or indirect connections through an intermediary, and they may refer to internal communication between two components or interactions between two components. Those skilled in the art will understand the specific meanings of these terms in the present invention based on the specific circumstances.

[0051] Traditional monitoring of bid evaluation rooms relies primarily on manual inspections and single-sensor data collection, resulting in issues such as insufficient real-time performance and limited monitoring dimensions. Due to the lack of multimodal data fusion mechanisms, existing systems struggle to detect subtle anomalies in evaluators' body movements, facial expressions, and voice interactions, such as suggestive language or brief violations. During the bid evaluation process, evaluators may convey information through body language or use electronic devices for unauthorized operations. Traditional monitoring systems, unable to simultaneously analyze dynamic features and audio keywords in video streams, create blind spots in monitoring.

[0052] To address these issues, researchers discovered that existing technologies were insufficiently able to capture dynamic features for behavior recognition and lacked cross-modal collaborative analysis. By analyzing the potential manifestations of unusual judges' behavior, such as abnormal joint movements, sudden facial expressions, or the presence of specific keywords, they realized the need for a multi-dimensional feature fusion mechanism. They further considered how to correlate changes in human posture and facial expressions in video streams over time, combining this with audio semantic analysis to form a complementary chain of evidence. Furthermore, they explored the use of a dynamic weight allocation strategy to handle fluctuations in time series data and designed a noise-robust recognition model to address environmental interference.

[0053] Example 1

[0054] This application proposes an analysis method for abnormal behavior of judges, comprising the following steps: using multiple cameras to collect real-time video streams of the area where the judges are located;

[0055] Performing recognition and analysis on the acquired video stream, performing frame processing on the video stream to obtain a plurality of video frames, and extracting human objects and environmental objects in each video frame, wherein the environmental objects include other tools except human objects;

[0056] Obtaining the facial image and body joint coordinate information of the person object in the video image, and obtaining the joint coordinate sequence of the current frame image;

[0057] The vector angles of the same joint points in adjacent frames are calculated to obtain the dynamic information of the joint points. The joint point positions and dynamic information are assigned according to weights and feature fusion is performed to express them as a posture feature information matrix.

[0058] Use the face detection network to extract the face area, and use the lightweight convolutional network to extract the expression features of the detected face image;

[0059] Fuse human expression information and human posture information to obtain a fusion feature matrix;

[0060] The fused feature matrix is ​​passed through the abnormal behavior recognition model of the residual network with noise robustness to obtain the final human posture behavior recognition result;

[0061] Extract audio information from the video stream, convert the audio information into text and extract keywords to obtain an audio keyword set;

[0062] According to the human body posture and audio keyword set, abnormal behavior is predicted to obtain abnormal behavior prediction results.

[0063] Frame processing refers to the process of segmenting a continuous video stream into discrete time units. This can be achieved using fixed time intervals or dynamic adaptive algorithms to capture the temporal characteristics of motion. A joint coordinate sequence refers to the set of skeletal node coordinates located by a human pose estimation algorithm. For example, the OpenPose framework is used to extract two-dimensional joints, whose spatial distribution can characterize limb motion trajectories. Vector angle calculation refers to the method of calculating joint bending angles from joint displacement vectors in adjacent frames. For example, the vector dot product formula is used to derive the angle change, which is used to quantify the amplitude of motion. Feature fusion weight allocation refers to a mechanism that dynamically adjusts the contribution of each frame based on the amplitude and velocity of joint motion. For example, the weighted summation of normalized angle and velocity changes can be used to enhance the recognition weight of key action frames. A pose feature information matrix refers to a tensor data structure that integrates spatial coordinates and temporal dynamic features. For example, the coordinates, velocity, and angle changes of each joint point are encoded into a multidimensional matrix for easy processing by deep learning models. A face detection network refers to an object detection model that can quickly locate facial regions. For example, the MTCNN achieves real-time face detection through a cascaded convolutional network, ensuring input quality for expression analysis. Lightweight convolutional networks are feature extraction models with fewer parameters and high computational efficiency. For example, MobileNet uses a depthwise separable convolutional structure, which can be deployed on mobile devices for real-time expression classification. Audio keyword sets are collections of sensitive words extracted through speech recognition and natural language processing technologies. For example, after an ASR system converts speech into text, it performs pattern matching against a pre-set vocabulary of violations to assist in identifying unusual speech.

[0064] Specifically, multiple cameras are used to synchronously capture video streams to ensure full coverage of the judges' activity area. After frame processing, the human object in each frame is separated from the surrounding objects using background subtraction. For example, detecting a non-stationary object such as a mobile phone triggers an alert. A pre-trained human pose estimation model is used to generate joint coordinate sequences, capturing the spatial positions of key parts such as the head and limbs. Changes in joint angles between frames are quantified using vector operations. For example, sudden changes in elbow flexion angle may indicate an abnormal gesture. A weight assignment module automatically adjusts the importance of each frame based on the amplitude of the movement, assigning higher weights to frames with intense movements. Posture and expression features are integrated through matrix concatenation, for example, combining the four-dimensional features of 18 joints with the 16-dimensional expression vector into a unified feature space. The abnormal behavior recognition model utilizes an improved residual network architecture, for example, by introducing a noise suppression module between convolutional layers to improve the model's robustness in complex lighting conditions. The audio analysis module uses voiceprint separation technology to extract independent audio tracks. For example, after separating the judges' voices from ambient noise, a keyword matching algorithm is used to detect sensitive speech. The final decision module combines the posture anomaly score and keyword matching degree. For example, when both exceed the threshold at the same time, it is determined to be a violation.

[0065] Compared with existing technologies, existing systems usually use a single camera and a fixed threshold alarm mechanism, which cannot analyze the temporal correlation of continuous actions. This method uses multimodal data fusion to collaboratively analyze body movements, facial expressions, and speech semantics. For example, when a judge's body is detected to be suddenly still, it can match specific keywords and improve the recognition accuracy of suggestive violations. Traditional methods rely on manually set rules to judge anomalies, while this solution automatically learns dynamic behavior patterns through deep learning models. For example, the model can recognize the micro-movement characteristics of fingers touching the screen when playing with a mobile phone, avoiding misjudging normal writing as a violation. Existing technologies do not achieve dynamic detection of environmental objects, while this solution enhances the comprehensiveness of monitoring by analyzing the status of environmental objects in real time, such as immediately triggering an early warning when an unauthorized electronic device is detected on the desktop.

[0066] Through the above technical solutions, this application realizes full-dimensional monitoring of the judges' behavior, and significantly reduces the misjudgment rate of a single sensor through the fusion of time series features and the construction of a cross-modal evidence chain. A dynamic weight distribution strategy is adopted to effectively distinguish the subtle differences between normal actions and abnormal behaviors, such as the difference in joint angle changes between rapid page turning and illegal gestures. Through lightweight model design, real-time analysis can be achieved under limited computing power conditions, meeting the response speed requirements of the bidding scenario. Combined with audio semantic analysis, language violations can be captured simultaneously to form a multi-dimensional basis for abnormal behavior judgment.

[0067] The angle of the joint point is calculated using the following formula:

[0068]

[0069] The weights f are assigned as follows t Calculation:

[0070] According to the intensity of the change in the direction and speed of the joint point, the weight f is assigned to the T frame image. T , represents the predicted probability value of a single frame, and calculates the final weight f of the T-th key frame T The formula is as follows

[0071]

[0072] where θ t Indicates the maximum angle change of the joint point in the T-th image frame, v t represents the maximum velocity change of the joint point in the Tth image frame, k1 and k2 are the weight values ​​of angle and velocity.

[0073] Among them, the vector dot product operation refers to calculating the sum of the projection products of two vectors in space. It can be achieved by multiplying the coordinate components and then accumulating them. It is used to measure the directional similarity between vectors.

[0074] The vector modulus refers to the geometric length of the vector in a two-dimensional plane, which can be calculated using the Euclidean norm formula and is used to quantify the displacement amplitude of the vector in space.

[0075] The inverse cosine function refers to a mathematical function that maps cosine values ​​to angle intervals. Specifically, it can be implemented using numerical calculation methods to convert vector direction differences into angle values.

[0076] Specifically, the formula calculates the spatial displacement vector of the same joint point between adjacent frames, uses a dot product operation to obtain the directional similarity between the two vectors, then normalizes them using the modulus parameter. Finally, the inverse cosine function is used to quantify the directional difference into an angle value. During the judges' behavioral analysis, if there are sudden changes in body movements, such as a quick head turn or a large wave of the hand, this angle calculation can accurately capture the magnitude of the change in the joint's motion trajectory. For adjacent frames of continuous movement, the angle value will show a smooth change; for abnormal behavior, the angle value will exhibit a sudden change.

[0077] Compared to existing technologies, traditional methods typically only determine movement amplitude based on coordinate displacement thresholds, failing to distinguish between changes in direction and positional translation. This solution effectively separates movement direction and amplitude through vector angle calculation, avoiding misjudgments caused by normal positional adjustments by the judges. For example, when a judge writes normally, the positional translation of the hand joints may produce a large displacement, but the change in direction and angle is small. However, abnormal behavior, such as suddenly pointing at others, will result in a significant sudden change in the angle of the hand joints.

[0078] Through the above technical solution, this application can accurately quantify the dynamic change characteristics of joint points, distinguish the movement pattern differences between normal behavior and abnormal movements, and thus improve the accuracy of judges' identification of abnormal behavior. This method can effectively avoid the false positives caused by simple displacement thresholds. In particular, when dealing with small, high-frequency movement scenes, the introduction of angle changes can more sensitively capture potential violations.

[0079] The joint point direction change refers to the angular offset formed by the motion trajectory of the same joint point between adjacent frames. This can be achieved using the vector angle calculation method. This parameter is used to characterize the amplitude difference in the judges' posture adjustments. The velocity change refers to the displacement of the joint point coordinates per unit time. This can be achieved by calculating the ratio of the Euclidean distance to the time interval. This parameter is used to capture the sudden characteristics of the judges' movements. The weight distribution ft refers to the weighting coefficient constructed based on the dynamic changes in angle and velocity. This can be achieved using the linear weighted fusion method. By adjusting the proportional relationship between k1 and k2, the degree of abnormality of the judges' movements can be quantitatively assessed.

[0080] Specifically, for each joint point in a continuous T-frame video image, the movement angle θt and speed vt of the joint point between adjacent frames are calculated to obtain the extreme values ​​of the angle change and speed change, respectively. After the two types of parameters are input into the standardization processing module for normalization, they are linearly combined according to the preset k1 and k2 proportional coefficients to finally form the weight distribution value of each frame. For example, when the judges suddenly raise their hands or turn their heads quickly, the corresponding angle change θt and speed vt will increase synchronously. At this time, the weight fT will be significantly increased, making these key frames occupy a higher proportion in the subsequent feature fusion process. The 18T4 posture feature information matrix S1 constructed in this way can effectively highlight the high dynamic features when abnormal behavior occurs.

[0081] Compared with existing technologies, traditional methods typically use fixed weights or simple mean calculations, failing to account for the dynamic nature of judges' movements over time. This solution precisely locates abnormal motion fragments by introducing a dual-factor evaluation mechanism based on angle change and speed change. For example, existing technologies can cause confusion when handling judges slowly adjusting their sitting posture or suddenly answering or making a phone call due to improper weight allocation. However, this solution, through a dynamic weight adjustment mechanism, can effectively distinguish subtle movements from abnormal behavior.

[0082] Through the above technical solution, this application can solve the problem of insufficient accuracy of existing monitoring systems in identifying abnormal judges' movements. By dynamically calculating the weight distribution of each frame of video, the effectiveness of posture feature extraction is significantly improved, allowing the abnormal behavior recognition model to focus on highly dynamic action clips. For example, in the detection scenario of judges using mobile phones in violation of regulations, this method can accurately capture the movement characteristics of the hand suddenly moving to a specific area, reducing the misjudgment rate caused by normal body movements.

[0083] Example 2

[0084] The step of converting the audio into text and extracting keywords to obtain an audio keyword set includes:

[0085] Based on ASR technology, the audio is converted into text to obtain the text to be analyzed;

[0086] Segmenting the text to be analyzed to obtain a phrase set;

[0087] Acquire a keyword set corresponding to the video stream from a preset keyword library as a target keyword set;

[0088] Match each phrase in the phrase set with the test keywords in the target keyword set to obtain a matching result;

[0089] If there is a successful matching result, each of the phrases corresponding to the successful matching result is used as the audio keyword set.

[0090] As a further improvement, the abnormal behavior detection model adopts a noise-robust residual network:

[0091] The residual network with INSP activation function is used as the abnormal behavior recognition model. The network adds residual units on the basis of VGG-19 and replaces the ReLU function with the NSP function for improvement.

[0092]

[0093]

[0094] Where k represents the control of noise scale, S represents the scale parameter, x represents the input value of the previous convolution layer, and w i Represents the weight of the previous convolution layer, i represents the i-th neuron, τ syn represents the synaptic time constant.

[0095] ASR technology refers to automatic speech recognition technology, which can be implemented using a speech recognition engine based on deep neural networks. Automatic speech recognition technology is used to convert the judges' voice content into processable text data. A preset keyword library refers to a collection of illegal keywords pre-established according to different bidding evaluation scenarios. For example, it can be generated through manual annotation or historical data mining and is used to limit the matching range to improve accuracy. Exam keyword matching refers to calculating the similarity between the segmented text fragment and the target keyword. For example, this can be achieved using the edit distance algorithm or the word vector cosine similarity method, thereby screening out sensitive words related to illegal behavior.

[0096] Specifically, the audio stream is first converted into text containing the judges' conversations using ASR technology. This text is then segmented into independent semantic units to form a phrase set. Simultaneously, a pre-set keyword library is used to identify the offending keywords based on the current bidding evaluation scenario. For example, in a bidding evaluation environment, a target keyword set containing sensitive terms such as "bribery," "collusion," and "score manipulation" can be pre-set. Each segmented phrase is then compared against the target keyword set. When the similarity between a phrase and a keyword reaches a preset threshold, the phrase is considered a match and included in the audio keyword set, effectively extracting potentially offending speech.

[0097] Compared with existing technologies, traditional methods rely on manual monitoring or fixed rule matching, resulting in low efficiency and an inability to dynamically adapt to different scenarios. This solution combines a scenario-based keyword library with an automated matching mechanism to achieve targeted identification of covert illegal speech while avoiding the risk of misjudgment caused by redundant keyword libraries.

[0098] Through the above technical solution, this application can effectively solve the problem of the existing system's lack of voice sensitive word detection dimension, improve the accuracy of illegal speech recognition through a dynamic matching mechanism, and at the same time reduce the impact of text conversion errors caused by environmental noise or accent differences on detection results, providing reliable voice data support for abnormal behavior prediction.

[0099] Among them, the INSP activation function refers to a nonlinear activation unit designed based on the dynamic characteristics of neural synapses. It can be implemented by using a pulse neuron response model. By introducing the synaptic time constant parameter, the dynamic response threshold of the neuron to the input signal is adjusted, thereby improving the network's filtering ability for timing noise. The residual unit refers to a modular structure composed of convolutional layers and cross-layer connections. It can be implemented by adding jump connections between the convolutional layers of the VGG-19 network. It is used to alleviate the gradient vanishing problem during deep network training. The NSP function refers to the neural synaptic pulse conversion function. It can be implemented by integrating the postsynaptic potential integral with the pulse triggering mechanism. Its mathematical expression includes the noise scale parameter k and the input signal weight value wi, which is used to dynamically adjust the neuron activation threshold to adapt to environmental interference.

[0100] Specifically, the network architecture inserts residual connections between the convolutional layers of VGG-19, forming residual modules with cross-layer information transfer. The feature maps output by each residual module undergo a nonlinear transformation using the NSP function. This function introduces a dynamic integration mechanism by introducing a synaptic time constant, τsyn, to attenuate high-frequency noise components in the input signal. During model training, the noise scaling parameter k is dynamically adjusted based on the signal-to-noise ratio of the input data. When increased environmental interference is detected, the k value is automatically increased to suppress abnormal spike triggering. The final layer of the network combines a fully connected layer with a softmax classifier to output a probability distribution for behavior recognition.

[0101] Compared with the existing technology, traditional abnormal behavior recognition models mostly use the standard ReLU activation function, which is easily affected by equipment noise in the complex sound and light environment of the bidding room, leading to misjudgment. The NSP function used in this application can effectively distinguish between the timing signals generated by normal actions and abnormal signals caused by environmental noise by simulating the pulse integration mechanism of biological neural synapses. The residual unit design avoids the model degradation problem caused by the increase in the depth of the VGG network, so that the number of network layers can be expanded to more than 32 layers. The existing technology does not consider the impact of sensor noise on behavior recognition. This solution realizes adaptive processing of different interference intensities by dynamically adjusting the noise scale parameter k value.

[0102] Through the above technical solution, this application solves the problem of reduced recognition accuracy in traditional monitoring systems due to noise interference in complex acoustic and optical environments, and improves the stability of detecting abnormal movements of judges. When there are interference sources such as electromagnetic interference from equipment and environmental noise at the bidding site, the system can effectively suppress the impact of noise on posture feature extraction, avoiding the misinterpretation of normal body movements as illegal behaviors such as playing with a mobile phone or hand gestures.

[0103] If it is detected that the total number of people in the video stream exceeds the set number and the duration exceeds the set time, an alarm is issued and it is preliminarily determined that abnormal behavior occurs; manual intervention is performed to verify.

[0104] The "set number of people" refers to the maximum number of evaluators allowed in the bid evaluation room. Specifically, the upper limit on the number of evaluators specified in the bid evaluation rules can be used as the threshold. For example, the threshold number of people can be set to 5 through the configuration management interface. The "set time" refers to the minimum length of time that an abnormal number of people must continue to trigger an alarm. Specifically, a timer module can be used for cumulative monitoring, for example, setting the duration to 30 seconds. Alarming refers to the issuance of a warning signal through audio and visual equipment or network notifications. Specifically, this can be achieved by linking a buzzer with an SMS platform. Manual intervention verification refers to the confirmation of abnormal behavior by supervisors through video playback or on-site inspection. Specifically, this can be done by assigning tasks to the administrator terminal through the work order system.

[0105] Specifically, during the bid evaluation process, the camera captures the video stream of the judging area in real time and uses a crowd recognition algorithm to count the total number of people in the picture. When the number of people detected exceeds the preset threshold, a timer is triggered to begin accumulating the duration of the abnormal state. If this state persists for a set time or longer, the system automatically activates the alarm device and marks the abnormal event as pending confirmation. After receiving the alarm information on the management terminal, the supervisor retrieves the video recording of the corresponding time period for manual review to confirm whether there are any violations. This mechanism avoids false alarms caused by instantaneous personnel entry and exit through dual condition judgment. For example, the entry of temporary staff to deliver materials will not be mistakenly judged as an abnormality.

[0106] Compared to existing technologies, traditional manual inspections rely on regular on-site inspections, making it difficult to detect short-term overcrowding. Single sensor monitoring can only trigger instantaneous alarms and cannot distinguish between occasional incidents and persistent violations. By introducing a duration check, interference signals generated by normal personnel flow are effectively filtered out. Combined with a manual review mechanism, this significantly improves the accuracy of abnormal behavior identification and the rationality of handling.

[0107] Through the above technical solution, this application can effectively identify the illegal addition of evaluators during the bid evaluation process, avoiding procedural violations caused by overstaffing. The dual judgment criteria reduce false alarms caused by temporary personnel entering and exiting the bid evaluation process, reducing the workload of supervisors. The manual review process ensures the credibility of the system's judgment results, provides a traceable verification basis for subsequent processing, and ensures the standardization and fairness of the bid evaluation process.

[0108] In addition, when the facial image is detected, facial expression samples are obtained after face detection, and the facial expression samples include various expression sequence samples, and the various expression sequences include neutral expression sequences, weak expression sequences, and exaggerated expression sequences;

[0109] For the facial expression sample, extracting features of the optical flow motion field by fusing the binary image and the grayscale image sequence to obtain a feature sequence;

[0110] Using the feature sequence as an observation sequence, training the neutral expression sequence and the weak expression sequence in the facial expression sample, and establishing corresponding hidden Markov models for various expressions;

[0111] Combine hidden Markov models of various expressions to form a facial expression classifier;

[0112] If the detected facial expression shows any changes beyond the preset expression compared to the preset facial surface, it is determined that abnormal behavior has occurred.

[0113] The temporal and spatial variations of facial muscle movement are described using the motion vector field of pixels in consecutive frames. This is achieved by jointly estimating the motion of binary facial contours and grayscale texture images using the Horn-Schunck optical flow algorithm. By fusing the motion information of these two image modalities, the robustness of facial dynamic features is enhanced. A hidden Markov model is a statistical learning model used to model the state transition probabilities of expression sequences. This can be achieved by training the parameters of basic expression sequences such as neutral and weak expressions using the Baum-Welch algorithm. By establishing the state transition patterns of different expressions, the temporal modeling of dynamic expressions is achieved.

[0114] Specifically, during video stream processing, typical sequences containing neutral, weak, and exaggerated expressions are first extracted from continuous frames as training samples. For each expression sample, a binary contour image and a grayscale texture image are simultaneously generated. The optical flow algorithm is used to calculate the motion vector fields of the two images in time sequence, and a weighted fusion is performed to form a joint optical flow feature. The fused optical flow feature sequence is input into a hidden Markov model for training, and a baseline state transition probability matrix for neutral and weak expressions is established. During the real-time detection phase, the likelihood probability values ​​of the expression sequence to be tested and various baseline models are calculated. When an abnormal expression state transition pattern exceeding a preset threshold is detected, an anomaly determination is triggered.

[0115] Compared with existing technologies, traditional methods rely solely on facial feature analysis of single-frame static images, making it difficult to capture the dynamic characteristics of facial expressions. This solution effectively enhances the detection sensitivity of subtle facial movements by fusing dual-mode optical flow features of binary contours and grayscale textures. It also uses a hidden Markov model to model the dynamic sequence of facial expressions, addressing the shortcomings of traditional methods in temporal correlation analysis.

[0116] Through the above technical solution, this application can accurately identify abnormal micro-expression changes that occur in the judges during the bidding process, such as sudden exaggerated expressions or continuous unnatural expression fluctuations, effectively distinguish normal expression adjustments from abnormal expression patterns that may indicate violations, and improve the temporal modeling capabilities and dynamic feature capture accuracy of facial behavior analysis.

[0117] The environmental objects include mobile phones, cigarettes, pens and paper. If smoke is detected, an alarm is issued. If the audio volume exceeds the set value, it is determined that abnormal behavior has occurred.

[0118] Environmental objects refer to physical objects other than people within the evaluation area captured by a camera. Specifically, image recognition algorithms can be used to classify and detect objects in video frames, such as using convolutional neural networks to locate and identify items like mobile phones and cigarettes. Smoke detection refers to identifying changes in particulate matter concentration in the air using smoke sensors or visual smoke feature analysis techniques, such as using infrared scattering sensors combined with optical flow changes in grayscale image sequences to analyze smoke diffusion trajectories. Volume detection refers to collecting audio signals and calculating decibel values ​​using a microphone array, such as using short-term energy analysis methods to trigger sound pressure levels that continuously exceed a threshold.

[0119] Specifically, during the bid evaluation process, cameras continuously monitor the area where the judges are located, using object detection algorithms to identify the presence of items such as mobile phones and cigarettes. When the image recognition module detects a cigarette, it simultaneously triggers the smoke sensor to sample and analyze air composition. If the visual algorithm detects suspected smoke diffusion characteristics or the sensor detects that the particulate matter concentration exceeds a safety threshold, a smoking violation is determined. Simultaneously, the audio acquisition module monitors the ambient volume in real time. If the decibel level continuously exceeds a preset threshold, such as when the judges engage in loud arguments or unusual noises, the system automatically triggers an alarm. These two detection modes achieve cross-validation of abnormal behavior through multimodal data fusion, preventing misjudgments by a single sensor.

[0120] Compared to existing technologies, existing bid evaluation and monitoring systems typically rely solely on single visual inspection methods, such as manual inspections or simple image recognition to determine the placement of items. However, these methods are unable to effectively identify dynamic smoke diffusion patterns or hidden violations. Traditional methods fail to integrate environmental sensor data with audio signal analysis, resulting in blind spots in detecting behaviors such as smoking and unusual noises. This solution, by integrating visual recognition, smoke sensing, and audio energy analysis technologies, enables multi-dimensional monitoring of static objects and dynamic environmental parameters, addressing the issue of insufficient data coverage from a single modality.

[0121] Through the above technical solution, this application can simultaneously capture the presence of illegal objects and dynamic anomaly signals in the bid evaluation environment. For example, if a judge violates the rules by carrying a mobile phone or smoking, the system can make a dual judgment based on object identification and changes in smoke concentration. Combined with volume anomaly detection, it can effectively identify behaviors such as arguments and suggestive language. This method overcomes the limitations of traditional monitoring systems in analyzing the association between static objects and dynamic behaviors, improving the integrity and real-time nature of bid evaluation supervision.

[0122] Example 3

[0123] A device for analyzing abnormal behavior of judges based on multimodal data includes a module for collecting video streams of the judges' area in real time through multiple cameras; a module for performing frame processing on the video streams and extracting human objects and environmental objects; a module for obtaining facial images and joint point coordinate information of human objects; a module for calculating the angles of joint point vectors in adjacent frames; a module for generating a posture feature matrix by performing feature fusion based on weight distribution; a module for extracting facial expression features and fusing them with posture features; a module for identifying abnormal behavior through a residual network; a module for extracting audio information and converting keywords; and a module for predicting abnormalities based on posture and audio keywords.

[0124] The data acquisition unit refers to the hardware device used to shoot video streams. Specifically, it can be implemented by multiple high-definition camera arrays, and the synchronous acquisition of multi-angle video streams can be achieved through a synchronous trigger mechanism. The conversion module refers to the computing unit used to process video stream data. Specifically, a GPU-accelerated parallel computing architecture can be used to implement frame processing and feature extraction. The judgment unit refers to the behavior recognition algorithm module. Specifically, a deep learning model based on a residual network can be used to implement multimodal data fusion analysis. The alarm unit refers to the abnormal behavior response module. Specifically, an audio-visual alarm device combined with a network communication module can be used to achieve local warnings and remote notifications.

[0125] Specifically, the device uses multiple cameras to synchronously capture video streams of the judges' area and utilizes frame processing and image segmentation techniques to separate people from environmental objects. For human posture analysis, dynamic tracking of joint coordinate sequences and angle change calculations are employed, combined with a weight distribution strategy to generate a fusion feature matrix. For facial expression recognition, regional images are extracted using a face detection model, and expression features are extracted using a lightweight convolutional network. The audio processing module converts text using speech recognition technology and performs matching and screening based on a preset keyword library. Multimodal data is fused via a residual network and then fed into an abnormal behavior classifier. If a violation is detected, the alarm unit is triggered to output a warning signal.

[0126] In some specific embodiments, the data acquisition unit can be configured as a circular camera array, for example, using six 1080P cameras to provide a 360-degree view of the judging area. The conversion module can utilize an embedded processor to implement real-time frame segmentation, for example, processing 30 frames per second. The decision unit can be deployed on a cloud server, for example, using a residual network model built using the TensorFlow framework for distributed inference.

[0127] Compared to existing technologies, existing surveillance devices typically rely solely on single video or audio sensors and lack the ability to integrate and analyze multimodal data. For example, traditional systems cannot simultaneously process changes in human posture and voice keywords, resulting in inability to detect suggestive language or covert movements. This device integrates vision, audio, and deep learning algorithms to perform multi-dimensional behavioral feature correlation analysis, significantly improving the accuracy of anomaly identification in complex scenarios.

[0128] Through the above technical solution, this application solves the problems of traditional evaluator behavior monitoring devices, which are limited in scope and real-time performance, and can automatically identify various violations such as mobile phone use, smoking, and unusual speech. Through collaborative analysis of multimodal data, this device reduces the frequency of manual inspections and the false alarm rate, providing objective and traceable supervision support for the bid evaluation process.

[0129] A system for analyzing abnormal behavior of judges based on multimodal data includes a data acquisition unit for capturing and obtaining a video stream; a conversion module for identifying and analyzing a unit for performing frame processing on the video stream; a determination unit for determining abnormal behavior; and an alarm unit for issuing a warning.

[0130] Among them, the data acquisition unit refers to the image acquisition device deployed in the judges' area, which can be implemented by a multi-angle high-definition camera array to capture the judges' behavior and surrounding environment information through real-time video streams. The conversion module refers to the operation unit that performs structured processing on the original video data. Specifically, a frame segmentation algorithm can be used to cut the continuous video stream into independent image sequences, such as equal segmentation based on time intervals or dynamic framing based on motion detection. The judgment unit refers to the logic processing module that performs multi-dimensional data analysis. Specifically, a neural network model can be used to jointly infer posture, expression, and audio features, such as inputting the fused feature matrix into the residual network for classification decision-making. The alarm unit refers to the response device after the abnormal event is triggered. Specifically, an audible and visual alarm or a network notification interface can be used. For example, when the abnormal score output by the judgment unit exceeds the threshold, an alarm signal is automatically activated.

[0131] Specifically, after the video stream is frame-by-frame processed and converted into an image sequence, the analysis unit extracts human posture features and environmental object information, such as calculating the angle of the motion vector through the coordinates of the joint points. The judgment unit associates the posture features with the audio keyword set for analysis. For example, when it detects that the judge's body suddenly leans forward and high-frequency sensitive words appear in the audio, an abnormality judgment is triggered. The alarm unit executes a graded response strategy based on the judgment result. For example, the first abnormality triggers a low-level buzzer warning, and persistent abnormalities initiate a remote manual verification process. The various units of the system achieve timing synchronization through the data bus to ensure the coordinated operation of video analysis, audio processing and alarm actions.

[0132] Compared to existing technologies, traditional systems rely on single sensors for isolated detection, such as monitoring the number of people through a camera or capturing volume through a microphone. This system, on the other hand, achieves cross-modal data fusion through the collaboration of multiple units. While existing technologies require manual comparison of video and audio recordings, this system establishes a correlation model between gesture features and keywords within the decision unit, enabling automated cross-verification of abnormal behavior. Existing alarm mechanisms are mostly post-event, but this system reduces abnormality determination latency to an acceptable level through real-time frame processing and a multi-threaded computing architecture.

[0133] Through the above-mentioned technical solution, this application solves the problem of traditional monitoring systems' inability to correlate multi-source data in real time, and achieves dynamic joint analysis of judges' behavior and environmental conditions. The system can automatically identify the spatiotemporal correlation between illegal actions and unusual speech, such as capturing the action sequence of a judge holding a cigarette while detecting smoke. The alarm unit's multi-level response mechanism avoids false alarms caused by a single threshold and improves the accuracy of abnormal behavior judgment. The combination of frame processing and feature fusion technology enables the system to process highly concurrent video stream data and adapt to the application requirements of large-scale bidding evaluation scenarios.

[0134] It should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent replacements for some of the technical features therein. However, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for analyzing abnormal behavior of judges, characterized in that: The method comprises the following steps: Use multiple cameras to collect real-time video streams of the area where the judges are located; Performing recognition and analysis on the acquired video stream, performing frame processing on the video stream to obtain N frames of video images, and extracting human objects and environmental objects in each frame of the video image, where the environmental objects include other tools except human objects; Obtain the facial image and body joint coordinate information of the person in the video image, and obtain the 18 joint coordinate sequences {(x11, y11), (x12, y12), …, (x1N, y1N)} of the current frame image; The vector angle θ of the same joint point in adjacent frames 1N The calculation of the joint point dynamic information {θ 11 ,θ 12 ,…,θ 1N }; The position and dynamic information of the joint points are calculated according to the weight f t Allocation, feature fusion is performed, expressed as 18*T*4 posture feature information matrix S1; Use MTCNN to extract the face area, and use MobileNet to extract the expression features of the detected face image, expressed as K1 of 16*T*1; The human expression information K1 and the human posture information S1 are fused and expressed as a 18*T*4 feature matrix S2; The fused feature matrix is ​​passed through the abnormal behavior recognition model of the residual network with noise robustness to obtain the final human posture behavior recognition result; Extract audio information from the video stream, convert the audio information into text and extract keywords to obtain an audio keyword set; According to the human body posture and audio keyword set, abnormal behavior is predicted to obtain abnormal behavior prediction results.

2. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: The angle of the joint point is calculated using the following formula:

3. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: The weights f are assigned as follows t Calculation: According to the intensity of the change in the direction and speed of the joint point, the weight f is assigned to the T frame image. T , represents the predicted probability value of a single frame, and calculates the final weight f of the T-th key frame T The formula is as follows where θ t Indicates the maximum angle change of the joint point in the T-th image frame, v t represents the maximum velocity change of the joint point in the Tth image frame, k1 and k2 are the weight values ​​of angle and velocity.

4. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: The step of converting the audio into text and extracting keywords to obtain an audio keyword set includes: Based on ASR technology, the audio is converted into text to obtain the text to be analyzed; Segmenting the text to be analyzed to obtain a phrase set; Acquire a keyword set corresponding to the video stream from a preset keyword library as a target keyword set; Match each phrase in the phrase set with the test keywords in the target keyword set to obtain a matching result; If there is a successful matching result, each of the phrases corresponding to the successful matching result is used as the audio keyword set.

5. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: The abnormal behavior detection model uses a noise-robust residual network: The residual network with INSP activation function is used as the abnormal behavior recognition model. The network adds residual units on the basis of VGG-19 and replaces the ReLU function with the NSP function for improvement. Where k represents the control of noise scale, S represents the scale parameter, x represents the input value of the previous convolution layer, and w i Represents the weight of the previous convolution layer, i represents the i-th neuron, τ syn represents the synaptic time constant.

6. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: If it is detected that the total number of people in the video stream exceeds the set number and the duration exceeds the set time, an alarm will be issued and a preliminary judgment will be made that abnormal behavior has occurred; manual intervention will be required for verification.

7. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: During the facial image detection, facial expression samples are obtained after face detection, wherein the facial expression samples include various expression sequence samples, and the various expression sequences include neutral expression sequences, weak expression sequences, and exaggerated expression sequences; For the facial expression sample, extracting features of the optical flow motion field by fusing the binary image and the grayscale image sequence to obtain a feature sequence; Using the feature sequence as an observation sequence, training the neutral expression sequence and the weak expression sequence in the facial expression sample, and establishing corresponding hidden Markov models for various expressions; Combine hidden Markov models of various expressions to form a facial expression classifier; If the detected facial expression shows any changes beyond the preset expression compared to the preset facial surface, it is determined that abnormal behavior has occurred.

8. The method for analyzing abnormal behavior of judges according to claim 1, characterized in that: The environmental objects include mobile phones, cigarettes, pens and paper. If smoke is detected, an alarm is issued. If the audio volume exceeds the set value, it is determined that abnormal behavior has occurred.

9. A device for analyzing abnormal behavior of judges based on multimodal data, characterized in that: The invention comprises an analysis method suitable for abnormal behavior of judges according to any one of claims 1 to 8.

10. A system for analyzing abnormal behavior of judges based on multimodal data, characterized in that: include: A data acquisition unit, used for capturing and obtaining a video stream; A conversion module, used for identifying and analyzing the unit, and for performing frame processing on the video stream; A determination unit, used for determining abnormal behavior; Alarm unit, used to issue warnings.

Images