A court in-house gown replacement behavior detection method based on a classification model and feature fusion

Abstract

The application provides a court in-house gown replacement behavior detection method based on a classification model and feature fusion, and belongs to the technical field of video image processing. The method first collects a court trial image, extracts a judge position by using a judge target detection model based on YOLOv8s, and assigns a tracking ID to the judge and performs EMA time sequence smoothing filtering by using a DeepSORT multi-target tracking algorithm. In the initial tracking stage, the detection box region corresponding to each tracking ID is sliced, an initial gown wearing state is obtained by inputting a ResNet50 gown classification model, and HSV color features and HOG texture features of each slice are extracted to construct personal clothing baseline features. In subsequent video frames, the real-time target region personal clothing baseline features of the same ID personnel are extracted, and the comprehensive feature distance between the same ID personnel and the bound baseline features is calculated. By comparing the comprehensive feature distance with a threshold, an abnormal alarm is output for the tracking ID with a dressing change behavior. The application realizes high-precision, low-delay and robust real-time monitoring.

Description

Technical Field

[0001] This invention relates to a method for detecting court robe changing behavior based on classification models and feature fusion, belonging to the field of video image processing technology. Background Technology

[0002] For a long time, the regulation of judges' robe attire has mainly relied on manual observation. This involves on-site supervision by court security personnel or real-time monitoring of court proceedings by video surveillance staff to manually identify whether judges have removed their robes or changed clothes in violation of regulations. This method is currently the mainstream management approach used by most courts in China, but it has inherent technical shortcomings: court proceedings are long, and there are many people in a collegial panel setting, making it difficult for staff to maintain a high level of concentration for extended periods. This can easily lead to missed or misjudged violations due to visual fatigue. At the same time, manual judgment relies on subjective experience and lacks unified quantitative standards, failing to meet the core requirements of objectivity, real-time performance, and traceability in the current construction of intelligent judicial systems.

[0003] In recent years, with the rapid development of computer vision and deep learning technologies, image processing and video analysis technologies have been widely applied in various scenarios of judicial intelligence. In the field of target detection and multi-target tracking, technologies such as YOLO, R-CNN, and DeepSORT have enabled accurate localization and continuous tracking of personnel targets in complex scenes. In the field of fine-grained image classification, convolutional neural network models such as ResNet have achieved high-precision recognition of clothing attributes through hierarchical feature extraction. These general technologies have laid a solid technical foundation for the intelligent analysis of court proceedings in court settings.

[0004] However, despite significant progress in general object detection and image classification technologies, existing solutions still face the following challenges and problems in the specific scenario of a courtroom, specifically for the automatic detection of judges changing their robes: First, in multi-object scenarios, personnel identity tracking is unstable and lacks utilization of prior knowledge of the courtroom scene. In the fixed courtroom setting, existing object detection models are prone to losing fine-grained features of robe edges and human contours due to stride convolution downsampling, resulting in insufficient fit between the detection box and the actual human body boundary. Simultaneously, fixed backgrounds such as the judge's bench, chairs, and microphones can interfere with target feature extraction, leading to jitter and jumps in single-frame detection results. Furthermore, existing tracking algorithms are highly susceptible to ID switching or tracking loss when personnel are occluded, move, or enter / exit the frame, making it impossible to consistently and stably bind the detection results of consecutive frames to the same identity. At the same time, existing technologies do not make full use of the inherent prior knowledge in court settings, such as the relatively fixed number of judges and the fact that judges are usually in fixed areas such as the judges' bench. The lack of constraints and optimizations on the tracking results makes it difficult to stably bind identity and behavior in scenarios where multiple judges appear in court together, and it is impossible to provide reliable identity evidence for tracing subsequent costume-changing behavior.

[0005] Secondly, the feature representation for judging the state of judicial robes is simplistic, lacking interpretability and discriminative power. Furthermore, limited by the insufficient localization accuracy of target detection, subsequent feature extraction is easily contaminated by background areas. Existing methods mostly rely on high-dimensional abstract features extracted by deep learning models for discrimination. While these features are highly generalizable, their physical meaning is unclear and their interpretability is weak, especially when judicial robes are highly similar in appearance to dark everyday clothing, limiting their discriminative effectiveness. As judicial robes are exclusively worn by the judiciary, their color and texture are intuitive and stable discriminative cues. However, existing technologies have failed to effectively integrate these physical and abstract features, resulting in insufficient robustness to discrimination under interference such as changes in lighting and posture adjustments.

[0006] Furthermore, there is a difficult trade-off between real-time performance and computational overhead, and an efficient dynamic scheduling mechanism is lacking. Ideally, to capture the instantaneous action of donning and removing robes, a deep learning model would be needed for state determination in every frame. However, the computational load of model inference is high, and calling the model frame-by-frame would result in excessive computational consumption. Reducing the determination frequency, on the other hand, could easily miss crucial clothing-changing actions. Current technologies lack a mechanism to dynamically adjust the detection strategy based on changes in clothing characteristics, making it difficult to reduce computational overhead while ensuring accuracy and continuity. Summary of the Invention

[0007] The purpose of this invention is to provide a method for detecting judges changing robes in court based on classification models and feature fusion. This method combines prior knowledge of the court scene, an improved target detection model, and lightweight feature distance calculation to achieve high-precision, low-latency, and robust real-time monitoring of judges' robe-changing behavior.

[0008] To achieve the above objectives, the present invention employs the following technical solution: A method for detecting court robe changing behavior based on classification model and feature fusion includes the following steps: Courtroom images were acquired, and the judge's position was extracted using a judge target detection model based on YOLOv8s. The judge target detection model replaced the original convolutional modules with SPD-Conv modules in the backbone and neck networks, and added a PA position attention module after the SPPF module. The detection head used a DyHead dynamic detection head. The DeepSORT multi-target tracking algorithm is used to assign tracking IDs to judges in court proceedings, and the coordinates of the detection boxes with assigned tracking IDs are subjected to EMA temporal smoothing filtering. In the initial tracking phase, the detection box region after EMA temporal smoothing filtering corresponding to each tracking ID is sliced, and the sliced ​​images are input into the pre-trained ResNet50 robe classification model to obtain the initial robe wearing state of the person with that ID. At the same time, the HSV color features and HOG texture features of each frame slice are extracted to construct the personal clothing baseline features, and the initial wearing state and personal clothing baseline features are bound and stored with the corresponding tracking ID. In subsequent video frames, the baseline features of personal clothing are extracted from the real-time target area of ​​the same ID person, and the comprehensive feature distance between the current personal clothing baseline features and the personal clothing baseline features bound to the tracking ID is calculated to form a comprehensive feature distance sequence for consecutive frames of the ID. Based on the comparison results of the comprehensive feature distance sequence and the threshold, and combined with continuous frame verification, it is determined whether a costume change has occurred; for the tracking ID that has undergone a costume change, an abnormal alarm containing the costume change information of that tracking ID is output.

[0009] Preferably, the backbone network adopts a structure of alternating cascaded SPD-Conv modules and C2f modules, and all downsampling nodes are replaced with SPD-Conv modules to output three sets of multi-scale basic feature maps; The backbone network of the judge target detection model includes, in sequence: a first SPD-Conv module, a second SPD-Conv module, a first C2f module, a third SPD-Conv module, a second C2f module, a fourth SPD-Conv module, a third C2f module, a fifth SPD-Conv module, a fourth C2f module, an SPPF module, and a PA position attention module.

[0010] Preferably, the neck network uses a bidirectional topology structure of FPN feature pyramid and PANet path aggregation to complete cross-scale feature fusion and output three sets of multi-scale fused feature maps; The neck network processes the feature map as follows: The feature map P5 output by the PA position attention module of the backbone network is upsampled and concatenated with the feature map P4 output by the third C2f module of the backbone network, and then input into the C2f module to obtain feature map F1. The feature map F1 is upsampled and concatenated with the feature map P3 output by the second C2f module of the backbone network, and then input into the C2f module to obtain feature map F2. The feature map F2 is concatenated with feature map F1 by the SPD-Conv module and then input into the C2f module to obtain feature map F3. The feature map F3 is concatenated with feature map P5 by the SPD-Conv module and then input into the C2f module to obtain feature map F4. The feature maps F2, F3, and F4 are input into the detection head.

[0011] Preferably, the detection head calls the feature alignment module to unify the spatial resolution of the three sets of tensors, and sends them to the dynamic detection head module composed of cascaded DyHeadBlocks to perform triple dynamic calculations of scale awareness, spatial awareness and task awareness, and outputs dynamically enhanced feature tensors, which are then split and mapped to three independent multi-scale detection heads by the feature segmentation module.

[0012] The feature alignment module unifies the spatial resolution of feature maps F2, F3, and F4 through bilinear interpolation algorithm, and combines 3×3 deformable convolution to complete spatial grid resampling and channel parameter mapping compression. After unifying and aligning heterogeneous features, the module stacks them to generate an aggregated feature tensor. The dynamic detection head module is composed of DyHeadBlock1 and DyHeadBlock2 cascaded together. It performs scale-aware weight multiplication and addition operations based on adaptive average pooling matrix, spatially aware deformable convolution operations based on offset coordinate learning, and dynamic activation function switching operations based on routing weight allocation, outputting a dynamically enhanced feature tensor with unchanged dimensions. The feature segmentation module slices and unbinds the dynamically enhanced feature tensor along the scale dimension, and then maps it to three independent multi-scale detection heads with three different resolutions.

[0013] Preferably, the initial tracking phase consists of five consecutive frames after the tracking ID is assigned to the personnel; The initial robe-wearing state is the classification result with the highest proportion of the ResNet50 robe classification model in five consecutive frames of images; The personal clothing baseline features are based on five consecutive frames of images. The HSV color features and HOG texture features of each frame are extracted. The element-wise arithmetic mean of the HSV color feature vector and the HOG texture feature vector is calculated to obtain the baseline color features and baseline texture features as personal clothing baseline features.

[0014] Preferably, the comprehensive feature distance between the current personal clothing baseline features and the personal clothing baseline features bound to the tracking ID is calculated, specifically as follows: The real-time target detection bounding box region of the same tracked person in subsequent video frames is sliced ​​and its size is normalized. HSV color features and HOG texture features consistent with the initial tracking stage are extracted. Calculate the Bartlett distance between the HSV color feature of the current tracking ID person and the baseline color feature bound to the tracking ID, and calculate the cosine distance between the HOG texture feature of the tracking ID person and the baseline texture feature bound to the tracking ID. First, the range of the Bach distance is normalized, and then it is weighted and fused with the cosine distance to obtain the comprehensive feature distance and form a distance sequence.

[0015] Preferably, the specific steps for determining whether a change of clothes has occurred are as follows: A comprehensive feature distance judgment threshold is set. The threshold is obtained by collecting multiple sets of image sample pairs of judges wearing robes and not wearing robes, calculating the average comprehensive feature distance of the sample pairs, and then multiplying it by an adjustment coefficient k. When the cumulative number of consecutive frames exceeding the comprehensive feature distance judgment threshold is greater than 10, it is comprehensively determined that the person with that ID has changed clothes.

[0016] Preferably, the sliced ​​image of the current detection box region of the tracked ID person who has changed clothes is re-input into the ResNet50 robe classification model to update the robe wearing status; select five consecutive frames after the change of clothes is determined, re-slice the target detection box region corresponding to the tracked ID, extract the personal clothing baseline features, and update the personal clothing baseline features of the tracked ID person.

[0017] The advantages of this invention are as follows: It enables real-time monitoring of judges' robe-changing behavior within the courtroom. By combining a robe status classification model with clothing dual-feature fusion comparison, it achieves accurate detection of judges' initial robe-wearing verification and in-court robe-changing behavior. Addressing issues such as ID switching and loss, and detection box jitter during personnel tracking, it introduces prior knowledge to optimize multi-target tracking and combines EMA temporal smoothing filtering with the lossless downsampling and background suppression capabilities of a YOLOv8s-based judge target detection model, further improving identity binding stability and target positioning accuracy. Employing HSV and HOG dual-feature fusion discrimination, it fully utilizes the inherent physical characteristics of the robe to effectively distinguish between robes and dark casual wear, enhancing detection robustness in complex scenarios such as lighting changes and posture adjustments. The process adopts a lightweight design, only calling the computationally intensive ResNet50 classification network for status verification at the initial moment and after robe-changing determination. During the real-time tracking phase of the trial, only lightweight feature extraction and distance calculation are used, significantly reducing computational resource consumption and providing stable and reliable technical support for standardized and intelligent management of the trial scene. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0020] Figure 2 This is a flowchart illustrating the specific implementation of the present invention.

[0021] Figure 3 This is a structural diagram of a judge object detection model based on YOLOv8s. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 like Figures 1-2 As shown, this invention presents a method for detecting robe-changing behavior in court based on a classification model and feature fusion. The essence of robe-changing behavior is a significant abrupt change in the visual features of the judge's upper garment, requiring accurate determination of the actual wearing state of the robe. Therefore, a two-level logic can be used to detect robe-changing behavior through initial screening based on continuous changes in clothing features and state verification using a classification model. This invention simplifies the detection of robe-changing behavior and ensures real-time detection by using a multi-feature fusion distance calculation method to transform complex changes in clothing appearance into quantifiable feature distance values. For feature selection, the HSV color histogram features and HOG gradient texture features of the clothing are extracted, and the Bach distance and cosine distance between the two types of features and the baseline feature are calculated and weighted to ensure a comprehensive representation of the appearance changes of the clothing. For state determination, a ResNet50 robe-wearing classification model is used to accurately label the robe-wearing state in the initial tracking frame and verify the state after suspected robe-changing behavior, clarifying the specific type of clothing change and avoiding misjudgments caused by relying solely on feature changes. In terms of discrimination logic, the algorithm combines prior knowledge of court trial rules and fully considers the differences in the number of personnel in different trial scenarios such as single judges and collegial panels. It adopts a two-level discrimination logic of initial screening with feature distance thresholds in multiple consecutive frames and ResNet50 state verification to ensure the comprehensiveness, accuracy and scenario adaptability of the algorithm.

[0024] Specifically, it includes the following steps: S1: Acquire courtroom images and extract the judge's position using a judge target detection model based on YOLOv8s; the judge target detection model replaces the original convolutional modules with SPD-Conv modules in the backbone and neck networks, and adds a PA position attention module after the SPPF module, and uses a DyHead dynamic detection head for the detection head; S2: Use the DeepSORT multi-target tracking algorithm to assign tracking IDs to the judges in the trial, and perform EMA time-series smoothing filtering on the coordinates of the detection boxes with assigned tracking IDs; S3: In the initial tracking stage, the detection box region after EMA temporal smoothing filtering corresponding to each tracking ID is sliced, and the sliced ​​images are input into the pre-trained ResNet50 robe classification model to obtain the initial robe wearing state of the person with that ID. At the same time, the HSV color features and HOG texture features of each frame slice are extracted to construct the personal clothing baseline features, and the initial wearing state and personal clothing baseline features are bound and stored with the corresponding tracking ID. S4: In subsequent video frames, extract the personal clothing baseline features of the real-time target area of ​​the same ID person, calculate the comprehensive feature distance between the current personal clothing baseline features and the personal clothing baseline features bound to the tracking ID, and form a comprehensive feature distance sequence for consecutive frames of the ID. S5: Based on the comparison results of the comprehensive feature distance sequence and the threshold, and combined with continuous frame verification, determine whether a costume change has occurred; for the tracking ID that has undergone a costume change, output an abnormal alarm containing the costume change information of that tracking ID.

[0025] As a refinement of the above embodiments, in step S1, the judge object detection model is based on the native YOLOv8s framework. The convolutional downsampling layers with a stride of 2 in the backbone and neck networks are replaced with SPD-Conv modules to retain fine-grained spatial features. A PA position attention module is introduced at the end of the backbone to effectively suppress visual interference from the background of the judge's bench on the judge's body outline, ensuring a more fitting detection box is output. The original detection head is replaced with the DyHead unified dynamic detection head to achieve cross-dimensional feature enhancement. The collected courtroom images must be acquired with the consent of the court and the parties involved.

[0026] like Figure 3 As shown, the specific implementation process is as follows: (1) In the Backbone and Neck stages of the YOLOv8s model, the SPD-Conv module replaces all convolutional downsampling layers with a stride of 2 in the original model. The SPD-Conv module includes a Space-to-Depth spatial rearrangement unit and a convolutional unit with a stride of 1. In courtroom scenarios, the original convolutional downsampling with a stride of 2 is prone to losing fine-grained features such as the edges of judges' robes and the folds of clothing, resulting in inaccurate detection box boundaries. In this embodiment, lossless downsampling is achieved through SPD-Conv: For the input feature map, the Space-to-Depth spatial rearrangement unit first performs spatial slicing and channel dimension reorganization on the feature map with a downsampling ratio of r=2 to obtain the rearranged feature map, realizing the lossless transfer of spatial dimension information to channel dimension and avoiding the loss of pixel information caused by the original convolutional layer with a stride of 2. Then, a convolutional unit with a stride of 1 is used to perform feature fusion and channel dimensionality reduction on the rearranged feature map. While halving the size of the feature map, the fine-grained spatial information of the judge's clothing edges and folds is fully preserved, providing a feature basis for the accurate localization of the detection box.

[0027] (2) A PA position attention module is introduced into the output feature layer at the end of the backbone. The long-distance dependency between any two positions in the feature map is captured through the self-attention mechanism. The multi-scale features are spatially weighted to solve the problem of visual interference between the fixed background of the judge's bench and the boundary of the judge's human body. The specific process is as follows: Feature embedding and dimensionality transformation: For the input feature map, feature embedding is first performed through three 1×1 convolutional layers to generate query feature Q, key feature K and value feature V, and their spatial dimensions are flattened.

[0028] Self-attention weight calculation: Perform matrix multiplication between the transpose of query feature Q and key feature K, and generate a spatial attention weight matrix through the Softmax activation function.

[0029] Attention weighting and feature fusion: The attention weight matrix S is multiplied with the value feature V, and then the feature is transformed through a 1×1 convolutional layer. Finally, the residual is fused with the original input feature map to obtain the output feature map Fout.

[0030] (3) Multi-scale feature unification and triple dynamic attention enhancement: The three feature maps of different scales output by the Neck network are aligned, and the three are unified to the same spatial dimension by spatial upsampling or downsampling. Then, they are spliced ​​along the hierarchical dimension to construct a three-dimensional feature tensor containing the hierarchical L, spatial S and channel C dimensions. The three-dimensional feature tensor is input into a unified dynamic detection head composed of two serially cascaded DyHeadBlocks. For each dynamic head block, the three-dimensional feature tensor is purified across dimensions through the following three attention mechanisms: First, the scale-aware attention module uses global average pooling to compress spatial dimensions and extract cross-level context. Then, it learns feature weights at different levels through 1×1 convolution and HardSigmoid activation, weighting the input features element-wise and adaptively fusing the judge's body and robe features from different receptive fields to address the issue of large differences in the judge's near and far scales in the collegial panel scenario. Second, the spatial awareness attention module uses deformable convolution to dynamically sample the spatial dimensions of the feature map. By learning the offset, the convolution kernel adaptively follows the edges of the severe non-rigid deformation when the robe is put on and taken off. Combined with Sigmoid activation, the output spatial attention weights are fused with residuals, forming a dual background suppression with the PA position attention module, further weakening the fixed background interference from the judge's table, chair, and microphone. Finally, the task-aware attention module uses global average pooling to extract global representations of the channels. Through dual fully connected layers and dynamic activation functions, it dynamically allocates the activation states of the feature channels, balancing the dual task requirements of subsequent target classification and bounding box regression, and resolving the feature conflict between fine classification of robes and dark casual clothes and accurate localization of detection boxes.

[0031] (4) The three-dimensional feature tensor after reconstruction by the unified dynamic detection head is decomposed into three independent scales of enhanced feature maps along the hierarchical dimension and sent into the corresponding decoupled detection head respectively. The decoupled detection head contains independent classification branches and regression branches. The classification branch extracts category features through convolutional layers. The regression branch extracts position features through convolutional layers and outputs target detection boxes that accurately fit the real boundary of the judge's body deformation. After the detection box coordinates are smoothed by subsequent EMA time-series filtering, they can be directly used for slicing the clothing area to eliminate the pollution caused by the background area to the subsequent HSV color features and HOG texture features extraction, thereby improving the accuracy of robe changing behavior detection from the source. (5) Model training: Collect judge target images in court scenes for two trial modes, namely, a single judge and a collegial panel, covering different lighting conditions, human postures and standing positions, and construct a special target detection dataset for court judges; use the official native YOLOv8s pre-trained weights as initialization parameters and adopt the transfer learning strategy to fine-tune the improved judge target detection model.

[0032] Specifically, the overall detection and processing steps include: Backbone Network Processing Stage: In the backbone network processing stage, the backbone network uses an input image with dimensions of 1×3×640×640 as the initial input. It employs an alternating cascaded structure of SPD-Conv and C2f modules to complete feature extraction and forward propagation computation. All original convolutional downsampling layers with a stride of 2 are replaced by the SPD-Conv module. The SPD-Conv module sets a spatial slicing stride r=2. First, it spatially divides the input feature map into a 2×2 grid, extracting local pixel blocks across rows and columns. Then, it concatenates the four sub-tensors obtained from the division along the channel dimension, reducing the spatial resolution of the output feature map to half of its original value while expanding the number of channels to four times. Subsequently, it uses convolutional layers with a kernel size of 3×3 and a stride of 1, combined with BatchNormalization and SiLU activation functions, to perform channel mapping and feature fusion on the concatenated features, thereby effectively preserving fine-grained spatial feature information during downsampling. The C2f module performs splitting processing on the input features through a multi-branch residual structure. One portion of the features is directly passed as a residual branch, while the other portion is refined step-by-step through multiple convolutional layers and finally fused with the residual branches. This achieves gradient separation and feature reuse, improving feature representation capabilities. During feature propagation, the backbone network sequentially derives three sets of multi-scale feature branches at different depth levels: when the feature map spatial resolution decreases to 80×80, a shallow feature branch is derived, which outputs a 1×C1×80×80 feature tensor after processing by the C2f module; when the resolution decreases to 40×40, a mid-level feature branch is derived, which outputs a 1×C2×40×40 feature tensor after processing by the C2f module; and when the resolution decreases to 20×20, a deep feature branch is derived. For deep features, the input is first processed by the SPPF module before output. The SPPF module extracts contextual information from the feature map at different receptive fields through three consecutive 5×5 max pooling operations, and concatenates the pooling results with the original features along the channel dimension to achieve multi-scale receptive field feature fusion. The fused features are then input into the PA positional attention module. The PA module first performs a linear mapping on the input feature map through three 1×1 convolutional layers to generate query features Q, key features K, and value features V, flattening the spatial dimension of the feature map into a one-dimensional vector. Then, a spatial correlation matrix is ​​obtained by matrix multiplication of the transposes of query features Q and key features K, and normalized using the Softmax function to obtain a spatial attention weight matrix. This attention weight matrix is ​​then multiplied with the value features V to achieve weighted aggregation of global spatial positional features. Finally, a 1×1 convolution is used to linearly transform the weighted features, and they are element-wise added to the original input features to form a residual connection, resulting in the enhanced output features.Finally, the backbone network outputs three sets of multi-scale basic feature maps, namely 1×C1×80×80, 1×C2×40×40 and 1×C3×20×20, which are transmitted to the neck network.

[0033] The backbone network of the judge target detection model includes, in sequence: a first SPD-Conv module, a second SPD-Conv module, a first C2f module, a third SPD-Conv module, a second C2f module, a fourth SPD-Conv module, a third C2f module, a fifth SPD-Conv module, a fourth C2f module, an SPPF module, and a PA position attention module.

[0034] The neck network processing stage: The neck network takes three sets of multi-scale basic feature maps output from the backbone network as input: 1×C1×80×80 (P3 feature map), 1×C2×40×40 (P4 feature map), and 1×C3×20×20 (P5 feature map). It employs a bidirectional topology structure combining FPN feature pyramid and PANet path aggregation to complete cross-scale feature fusion. In the top-down FPN path, the nearest neighbor interpolation upsampling module is first called to enlarge the 1×C3×20×20 deep feature tensor to a 40×40 spatial size by a factor of 2. This tensor is then concatenated with the laterally input 1×C2×40×40 mid-level feature tensor in the channel dimension. After concatenation, the C2f module processes and outputs a 1×C2'×40×40 tensor. This tensor is then enlarged to an 80×80 spatial size again by the UpSample module and concatenated with the 1×C1×80×80 shallow feature tensor in the channel dimension. Processed with the C2f module, it outputs a 1×C1'×80×80 tensor. In the bottom-up PANet path, the SPD-Conv module (spatial slicing stride r=2) is called to perform lossless downsampling of the 1×C1'×80×80 fused feature tensor by spatial slicing and channel augmentation, changing its spatial size to 40×40. Then, it is concatenated with the 1×C2'×40×40 tensor and computed with the C2f module to output a 1×C2' tensor. The '×40×40 tensor is then downsampled to a 20×20 spatial size by calling the SPD-Conv module again. This tensor is then combined with the 1×C3×20×20 tensor to complete channel concatenation and C2f module calculation, outputting a 1×C3''×20×20 tensor. Finally, the neck network outputs three sets of multi-scale fused feature maps: 1×C1'×80×80, 1×C2''×40×40, and 1×C3''×20×20, which are directly transmitted to the head network.

[0035] In the detection head processing stage: the detection head takes three sets of multi-scale fused feature maps output by the neck network as input. First, it calls the FeatureAlignment module to unify the spatial resolution of the three feature maps using bilinear interpolation. Then, it combines 3×3 deformable convolution to achieve adaptive spatial sampling and feature fusion, resulting in a unified multi-scale feature representation. This feature is then fed into the dynamic detection head module, which consists of a cascade of DyHeadBlock1 and DyHeadBlock2. It performs three dynamic calculations sequentially: scale-aware weight calculation based on global pooling, spatially aware deformable convolution operation based on offset learning, and a dynamic feature modulation mechanism based on routing weight allocation, thus ensuring consistent output dimensionality. The feature representation is dynamically enhanced. The FeatureSplit module distributes the enhanced features to three multi-scale Detect heads according to scale information, corresponding to resolutions of 80×80, 40×40, and 20×20. Each Detect module employs a decoupled classification and regression architecture. The classification branch uses multiple layers of stacked 3×3 convolutions and 1×1 convolutions to output a class confidence probability matrix with dimensions 1×Num_Classes (total number of target classes)×H×W. The regression branch simultaneously uses multiple layers of stacked 3×3 convolutions and 1×1 convolutions to output bounding box center coordinate offsets and normalized width and height parameter matrices with dimensions 1×4×H×W. Both outputs together generate the initial candidate detection box set. H is the feature map height, and W is the feature map width. Finally, the candidate detection boxes are screened by the non-maximum suppression (NMS) module. First, candidate boxes with confidence scores below a preset threshold (preferably 0.5) are filtered out. Then, overlapping detection boxes are suppressed based on the intersection-over-union (IoU) calculation. Redundant detection boxes with IoU scores greater than a preset threshold (preferably 0.45) are removed. The final output is a detection result matrix with a dimension of M×6, where M represents the number of effective detection targets, and each row corresponds to the bounding box coordinates, confidence score, and category information, respectively.

[0036] As a refinement of the above embodiment, in step S2, the detection result of the judge target detection model is input into the DeepSORT multi-target tracking algorithm to assign a unique and continuous tracking ID to the judge.

[0037] EMA (Exponential Moving Average) temporal smoothing filtering is applied to the target detection box coordinates assigned a unique tracking ID. The initial frame smoothing detection box for the corresponding tracking ID directly uses the original detection box output by the model. Weighted smoothing calculation is performed starting from the second frame of that ID to obtain a stable target detection box. This mitigates the differences in subsequent feature extraction caused by detection box jitter and jumps. The calculation logic and formula are as follows: In the formula, the detection box coordinates include four dimensions: the x-coordinate of the top-left corner of the target box, the y-coordinate, the width w, and the height h. The smoothing calculation is performed independently for each of the four dimensions. For this tracking ID number Single-dimensional parameter values ​​of the detection box after frame smoothing; For the judge object detection model The frame output contains the dimension parameter values ​​corresponding to the original detection box with that ID; For this ID number The corresponding dimension parameter values ​​of the detection box after frame smoothing; The smoothing weight coefficient has a value range of [0.2, 0.5].

[0038] As a refinement of the above embodiments, step S3 specifically includes: S301: In the initial frame (within 5 consecutive frames) after the tracking ID first appears, slice the target detection box corresponding to the ID in each frame, scale the image to a fixed size of 224×224, complete pixel normalization processing, and obtain 5 sets of standardized images. S302: Five sets of standardized images are obtained. The HSV color histogram features and HOG texture features of each frame are extracted. The element-wise average of the HSV feature vectors from the five frames is taken as the baseline color feature, and the element-wise average of the HOG feature vectors from the five frames is taken as the baseline texture feature. Together, they constitute the clothing baseline features for this ID. Multi-frame averaging effectively suppresses single-frame noise and transient interference, obtaining a stable and reliable feature representation. The calculation formula is as follows: right The baseline color feature is obtained by averaging the element-wise feature vectors obtained from frame slices. : , in, , , These are the hue, saturation, and luminance components of the frame, respectively. For the first The HSV features of a frame slice are three-dimensional vectors. The baseline texture features are obtained by taking the element-wise average of the feature vectors obtained from N frame slices. : , in, For HOG feature dimensions, For the first HOG feature vectors of frame slices.

[0039] S303: Input the normalized sliced ​​image into the pre-trained ResNet50 robe classification model, and count the frequency of occurrence of the two categories of wearing robes and not wearing robes. Take the category with the highest frequency in five frames as the initial robe wearing state of the person with that ID, and bind and store this state with the corresponding tracking ID. The ResNet50 model is based on ImageNet pre-trained weights and is fine-tuned through image samples of judges wearing robes and not wearing robes in court scenes. The model output is the probability value of the two categories, corresponding to wearing robes and not wearing robes, respectively. Take the category with the maximum probability as the robe wearing state of the person with that ID in that frame.

[0040] As a refinement of the above embodiments, step S4 specifically includes: 401: Slice the real-time target detection bounding box region of the same tracking ID person in the subsequent video frames after the end of the initial tracking phase and perform size normalization processing to extract HSV color features and HOG texture features consistent with the initial tracking phase. 402: Calculate the Bartlett distance between the current HSV color feature and the baseline HSV color feature bound to this ID, and the cosine distance between the current HOG texture feature and the baseline HOG texture feature bound to this ID. The calculation formula is denoted as: , in, The Bartholomew's distance is the color feature of HSV, with a value ranging from [0, +∞). A larger value indicates a greater difference in the distribution of the current clothing color and the baseline color features. The first HSV color feature of the current frame The probability values ​​of each component after L1 normalization The first of the baseline HSV color features The probability values ​​of each component after L1 normalization, where L1 normalization means dividing each component of the vector by the sum of its absolute values ​​so that the sum of the normalized components is 1. , in, This represents the cosine distance of the HOG texture features, with a value ranging from [0, 1]. A larger value indicates a greater difference between the current clothing texture and the baseline texture features. This is the HOG texture feature vector of the current frame. The baseline HOG texture feature vector, The L2 norm of an eigenvector is the square root of the sum of the squares of its components.

[0041] 403: First, normalize the range of the Bach distance, then... and Weighted fusion yields a comprehensive feature distance, forming a distance sequence, where... The feature weights are set to 0.6. The feature weights are set to 0.4.

[0042] Specifically, the feature weights are obtained as follows: (1) Collect no less than 30 sets (preferably 50-100 sets) of image sample pairs of judges wearing robes and not wearing robes. Each sample pair consists of two images of the same judge in the same scene and in the same posture. Then, manually label all sample pairs with the true value of changing clothes. The labeling rule is: if one image in the sample pair is wearing a robe and the other is not wearing a robe, it is labeled as a positive sample of changing clothes; if both images in the sample pair are wearing robes or not wearing robes, it is labeled as a negative sample of not changing clothes.

[0043] (2) Based on the sample set completed by the annotation, the double-layer nested traversal method is used to jointly select the feature weight and adjustment coefficient k. The outer traversal takes the color feature weight candidate value with a value range of [0,1] and a step size of 0.05. The texture feature weight takes the value of 1 minus the color feature weight.

[0044] (3) For each group of candidate feature weights, calculate the weighted fusion result of normalized Bach distance and cosine distance of all labeled sample pairs to obtain the comprehensive feature distance of all samples, and calculate the average value of the comprehensive feature distance.

[0045] While fixing the outer layer weights, the inner layer iterates through the adjustment coefficient k candidate values ​​with a range of (0.5, 0.9) and a step size of 0.05. The average distance of the comprehensive features is multiplied by the current k candidate value to generate the corresponding temporary judgment threshold. (4) Based on the temporary feature weight and temporary judgment threshold, the static sample pairs are judged. If the comprehensive feature distance of the sample pair exceeds the temporary threshold, it is judged that the clothing change behavior has occurred.

[0046] (5) Compare the judgment results with the manually labeled true values ​​of the costume change, and calculate the F1 score of the costume change detection for the dataset under the set of parameters.

[0047] Using the maximization of the F1 score as the optimal screening criterion, after traversing all candidate parameter combinations, the optimal color feature weight and the optimal texture feature weight are finally determined, and the optimal adjustment coefficient k is obtained. The optimal color feature weight was determined to be 0.6, the optimal texture feature weight to be 0.4, and the optimal adjustment coefficient k for matching was determined to be 0.75.

[0048] 404: Store the comprehensive feature distances calculated for each frame in the video stream in chronological order to form a sequence of comprehensive feature distances for consecutive frames corresponding to the tracking ID.

[0049] As a refinement of the above embodiments, step S5 specifically includes: 501: Set a comprehensive feature distance judgment threshold. This threshold is obtained by collecting multiple sets of image samples of judges wearing and not wearing robes, calculating the average comprehensive feature distance of the sample pairs, and then multiplying it by an adjustment coefficient k. The adjustment coefficient k ranges from (0.5, 0.9), and the calculation formula is as follows: , in, The threshold is determined by the distance of the comprehensive features. This represents the total number of sample pairs. For the first The comprehensive feature distance of a group of sample pairs.

[0050] 502: When the cumulative number of consecutive frames exceeding the comprehensive feature distance judgment threshold is greater than 10, it is comprehensively judged that the person with this ID has changed clothes.

[0051] As a refinement of the above embodiments, in response to the determination of the clothing change behavior, the following operations are performed only on the tracking ID for which the clothing change is determined: the slice of the current detection box region of the person with that ID is re-inputted into the pre-trained robe classification model to update the robe wearing status; the personal clothing baseline features bound to the ID are updated; and a structured anomaly alarm containing the clothing change information of the ID is output, the specific content of which is as follows: For the tracking ID that determines that a change of clothes has occurred, the target detection box region of the current frame of the ID is sliced ​​and input into the pre-trained ResNet50 robe classification model to obtain the current robe wearing status of the person with that ID. Compare the current robe-wearing status with the historical robe-wearing status stored for this ID to identify the type of clothing change, including two categories: "wearing robe → not wearing robe" and "not wearing robe → wearing robe". Update and store the robe-wearing status for this ID. Five consecutive frames after the ID is determined to have changed clothes are selected. The corresponding target detection box area is re-sliced ​​and its size is normalized. HSV color features and HOG texture features are extracted. The element-wise arithmetic mean of the HSV color feature vector and HOG texture feature vector is calculated respectively. The personal clothing baseline features bound to the ID are updated. During the baseline feature update, the repeated determination of the changing clothes behavior of the ID is paused until the baseline update is completed. The output includes structured anomaly alert information with the corresponding tracking ID, clothing change behavior, and clothing change type.

[0052] Example 2 This disclosure also provides a courtroom robe-changing behavior detection device based on a classification model and feature fusion, including a processor and a memory. Optionally, the device may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can call logical instructions in the memory to execute the courtroom robe-changing behavior detection method based on a classification model and feature fusion described in the above embodiments.

[0053] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0054] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes the program instructions / modules stored in the memory to perform functional applications and data processing, thereby realizing the courtroom robe-changing behavior detection method based on classification models and feature fusion in the above embodiments.

[0055] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory.

[0056] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the above-described method for detecting court robe changing behavior based on classification models and feature fusion.

[0057] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0058] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code. It can also be a transient storage medium.

[0059] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting court robe changing behavior based on classification model and feature fusion, characterized in that, Includes the following steps: Courtroom images were acquired, and the judge's position was extracted using a judge target detection model based on YOLOv8s. The judge target detection model replaced the original convolutional modules with SPD-Conv modules in the backbone and neck networks, and added a PA position attention module after the SPPF module. The detection head used a DyHead dynamic detection head. The DeepSORT multi-target tracking algorithm is used to assign tracking IDs to judges in court proceedings, and the coordinates of the detection boxes with assigned tracking IDs are subjected to EMA temporal smoothing filtering. In the initial tracking phase, the detection box region after EMA temporal smoothing filtering corresponding to each tracking ID is sliced, and the sliced ​​images are input into the pre-trained ResNet50 robe classification model to obtain the initial robe wearing state of the person with that ID. At the same time, the HSV color features and HOG texture features of each frame slice are extracted to construct the personal clothing baseline features, and the initial wearing state and personal clothing baseline features are bound and stored with the corresponding tracking ID. In subsequent video frames, the baseline features of personal clothing are extracted from the real-time target area of ​​the same ID person, and the comprehensive feature distance between the current personal clothing baseline features and the personal clothing baseline features bound to the tracking ID is calculated to form a comprehensive feature distance sequence for consecutive frames of the ID. Based on the comparison results of the comprehensive feature distance sequence and the threshold, and combined with continuous frame verification, it is determined whether the clothing change behavior has occurred. For the tracking ID that has undergone a clothing change, output an abnormal alarm containing the clothing change information for that tracking ID.

2. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 1, characterized in that, The backbone network adopts an alternating cascade structure of SPD-Conv modules and C2f modules, and all downsampling nodes are replaced with SPD-Conv modules, outputting three sets of multi-scale basic feature maps. The backbone network of the judge target detection model includes, in sequence: a first SPD-Conv module, a second SPD-Conv module, a first C2f module, a third SPD-Conv module, a second C2f module, a fourth SPD-Conv module, a third C2f module, a fifth SPD-Conv module, a fourth C2f module, an SPPF module, and a PA position attention module.

3. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 2, characterized in that, The neck network uses a bidirectional topology structure of FPN feature pyramid and PANet path aggregation to complete cross-scale feature fusion and output three sets of multi-scale fused feature maps. The neck network processes the feature map as follows: The feature map P5 output by the PA position attention module of the backbone network is upsampled and concatenated with the feature map P4 output by the third C2f module of the backbone network, and then input into the C2f module to obtain feature map F1. The feature map F1 is upsampled and concatenated with the feature map P3 output by the second C2f module of the backbone network, and then input into the C2f module to obtain feature map F2. The feature map F2 is concatenated with feature map F1 by the SPD-Conv module and then input into the C2f module to obtain feature map F3. The feature map F3 is concatenated with feature map P5 by the SPD-Conv module and then input into the C2f module to obtain feature map F4. The feature maps F2, F3, and F4 are input into the detection head.

4. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 3, characterized in that, The detection head calls the feature alignment module to unify the spatial resolution of the three sets of tensors, and sends them to the dynamic detection head module composed of DyHeadBlock cascades to perform triple dynamic calculation of scale perception, spatial perception and task perception, and outputs dynamically enhanced feature tensors, which are then split and mapped to three independent multi-scale detection heads by the feature segmentation module. The feature alignment module unifies the spatial resolution of feature maps F2, F3, and F4 through bilinear interpolation algorithm, and combines 3×3 deformable convolution to complete spatial grid resampling and channel parameter mapping compression. After unifying and aligning heterogeneous features, the module stacks them to generate an aggregated feature tensor. The dynamic detection head module is composed of DyHeadBlock1 and DyHeadBlock2 cascaded together. It performs scale-aware weight multiplication and addition operations based on adaptive average pooling matrix, spatially aware deformable convolution operations based on offset coordinate learning, and dynamic activation function switching operations based on routing weight allocation, outputting a dynamically enhanced feature tensor with unchanged dimensions. The feature segmentation module slices and unbinds the dynamically enhanced feature tensor along the scale dimension, and then maps it to three independent multi-scale detection heads with three different resolutions.

5. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 1, characterized in that, The initial tracking phase consists of five consecutive frames after the personnel are assigned a tracking ID. The initial robe-wearing state is the classification result with the highest proportion of the ResNet50 robe classification model in five consecutive frames of images; The personal clothing baseline features are based on five consecutive frames of images. The HSV color features and HOG texture features of each frame are extracted. The element-wise arithmetic mean of the HSV color feature vector and the HOG texture feature vector is calculated to obtain the baseline color features and baseline texture features as personal clothing baseline features.

6. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 1, characterized in that, The comprehensive feature distance between the current personal clothing baseline features and the personal clothing baseline features bound to this tracking ID is calculated as follows: The real-time target detection bounding box region of the same tracked person in subsequent video frames is sliced ​​and its size is normalized. HSV color features and HOG texture features consistent with the initial tracking stage are extracted. Calculate the Bartlett distance between the HSV color feature of the current tracking ID person and the baseline color feature bound to the tracking ID, and calculate the cosine distance between the HOG texture feature of the tracking ID person and the baseline texture feature bound to the tracking ID. First, the range of the Bach distance is normalized, and then it is weighted and fused with the cosine distance to obtain the comprehensive feature distance and form a distance sequence.

7. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 1, characterized in that, The specific steps for determining whether a change of clothes has occurred are as follows: A comprehensive feature distance judgment threshold is set. The threshold is obtained by collecting multiple sets of image sample pairs of judges wearing robes and not wearing robes, calculating the average comprehensive feature distance of the sample pairs, and then multiplying it by an adjustment coefficient k. When the cumulative number of consecutive frames exceeding the comprehensive feature distance judgment threshold is greater than 10, it is comprehensively determined that the person with that ID has changed clothes.

8. The method for detecting court robe changing behavior based on classification model and feature fusion according to claim 1, characterized in that, The slice of the current detection frame region of the personnel whose tracking ID has been determined to have changed clothes is re-input into the ResNet50 robe classification model to update the robe wearing status. Five consecutive frames after the clothing change are selected, the target detection box region corresponding to the tracking ID is re-sliced, and the personal clothing baseline features are extracted and updated to reflect the personal clothing baseline features of the person with the tracking ID.

9. A courtroom robe-changing behavior detection device based on classification model and feature fusion, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when running the program instructions, execute the courtroom robe-changing behavior detection method based on classification model and feature fusion as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the courtroom robe-changing behavior detection method based on classification model and feature fusion as described in any one of claims 1-8.

Images