Method for identifying specific personnel violation behavior based on image fusion re-identification technology

Abstract

The application discloses a specific person rule violation behavior recognition method based on image fusion re-recognition technology, belongs to the technical field of computer vision and artificial intelligence, and comprises the following steps: acquiring a video stream to perform quality enhancement, and generating time sequence preprocessed image frames; performing time sequence target detection and pedestrian detection on the image frames, and generating a rule violation behavior region and a plurality of pedestrian regions; screening out candidate pedestrian regions based on a double criterion of intersection over union and time persistence; cropping the candidate pedestrian images and performing identity feature extraction, and generating pedestrian re-recognition features and face features; adaptively weighting and fusing the features to generate fused identity information; and determining a rule violation behavior determination result through identity comparison and in combination with scene context information. The application adopts image quality enhancement, time sequence target detection and tracking, double-mode adaptive fusion identity recognition and scene context intention analysis technology, can locate a rule violation person, and realizes intelligent determination of a rule violation behavior with high accuracy and low false alarm rate.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method for identifying specific personnel violations based on image fusion and re-identification technology. Background Technology

[0002] Computer vision technology plays an increasingly important role in the field of security monitoring, especially in the understanding, analysis, and recognition of image and video content. Image recognition technology, particularly the detection and tracking of specific targets in video streams, and the subsequent identification of individuals and judgment of abnormal behavior, has become a hot research topic in intelligent monitoring systems. With the development of deep learning technology, image recognition and target detection algorithms have made progress in both accuracy and efficiency, and are widely used in tasks such as face recognition, person re-identification, and behavior analysis.

[0003] In existing technologies, independent modules are typically used for object detection, pedestrian tracking, and identity recognition, and behavior is judged based on preset rules. For example, object detection models such as YOLO are used to locate people or objects in images, and then ReID networks are used for cross-camera identity matching of pedestrians, or facial recognition systems are used for identity verification. In terms of behavior analysis, most systems rely on predefined behavior patterns or thresholds to mark abnormal events, such as the presence of a target in a certain area or a target lingering for an excessively long time. These methods have achieved certain results in their respective sub-fields and can complete basic recognition tasks in standard environments.

[0004] However, existing technical solutions have many shortcomings in practical applications. First, due to factors such as changes in lighting, differences in viewing angles, partial occlusion, pose diversity, and fluctuations in image quality, a single detection or recognition module often struggles to guarantee consistently high performance. For example, face recognition performance deteriorates when faces are occluded or at incorrect angles; pedestrian re-identification is prone to confusion when clothing changes or individuals have high similarity. Second, in systems where modules are used in series, an error in any one component can lead to overall performance degradation. More importantly, existing methods often make judgments about behavior that are too one-sided and mechanical, lacking a deep understanding of the scene context and the ability to analyze behavioral intent. This can easily lead to misjudging normal authorized behavior as violations or failing to identify cleverly disguised abnormal behavior, resulting in a high false alarm rate. The system's decision-making lacks intelligence and robustness, failing to meet the security monitoring needs in complex and ever-changing environments. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for identifying specific individuals' violations based on image fusion and re-identification technology. This method employs image quality enhancement, temporal target detection and tracking, dual-modal adaptive fusion identity recognition, and scene contextual intent analysis techniques to locate violators and achieve intelligent determination of violations with high accuracy and low false alarm rate.

[0006] The above objectives can be achieved through the following approach: A method for identifying specific personnel violations based on image fusion re-identification technology includes: acquiring a video stream and extracting a continuous sequence of image frames for quality enhancement processing to generate temporally preprocessed image frames; performing parallel temporal target detection and pedestrian detection on the temporally preprocessed image frames, generating violation behavior regions and multiple pedestrian regions through temporal smoothing and trajectory association processing; calculating the spatial correlation between the violation behavior regions and the multiple pedestrian regions within a continuous time window based on the dual criteria of spatial intersection-union ratio and temporal persistence, and filtering out candidate pedestrian regions that are continuously associated with the violation; cropping candidate pedestrian images based on the candidate pedestrian regions, extracting identity features from the candidate pedestrian images, and generating pedestrian re-identification features and facial features respectively; evaluating the quality parameters of the candidate pedestrian images, dynamically determining the fusion weight, and adaptively weighting and fusing the pedestrian re-identification features and facial features to generate fused identity information; comparing the fused identity information with the list of on-duty personnel, and dynamically correcting the behavioral intent by combining the scene context information of the violation behavior region, generating a violation judgment result including behavior type, severity level, and timestamp.

[0007] Optionally, generating temporally preprocessed image frames includes: extracting a sequence of consecutive image frames from the video stream, performing temporal consistency quality enhancement processing to eliminate inter-frame noise and motion blur, and generating enhanced image frames; and performing size normalization processing on the enhanced image frames to generate temporally preprocessed image frames.

[0008] Optionally, generating the violation behavior region includes: performing temporal target detection on the temporally preprocessed image frame and outputting multiple behavior region proposals with confidence scores; performing temporal smoothing processing on the multiple behavior region proposals and filtering out regions whose confidence scores are stably higher than a preset threshold within a continuous time window to generate the violation behavior region.

[0009] Optionally, generating multiple pedestrian regions includes: performing pedestrian detection processing on the temporally preprocessed image frames to obtain initial pedestrian region bounding boxes; applying a non-maximum suppression algorithm to the initial pedestrian region bounding boxes to eliminate redundancy, and linking the same pedestrian bounding boxes in different frames through a temporal association algorithm, assigning a unique trajectory identifier to each pedestrian target, and generating multiple pedestrian regions.

[0010] Optionally, the step of filtering out candidate pedestrian regions that are continuously associated with the violation includes: extracting the coordinate information of the violation region and the multiple pedestrian regions, calculating the intersection-union ratio (IUGR) of each frame within a continuous time window; calculating the cumulative duration for which the IUGR exceeds a preset overlap threshold, and filtering out candidate pedestrian regions when the cumulative duration exceeds the preset time threshold.

[0011] Optionally, generating pedestrian re-identification features includes: extracting global features representing the overall appearance and local features representing local details from the candidate pedestrian images; and performing multi-scale stitching of the global features and the local features to generate pedestrian re-identification features.

[0012] Optionally, generating facial features includes: detecting and locating facial regions in the candidate pedestrian image; extracting biometric points that can characterize identity from the facial regions to generate facial features.

[0013] Optionally, generating fused identity information includes: quantitatively evaluating the clarity, pose angle, and occlusion degree of the face region to generate quality parameters; using the quality parameters as input, determining the fusion weight of the face features through a piecewise function; and performing a weighted summation operation on the face features and the pedestrian re-identification features based on the complementary relationship between the fusion weights and the weights of the pedestrian re-identification features to generate fused identity information.

[0014] Optionally, generating the violation judgment result includes: comparing the fused identity information with the list of on-duty personnel to obtain personnel on-duty status data; performing intent analysis on the scene context information to generate behavior intent correction information; and combining the personnel on-duty status data with the behavior intent correction information to generate a violation judgment result containing behavior type, severity level, and timestamp.

[0015] Based on the same inventive concept, this invention also provides a system for identifying specific personnel violations based on image fusion re-identification technology, including a video stream preprocessing module for acquiring a video stream and extracting a continuous sequence of image frames for quality enhancement processing to generate temporal preprocessed image frames; a temporal target detection module for performing parallel temporal target detection and pedestrian detection on the temporal preprocessed image frames, generating violation behavior regions and multiple pedestrian regions through temporal smoothing and trajectory association processing; and a behavior association filtering module for calculating the spatial correlation between the violation behavior regions and the multiple pedestrian regions within a continuous time window based on the dual criteria of spatial intersection-union ratio and temporal persistence, filtering out regions that are persistent with the violation behavior. The system includes: a candidate pedestrian region; an identity feature extraction module, used to crop candidate pedestrian images based on the candidate pedestrian regions, extract identity features from the candidate pedestrian images, and generate pedestrian re-identification features and face features respectively; a multi-feature fusion module, used to evaluate the quality parameters of the candidate pedestrian images, dynamically determine the fusion weights, and adaptively weight the pedestrian re-identification features and face features to generate fused identity information; and a comprehensive decision-making and judgment module, used to compare the fused identity information with the list of on-duty personnel, and dynamically correct the behavior intent based on the scene context information of the violation area, generating a violation judgment result including behavior type, severity level, and timestamp.

[0016] Compared with the prior art, the present invention has the following advantages: This invention, by integrating image quality enhancement and temporal target tracking technologies, improves the detection accuracy and stability of areas with violations and pedestrian targets in complex surveillance scenarios, thus laying the foundation for subsequent identification. This reduces identification errors caused by factors such as video quality fluctuations or temporary target occlusion, making the initial perception stage of the entire system more reliable.

[0017] This invention innovatively employs a dual spatial and temporal correlation criterion to screen candidate pedestrians and introduces an adaptive pedestrian re-identification and facial feature fusion mechanism. This mechanism dynamically adjusts the weights of different identity modalities based on the actual quality of the facial image, thereby addressing challenges such as damaged faces, poor poses, or changes in pedestrian appearance. This strategy enhances the robustness and accuracy of identity recognition, ensuring reliable identification of specific individuals violating regulations even in changing environments.

[0018] This invention further combines the identified identity information with scene context information to perform in-depth analysis and correction of behavioral intent. This enables the system to go beyond simple visual judgment, understanding the rationality or authorization behind the behavior, effectively avoiding misjudgments. Through this intelligent judgment process, it can ultimately output refined violation judgment results including behavior type, severity level, and timestamp, providing more insightful support for security management and decision-making.

[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for identifying violations by specific individuals based on image fusion re-identification technology, according to an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram illustrating the timing smoothing effect of an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the crossover-union ratio calculation principle of an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the feature fusion weight adaptive function according to an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of a specific person violation recognition system based on image fusion re-identification technology according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Reference Figure 1 One embodiment of the present invention proposes a method for identifying violations by specific personnel based on image fusion re-identification technology. It employs image quality enhancement, temporal target detection and tracking, dual-modal adaptive fusion identity recognition, and scene context intent analysis technology to locate violators and achieve intelligent judgment of violations with high accuracy and low false alarm rate.

[0028] The method described in this embodiment specifically includes: S1. Acquire the video stream and extract the continuous image frame sequence for quality enhancement processing to generate temporal preprocessed image frames; S2. Perform parallel temporal target detection and pedestrian detection on the temporal preprocessed image frame, and generate violation behavior areas and multiple pedestrian areas through temporal smoothing and trajectory association processing respectively; S3. Based on the dual criteria of spatial intersection-union ratio and temporal persistence, calculate the spatial correlation between the violation area and the multiple pedestrian areas within a continuous time window, and filter out candidate pedestrian areas that are continuously associated with the violation. S4. Based on the candidate pedestrian region, crop out the candidate pedestrian image, perform identity feature extraction on the candidate pedestrian image, and generate pedestrian re-identification features and face features respectively; S5. Evaluate the quality parameters of the candidate pedestrian images, dynamically determine the fusion weights, and adaptively weight and fuse the pedestrian re-identification features with the facial features to generate fused identity information. S6. Compare the fused identity information with the list of on-duty personnel, and dynamically correct the behavior intent by combining the scene context information of the violation area, and generate a violation judgment result including behavior type, severity level and timestamp.

[0029] Optionally, the generation of time-series preprocessed image frames includes: Extract a continuous sequence of image frames from the video stream and perform temporal consistency quality enhancement processing to eliminate inter-frame noise and motion blur, generating enhanced image frames; The enhanced image frame is subjected to size normalization processing to generate a time-series preprocessed image frame.

[0030] Specifically, the process begins by acquiring real-time video streams from devices such as surveillance cameras. Using a video decoding program and the decoding capabilities of an open-source computer vision library, the video stream is parsed frame by frame, forming a continuous sequence of image frames arranged chronologically. To address common image quality issues in real-world surveillance environments, such as inter-frame noise caused by lighting changes and motion blur due to rapid target movement, temporal consistency quality enhancement processing is performed on the extracted continuous image frame sequence. This processing utilizes a deep learning video enhancement model, analyzing multiple adjacent frames in the temporal dimension—for example, a five-frame window consisting of the current frame and the two frames before and after it—to learn the static background information of the scene and the dynamic motion trajectory of the target. Through this multi-frame information fusion, the model can distinguish between realistic scene details and random inter-frame noise, and suppress the noise. Simultaneously, the model performs inverse convolution operations on motion blur based on motion trajectory information, thereby recovering clear object edges and textures. After this processing, each image frame is improved in quality, generating clearer and more stable enhanced image frames. Subsequently, to meet the uniform input data size requirements of subsequent deep learning models, size normalization processing is performed on all enhanced image frames. This process employs image scaling algorithms such as bilinear interpolation or bicubic interpolation to uniformly adjust enhanced image frames of different resolutions to a preset fixed size, such as 640x640 pixels, ultimately generating time-series preprocessed image frames with uniform format and optimized quality.

[0031] For example, the system accesses a real-time video stream from a surveillance camera in the computer room with a resolution of 1920x1080 pixels and a frame rate of 25 frames per second. By calling the video decoding function in the open-source computer vision library OpenCV, the video stream is parsed into a continuous sequence of image frames. To address high-frequency jitter noise that may be caused by server cooling fans and motion blur that may occur due to rapid movement of people, temporal consistency quality enhancement processing is performed on this image frame sequence. This processing employs a video enhancement model based on a recurrent convolutional neural network. This model analyzes a five-frame window consisting of the current frame and the two frames before and after it, learns the static background and motion patterns of people, suppresses random noise through multi-frame information fusion, and uses the learned motion trajectory to deblur blurred areas, generating enhanced image frames. Subsequently, size normalization processing is performed on all enhanced image frames. A bilinear interpolation algorithm is used to uniformly adjust the size of each frame to 640x640 pixels, generating temporally preprocessed image frames. This method achieves effective preprocessing of the original video stream by introducing temporal consistency quality enhancement processing and size normalization. This preprocessing not only improves the clarity of a single frame image, but more importantly, it eliminates noise and motion blur by utilizing inter-frame information, ensuring the stability and consistency of the entire image sequence in the time dimension.

[0032] Optionally, the area where the violation is generated includes: Temporal target detection is performed on the temporally preprocessed image frames, and multiple behavioral region proposals with confidence scores are output. The multiple proposed behavior regions are subjected to temporal smoothing processing to filter out regions whose built-in confidence scores are consistently higher than a preset threshold within a continuous time window, thereby generating violation behavior regions.

[0033] Specifically, the process begins by using pre-processed temporal image frames as input and performing temporal object detection on each frame using a pre-trained YOLOv8 model. YOLOv8 is a highly efficient single-stage object detector. The model used in this method is fine-tuned for specific violation categories, such as using a mobile phone, loitering, and smoking, based on its standard architecture. When the model processes a pre-processed temporal image frame, it outputs a series of rectangular bounding boxes that may contain violations; these bounding boxes are called behavior region proposals. Each behavior region proposal is accompanied by a class label and a confidence score. The confidence score is a floating-point number between 0 and 1, quantifying the model's certainty that a specific violation exists within that region. Subsequently, to eliminate false positives or false negatives that may occur in single-frame detection, temporal smoothing is performed on multiple behavior region proposals output from consecutive frames. This smoothing is performed within a pre-defined continuous time window containing N consecutive images. For a fixed spatial location in the scene, the confidence score sequence of those detected as behavior region proposals within the time window is tracked. The stability of the behavior is determined by calculating the smoothness confidence score of the sequence. The smoothness confidence score can be calculated using the following formula: , in, Representative at the The smoothness confidence of the region within a frame, where N is the size of the time window. This is the confidence score corresponding to the behavioral region proposal with the highest spatial overlap with the current analysis region in the i-th frame image within the time window. Those with smooth confidence scores are then selected. The region whose confidence level remains consistently above a preset threshold throughout the entire continuous time window is considered a valid violation region. Only regions that simultaneously meet both the conditions of spatial location stability and a confidence level consistently above the threshold will be ultimately identified as valid violation regions, and their spatial coordinate information will be output. Figure 2 As shown in the figure, the timing curve visually demonstrates the effect of timing smoothing.

[0034] For example, preprocessed image frames are fed frame by frame into a finely tuned YOLOv8 model. This model has been specifically trained on a dataset containing behaviors such as "playing on a mobile phone". At frame t, the model detects a suspected "playing on a mobile phone" behavior and outputs a proposed behavior region with bounding box coordinates (200, 250, 280, 350) and a confidence score of 0.95. To confirm that the behavior is persistent rather than a momentary misjudgment, temporal smoothing is performed. This process sets a time window of size N=10 frames. The system tracks detections in the region near (200, 250, 280, 350) and calculates the smoothing confidence. Within the 10-frame window from frame t-9 to frame t, the confidence score sequence matching this region is {0.88, 0.91, 0.85, 0.79, 0.93, 0.95, 0.89, 0.92, 0.94, 0.95}. Then, at frame t, the calculated smoothing confidence level The system's preset confidence threshold is 0.85. Since the calculated smoothed confidence score of 0.901 consistently exceeds this threshold, the region is confirmed as a valid violation area. This method uses the YOLOv8 model for initial detection and improves the accuracy and reliability of violation area detection by introducing a temporal smoothing mechanism. Temporal smoothing filters out transient and unstable detection results caused by sudden changes in lighting, brief occlusions, or accidental model misjudgments, avoiding misclassification of occasional, non-persistent behaviors as violations. By requiring the behavior's confidence score to remain consistently high over a period of time, this method ensures that the final output violation area has high temporal consistency and spatial stability, providing a foundation for subsequently associating the behavior with specific individuals, thereby reducing the false alarm rate of the entire recognition system.

[0035] Optionally, generating multiple pedestrian regions includes: Pedestrian detection processing is performed on the temporally preprocessed image frames to obtain initial pedestrian region bounding boxes; The initial pedestrian region bounding box is subjected to redundancy elimination by applying a nonmaximum suppression algorithm, and the same pedestrian bounding box in different frames is linked by a temporal association algorithm. A unique trajectory identifier is assigned to each pedestrian target to generate multiple pedestrian regions.

[0036] Specifically, a pre-trained deep learning pedestrian detection model is first used to process the input temporal preprocessed image frames one by one. The model scans each frame, identifies all potential pedestrian targets, and outputs an initial pedestrian region bounding box for each target. These initial bounding boxes not only contain the pedestrian's position and size information but also include a confidence score, representing the model's degree of confidence that the detection result is a pedestrian. Since a single pedestrian may be detected simultaneously by multiple overlapping bounding boxes within a single frame, a non-maximum suppression (NMS) algorithm is applied to these initial pedestrian region bounding boxes to eliminate redundancy. NMS is a post-processing technique that first sorts all bounding boxes according to their confidence scores, retains the bounding box with the highest score, and then calculates the Intersection over Union (IOU) of that bounding box with all other bounding boxes. The algorithm suppresses or deletes redundant bounding boxes whose IOU with the highest-scoring box exceeds a preset threshold. This process is iterated until all bounding boxes have been processed, ensuring that each pedestrian in each frame is represented by only one optimal bounding box. After obtaining the pedestrian bounding boxes for each frame, a temporal association algorithm is used to link the same pedestrian bounding boxes in different frames to construct the pedestrian's trajectory. This algorithm assigns a unique trajectory identifier to each newly appearing pedestrian target. When processing subsequent frames, the algorithm comprehensively considers motion and appearance information, matching the detection boxes in the new frame with existing trajectories. Motion information is typically estimated using prediction models such as Kalman filters to estimate the pedestrian's possible position in the current frame; appearance information is characterized by depth feature vectors extracted from the bounding boxes, such as color histograms or features extracted by a re-identification network. The algorithm calculates the association cost between each new detection box and all existing trajectory prediction positions, which is a comprehensive measure of spatial distance and appearance similarity. Then, optimization methods such as the Hungarian algorithm are used to find the assignment scheme that minimizes the matching cost. Successfully matched detection boxes are assigned to their corresponding trajectories, while unmatched detection boxes are treated as newly appearing targets, and new trajectory identifiers are created for them. Through this process, a continuous temporal trajectory is established for each pedestrian target in the scene, ultimately generating multiple pedestrian regions with unique trajectory identifiers. These regions dynamically describe the complete motion process of each pedestrian in the video sequence.

[0037] For example, the system feeds each temporally preprocessed image frame into a pre-trained deep learning pedestrian detection model, such as a YOLOv5 model specifically designed for pedestrian detection. When processing a particular frame, the model may output multiple overlapping initial pedestrian region bounding boxes around the same pedestrian, for example, outputting three bounding boxes with confidence scores of 0.95, 0.88, and 0.82 for a single pedestrian. To eliminate redundancy, the system immediately applies a non-maximum suppression algorithm. This algorithm first selects the bounding box with the highest confidence score of 0.95, and then calculates its Intersection over Union (IOU) with the other two bounding boxes. The calculated IOU values ​​are 0.8 and 0.7, both exceeding the preset IOU threshold of 0.5. Therefore, the two bounding boxes with confidence scores of 0.88 and 0.82 are suppressed and deleted, retaining only the optimal bounding box with a confidence score of 0.95. This process is performed iteratively in each frame to ensure that each pedestrian is represented by a unique bounding box. After obtaining the accurate pedestrian bounding boxes for each frame, the system constructs the pedestrian's motion trajectory using a temporal correlation algorithm. For example, if a new pedestrian is detected in frame t, the system assigns it a unique track identifier, such as "TrackID_001", and records its location and appearance feature vector extracted by the re-identification network. When processing frame t+1, the system uses a Kalman filter to predict the possible location of "TrackID_001" in the current frame. Simultaneously, for the bounding box of a newly detected pedestrian in frame t+1, the system calculates its spatial distance to the predicted location of "TrackID_001" and calculates the cosine similarity between its appearance feature vector and the existing feature vector of "TrackID_001". By weighting and combining spatial distance and appearance similarity into a single association cost, the system uses the Hungarian algorithm to find the globally optimal match with the minimum cost for all existing tracks and new detection boxes. If a new detection box successfully matches "TrackID_001", the box is assigned to this track; if it fails to match, it is considered a newly appearing pedestrian, and a new track identifier, such as "TrackID_002", is created for it. This process generates multiple pedestrian regions with unique trajectory identifiers, dynamically describing the complete movement of each pedestrian in the video. This process is continuous, fully recording the individual's movement trajectory. This method, through the organic combination of pedestrian detection, non-maximum suppression, and temporal correlation, achieves a leap from static single-frame detection to dynamic multi-frame tracking. It not only locates pedestrians in each frame, but more importantly, it assigns each pedestrian a stable, time-spanning identifier, providing a data foundation for subsequent calculations of the continuous correlation between specific individuals and violation areas. This ensures the continuity and accuracy of behavioral responsibility determination, improving the robustness and intelligence of the entire recognition system.

[0038] Optionally, the process of filtering out candidate pedestrian areas that are persistently associated with violations includes: Extract the coordinate information of the violation area and the multiple pedestrian areas, and calculate the intersection-union ratio of each frame within a continuous time window; The cumulative duration for which the intersection-union ratio exceeds a preset overlap threshold is counted. When the cumulative duration exceeds a preset time threshold, candidate pedestrian areas are selected.

[0039] Specifically, the process first addresses individual violation areas and multiple pedestrian areas with unique trajectory identifiers. Within a pre-defined continuous time window, the spatial relationship between each pedestrian area and the violation area is analyzed frame by frame. For any frame within the time window, the bounding box of the violation area is extracted. bounding box of a specific pedestrian area The coordinate information is typically represented as the pixel coordinates of the top-left and bottom-right corners of the bounding box. Based on this coordinate information, the intersection-over-union ratio (IoU) is calculated. IoU is a standard metric that measures the degree of overlap between two bounding boxes. The calculation method is as follows: , in, This represents the intersection-union ratio, and its range is between 0 and 1. It is the area of ​​the intersection of the boundary boxes of the violation area and the pedestrian area; This is the total area of ​​the merged region. Both area values ​​are calculated using coordinate information. For example... Figure 3 As shown, this spatial layout diagram illustrates the principle of using the intersection-union ratio (IUU) to determine spatial correlation. The calculation for each frame... The value is compared to a preset overlap threshold. If the current frame's overlap is... If the value exceeds the threshold, it is considered that at that instant, the pedestrian and the violation are strongly correlated spatially. Next, this strong correlation state of each pedestrian trajectory is accumulated over the entire continuous time window. A cumulative duration counter is maintained for each pedestrian trajectory. When the intersection-over-union ratio (IoU) of a frame exceeds a preset overlap threshold, the time length represented by that frame, i.e., the reciprocal of the video frame rate, is added to the corresponding pedestrian's counter. After traversing the entire time window, it is checked whether the cumulative duration of each pedestrian exceeds a preset time threshold. Only pedestrian regions whose cumulative duration exceeds the preset time threshold are ultimately selected and considered as candidate pedestrian regions continuously correlated with the violation.

[0040] For example, the system detects a confirmed area of ​​"using a mobile phone" violation, which lasts from time point t1 to t2, for a total duration of 5 seconds. During this period, there are two tracked pedestrian areas in the scene, designated "TrackID_001" and "TrackID_002". The system analyzes the spatial relationship between these two pedestrian areas and the violation area frame by frame within a continuous time window covering this duration. Assuming a video frame rate of 25 frames per second, this time window contains 125 frames. For each frame within the window, the system calculates the bounding box of each pedestrian area. Boundary box of the area of ​​violation Intersection over Union (IoU) between them. For example, in a specific frame, the violation area... The bounding box of the pedestrian area “TrackID_001” The area of ​​the intersecting region is 15,000 pixels, and the total area of ​​the merged region is 21,600 pixels. The intersection-union ratio (IUU) can be calculated using the formula: The system's preset overlap threshold is 0.6. Since the calculated IoU value of 0.694 exceeds this threshold, "TrackID_001" is considered strongly associated with the violation in this frame. The system performs this calculation on all 125 frames within the time window. After the traversal, the system found that there were 100 frames where the IoU value between the bounding box of "TrackID_001" and the violation area exceeded 0.6. "TrackID_002" simply passed through, with only 10 frames showing an IoU value exceeding 0.6. Next, the system calculates the cumulative association duration between each pedestrian and the violation. The duration of each frame is 1 / 25 = 0.04 seconds. Therefore, the cumulative duration of "TrackID_001" is... Seconds. The cumulative duration of “TrackID_002” is The system's preset time threshold is 3 seconds. Since the cumulative duration of "TrackID_001" was 4.0 seconds, exceeding the 3-second threshold, this pedestrian area was successfully filtered as a candidate pedestrian area continuously associated with the violation. "TrackID_002," however, had a cumulative duration of 0.4 seconds, which did not meet the threshold and was therefore excluded. This method improves the accuracy of associating behavior with individuals by establishing dual spatial and temporal judgment thresholds. It not only requires spatial overlap between the person and the violation area but also requires that this overlap last for a certain period. This mechanism can filter out pedestrians who accidentally pass through the violation area, briefly interact, or are falsely detected, avoiding involving irrelevant individuals in the determination of violations. By identifying individuals with persistent gaze patterns at the location of the violation, this method provides highly relevant, high-quality target samples for subsequent identity recognition, reducing the system's false positive rate and improving the reliability and robustness of the entire violation recognition process.

[0041] Optionally, the generated pedestrian re-identification features include: Global features representing the overall appearance and local features representing local details are extracted from the candidate pedestrian images, respectively. The global features and the local features are spliced ​​together at multiple scales to generate pedestrian re-identification features.

[0042] Specifically, this process employs a specially trained deep convolutional neural network (CNN) specifically designed for pedestrian re-identification tasks. Its core capability lies in learning deep appearance representations insensitive to changes in lighting, pose, and viewpoint. First, to extract global features representing the overall appearance, complete candidate pedestrian images are input into this deep CNN. The network abstracts image information layer by layer through a series of convolutional layers, activation functions, and pooling layers. Finally, at the end of the network backbone, a global average pooling layer compresses the entire high-dimensional feature map into a fixed-length feature vector. This vector encapsulates global information of the image, such as the person's body shape, the main color tone of their clothing, and the overall style, constituting the global features. In parallel, to extract local features representing local details, the network does not directly perform global pooling after generating the final feature map. Instead, it employs a component-level feature extraction strategy, uniformly dividing the feature map vertically into multiple horizontal strips, for example, dividing it from top to bottom into six parts: head and shoulders, upper body, lower body, legs, and feet. Subsequently, pooling operations are performed on the features within each horizontal strip region to generate an independent local feature vector for each body part. These local features capture finer details, such as specific patterns on hats, backpacks, trousers, or shoe styles. Finally, a multi-scale concatenation operation is performed, linking the extracted single global feature vector with multiple local feature vectors in a dimensional manner to form a higher-dimensional, more information-rich combined feature vector. This combined vector is the final pedestrian re-identification feature, containing both a holistic description of the pedestrian and detailed information about each key body part.

[0043] For example, the system extracts the corresponding candidate pedestrian images from the selected candidate pedestrian regions. These images are fed into a deep convolutional neural network specifically designed for pedestrian re-identification tasks, such as a model based on the ResNet50 architecture and employing a component-level feature extraction strategy. First, to extract global features representing the overall appearance, the image undergoes forward propagation through the network backbone, resulting in a high-dimensional feature map after the final convolutional layer. Global average pooling is performed on this feature map, compressing it into a fixed-length 2048-dimensional feature vector. This vector encapsulates overall information such as the person's body shape and main clothing color, constituting the global features. In parallel, to extract local features representing local details, the system uniformly divides the same high-dimensional feature map vertically into six horizontal strips, corresponding to the head and shoulders, upper body, waist, thighs, calves, and feet of the human body, respectively. Subsequently, local average pooling is performed on the features within each horizontal strip region, generating an independent 2048-dimensional local feature vector for each body component. Finally, the system performs a multi-scale concatenation operation, concatenating the extracted global feature vector with the six local feature vectors in a specific dimension to form a concatenated vector. A combined feature vector is generated. This vector contains both a holistic description of the pedestrian and detailed information about key body parts, ultimately serving as the pedestrian re-identification feature for the candidate pedestrian. This method constructs a multi-scale pedestrian appearance descriptor by combining global and local features, thereby enhancing the discriminative power and robustness of pedestrian re-identification features. Global features provide an overall impression of the pedestrian, enabling the broad differentiation of individuals with significant appearance differences. Local features compensate for the shortcomings of global features, relying on unobstructed or most discriminative local details for identification in complex situations such as partial occlusion, pose changes, or different people wearing similar clothing.

[0044] Optionally, the generated facial features include: Detect and locate face regions in the candidate pedestrian images; Biometric feature points that can characterize identity are extracted from the face region to generate facial features.

[0045] Specifically, a face detection module is first activated within the selected candidate pedestrian images. This module employs a high-performance face detection algorithm, such as the deep learning-based RetinaFace model, to scan the candidate pedestrian images and detect and locate face regions. This algorithm can effectively handle different lighting conditions, poses, and partial occlusion. Its output is the bounding box coordinates of the face region, thus narrowing the focus of analysis from the entire pedestrian image to the face with the highest identification value.

[0046] After successfully locating the face region, it undergoes preprocessing, including face alignment based on detected key points such as eyes, nose, and mouth, correcting tilted faces to a standard pose, and cropping and scaling it to a fixed size to meet the input format requirements of subsequent feature extraction models. This standardized face image is then fed into a pre-trained deep face recognition network, such as a model trained using ArcFace or CosFace loss functions. This network uses complex convolutions and nonlinear transformations to map the two-dimensional face image information into a high-dimensional feature space, outputting a fixed-length floating-point vector. This vector is a highly condensed representation of biometric points that characterize identity, i.e., the final generated face feature. The feature vector is designed so that faces with the same identity are as close as possible in the feature space, while faces with different identities are as far apart as possible, thus possessing strong identity discrimination capabilities.

[0047] For example, within the candidate pedestrian image extracted from the candidate pedestrian region, the system initiates a face detection module. This module employs the high-performance RetinaFace face detection algorithm to scan the image, successfully detecting and locating a face region and outputting its bounding box coordinates. After successfully locating the face region, the system uses the coordinates of five facial key points simultaneously output by RetinaFace—the centers of the eyes, the tip of the nose, and the sides of the mouth—to align the face. An affine transformation is used to correct the tilted face to a standard pose, ensuring the eyes are horizontal. Subsequently, the aligned face region is cropped and uniformly scaled to 112x112 pixels. This standardized face image is fed into a pre-trained deep face recognition network trained using the ArcFace loss function. This network maps the 112x112 face image to a high-dimensional feature space and outputs a fixed-length 512-dimensional floating-point vector. This vector is a highly condensed representation of the biometric points that characterize the person's identity, ultimately serving as the face feature of the candidate pedestrian. This method introduces an independent facial feature extraction process, adding a biometric dimension to personnel identification. Compared to pedestrian re-identification features that rely solely on clothing and body posture, facial features possess inherent uniqueness and stability, making them less susceptible to external factors such as clothing changes. By extracting high-quality facial features from candidate pedestrian images, this method enhances the accuracy and certainty of identification, providing a core basis for subsequent high-precision comparison with on-duty personnel lists, and improving the overall system's performance in identifying specific violators in complex scenarios.

[0048] Optionally, the generation of fused identity information includes: The clarity, pose angle, and occlusion degree of the face region are quantitatively evaluated to generate quality parameters; The quality parameters are used as input, and the fusion weights of the facial features are determined through a piecewise function. Based on the complementary relationship between the fusion weights and the pedestrian re-identification features, a weighted summation operation is performed on the face features and the pedestrian re-identification features to generate fused identity information.

[0049] Specifically, the process first performs a multi-dimensional quantitative evaluation of the extracted face region to generate a comprehensive quality parameter. Specifically, it evaluates three key aspects. First, sharpness: using image gradient-based algorithms, such as the Laplacian operator, the sum of gradient magnitudes of the face region image is calculated. Higher gradient values ​​generally indicate a sharper image and sharper edges. Second, pose angles: using a facial landmark detection model, the yaw, pitch, and roll angles of the head are estimated, and the deviation of these angles from the standard frontal pose is calculated. Third, occlusion: a face parsing model identifies the visible areas of key facial components such as the eyes, nose, and mouth, and calculates their proportion of the entire face area. The evaluation results of these three dimensions are normalized and integrated into a single, quantitative quality parameter. Next, this quality parameter is used as input, and a predefined piecewise function is used to dynamically determine the fusion weights of facial features. This piecewise function is designed to reflect the non-linear relationship between quality and reliability. For example, when the quality parameter is below a lower threshold, it indicates extremely poor face quality, and its fusion weight is set to 0; when the quality parameter is above an upper threshold, it indicates extremely good face quality, and its fusion weight is assigned a higher preset value; between the two thresholds, the fusion weight increases linearly or non-linearly with the increase of the quality parameter. This calculated weight is the fusion weight of the face features. Based on the complementary relationship between this fusion weight and the weight of the person re-identification features, i.e., the sum of their weights is always 1, the weight of the person re-identification features is then determined. Finally, a weighted summation operation is performed on these two feature vectors to generate the final fused identity information. Before performing this operation, it is ensured that, through network structure design or linear projection, the face feature vector and the person re-identification feature vector are mapped to a feature space of the same dimension and scale. The fusion process can be represented by the following formula: , in, This represents the final generated fused identity information vector. The fusion weights of facial features are calculated based on quality parameters using a piecewise function. It is a pedestrian re-identification feature vector. It is a facial feature vector. For example... Figure 4 As shown, the function curve depicts the adaptive determination mechanism of the face feature fusion weights. The horizontal axis represents the comprehensive quality parameters of the face region, and the vertical axis represents the fusion weights assigned to the face features.

[0050] For example, the system first performs a multi-dimensional quantitative evaluation of the extracted face region. Clarity evaluation uses the Laplacian operator to calculate the image variance; a higher score indicates greater clarity, and this evaluation score is 80. Pose angle evaluation uses facial landmarks to estimate the head yaw angle at 25 degrees and the pitch angle at 10 degrees, which deviates significantly from the overall frontal pose, resulting in an evaluation score of 60. Occlusion evaluation uses the face analysis model to find that the mouth is partially obscured by the hand, with an obscuration area of ​​approximately 15%, resulting in an evaluation score of 85. The system then weights and averages the scores from these three dimensions to generate a comprehensive quality parameter. After normalization, the value is 0.73. Next, this quality parameter of 0.73 is used as input to determine the fusion weights of facial features through a predetermined piecewise function. The function is defined as follows: when the mass parameter is below 0.4... When it is 0, or higher than 0.9 It has a maximum value of 0.8, and grows linearly between 0.4 and 0.9. According to this rule, The weights of pedestrian re-identification features are complementary. Before performing the weighted summation, the system uses a linear projection layer to map the 14336-dimensional person re-identification features into the same feature space as the 512-dimensional face features, resulting in a 512-dimensional vector. Assuming that after projection... The first dimension has a value of -0.25. The first dimension value is 0.8. Therefore, according to the fusion formula, the final generated fused identity information... The calculation process for the first dimension value is as follows: This calculation process will be applied to all 512 dimensions of the two vectors, ultimately generating a brand new 512-dimensional fused identity information vector. This method achieves intelligent fusion of identity features by introducing face quality assessment and an adaptive weighting mechanism. It dynamically adjusts the trust level of different feature sources based on real-time image quality. When the face is clearly visible, it relies more on highly accurate facial biometric features; conversely, when the face quality is poor, such as blurry, occluded, or a profile view, its weight is automatically reduced, and the system relies more on relatively stable overall appearance features. This dynamic adjustment strategy enhances the robustness and accuracy of the identity recognition system under various complex and non-ideal monitoring conditions, ensuring that the system can always use the most reliable information source for identity determination regardless of the target's state.

[0051] Optionally, the generation of violation determination results includes: The fused identity information is compared with the list of on-duty personnel to obtain personnel on-duty status data; The scene context information is analyzed to generate behavioral intent correction information; The data on personnel on duty status and the information on behavioral intent correction are processed to generate a result that includes the type of behavior, severity level, and timestamp of the violation.

[0052] Specifically, the process first compares the fused identity information, i.e., high-dimensional feature vectors, with a pre-built database of on-duty personnel. This database stores the identity feature vectors of all authorized personnel and their real-time shift information. The comparison process finds the best match by calculating the similarity score, such as cosine similarity, between the input fused identity information and the feature vectors of each registered person in the database. Once the similarity score exceeds a preset threshold, the person's identity can be identified, and their current on-duty status can be retrieved from the database, thus obtaining the personnel's on-duty status data. Simultaneously, a deep analysis of the scene context information of the area where the violation occurred is performed. This scene context information includes not only the initial category of the violation but also the time of the violation, key equipment or items in the area, and the attributes of the area. Using an intent analysis model or a set of preset business logic rules, this information is comprehensively evaluated to generate behavior intent correction information. For example, a "lingering" behavior, if it occurs late at night in a non-working area, might be judged as suspicious; however, if it occurs during working hours, and the person has a toolbox and equipment to be repaired nearby, the intent of the behavior is likely to be corrected to "normal maintenance preparation." Finally, the acquired personnel duty status data and the generated behavioral intent correction information are subjected to final logical processing to output a structured violation judgment result. This processing is a decision engine that makes judgments based on a predefined set of rules. For example, a rule can be defined as follows: if the identified personnel is not on the duty roster and their behavioral intent has not been corrected, the event is judged as a "unauthorized entry" violation and set to a "high" severity level. Conversely, if the identified personnel is an on-duty engineer and their behavioral intent has been corrected to "normal maintenance," no violation alert will be generated, or only a "normal operation" record will be generated. The final output is a data packet containing a clear behavior type, a quantified severity level, and an event timestamp.

[0053] For example, the system compares the generated 512-dimensional fused identity information with a pre-built database of on-duty personnel lists. This database stores the identity feature vectors of all authorized personnel and their daily shift schedules. The comparison process calculates the cosine similarity between the input fused identity information and the feature vector of each registered person in the database. The calculation results show that the fused identity information has the highest similarity to the feature vector of "Engineer Li Si," at 0.91, far exceeding the preset matching threshold of 0.8. Therefore, the system identifies the person as Li Si and retrieves his shift schedule from the database, obtaining the "personnel on-duty status data" as "currently on duty." Simultaneously, the system performs intent analysis on the contextual information of the scenario that caused the violation. The initial detected violation is "detention," occurring at 14:30 in the afternoon at "rack 3." Further analysis reveals that the area is classified as "core equipment area," and an object consisting of a "ladder" and a "toolbox" was detected next to "Li Si." The intent analysis model comprehensively judges the behavior intent based on preset rules (behavior = loitering) + (location = core equipment area) + (time = working hours) + (associated items = tools / ladders) + (personnel = on-duty engineer)), generating the behavior intent correction information as "suspected normal equipment maintenance". Finally, the decision engine processes the data by combining the personnel's on-duty status data with the behavior intent correction information. Since the identified personnel "Li Si" is the on-duty engineer, and their behavior intent is highly corrected to "normal maintenance", the system determines that this event does not constitute a violation. The final violation judgment result is a structured log record with the content {behavior type "normal operation", severity level "information", timestamp "2024-10-27 14:30:00", associated personnel "engineer Li Si", and remarks "using ladder and toolbox for work in rack 3"}, rather than a violation alarm. This method, by combining identity recognition results with dynamic scene context analysis, achieves a leap from simple behavior detection to intelligent behavior intent understanding. It can distinguish between genuine violations and authorized normal work behavior, reducing the system's false alarm rate. By comprehensively assessing violations from multiple dimensions and outputting structured results including severity levels, this method not only improves the accuracy of the assessment but also provides security managers with richer and more valuable information for decision-making. This enables security responses to be prioritized according to the urgency of the incident, thereby improving overall management efficiency and intelligence.

[0054] Based on the same inventive concept, such as Figure 5 As shown, the present invention also provides a system for identifying specific personnel violations based on image fusion and re-identification technology, the system comprising: The video stream preprocessing module is used to acquire the video stream and extract a continuous sequence of image frames for quality enhancement processing, generating time-series preprocessed image frames; The temporal target detection module is used to perform parallel temporal target detection and pedestrian detection on the temporal preprocessed image frames. Through temporal smoothing and trajectory association processing, it generates violation behavior regions and multiple pedestrian regions respectively. The behavior association filtering module is used to calculate the spatial correlation between the violation behavior area and the multiple pedestrian areas within a continuous time window based on the dual criteria of spatial intersection-union ratio and temporal persistence, and to filter out candidate pedestrian areas that are continuously associated with the violation behavior. The identity feature extraction module is used to crop out candidate pedestrian images based on the candidate pedestrian region, perform identity feature extraction on the candidate pedestrian images, and generate pedestrian re-identification features and face features respectively; The multi-feature fusion module is used to evaluate the quality parameters of the candidate pedestrian images, dynamically determine the fusion weights, and adaptively weight the pedestrian re-identification features and facial features to generate fused identity information. The integrated decision-making and judgment module is used to compare the fused identity information with the list of on-duty personnel, and dynamically correct the behavior intent by combining the scene context information of the violation area, and generate a violation judgment result including behavior type, severity level and timestamp.

[0055] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0056] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A specific person violation behavior recognition method based on image fusion re-identification technology, characterized in that, The method includes: The video stream is acquired and a continuous sequence of image frames is extracted for quality enhancement processing to generate temporally preprocessed image frames. Parallel temporal target detection and pedestrian detection are performed on the temporal preprocessed image frames. Through temporal smoothing and trajectory association processing, violation behavior regions and multiple pedestrian regions are generated respectively. Based on the dual criteria of spatial intersection-union ratio and temporal persistence, the spatial correlation between the violation area and the multiple pedestrian areas within a continuous time window is calculated, and candidate pedestrian areas that are continuously associated with the violation are selected. Candidate pedestrian images are cropped from the candidate pedestrian regions, and identity feature extraction is performed on the candidate pedestrian images to generate pedestrian re-identification features and face features respectively. The quality parameters of the candidate pedestrian images are evaluated, the fusion weights are dynamically determined, and the pedestrian re-identification features and facial features are adaptively weighted and fused to generate fused identity information. The fused identity information is compared with the list of on-duty personnel, and the behavioral intent is dynamically corrected by combining the scene context information of the area of ​​violation, generating a violation judgment result that includes behavior type, severity level and timestamp.

2. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 1, characterized in that, The generation of time-series preprocessed image frames includes: Extract a continuous sequence of image frames from the video stream and perform temporal consistency quality enhancement processing to eliminate inter-frame noise and motion blur, generating enhanced image frames; The enhanced image frame is subjected to size normalization processing to generate a time-series preprocessed image frame.

3. The method for identifying specific personnel violations based on image fusion and re-identification technology according to claim 1, characterized in that, The areas where violations are generated include: Temporal target detection is performed on the temporally preprocessed image frames, and multiple behavioral region proposals with confidence scores are output. The multiple proposed behavior regions are subjected to temporal smoothing processing to filter out regions whose built-in confidence scores are consistently higher than a preset threshold within a continuous time window, thereby generating violation behavior regions.

4. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 1, characterized in that, The generation of multiple pedestrian regions includes: Pedestrian detection processing is performed on the temporally preprocessed image frames to obtain initial pedestrian region bounding boxes; The initial pedestrian region bounding box is subjected to redundancy elimination by applying a nonmaximum suppression algorithm, and the same pedestrian bounding box in different frames is linked by a temporal association algorithm. A unique trajectory identifier is assigned to each pedestrian target to generate multiple pedestrian regions.

5. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 1, characterized in that, The candidate pedestrian areas that are continuously associated with violations include: Extract the coordinate information of the violation area and the multiple pedestrian areas, and calculate the intersection-union ratio of each frame within a continuous time window; The cumulative duration for which the intersection-union ratio exceeds a preset overlap threshold is counted. When the cumulative duration exceeds a preset time threshold, candidate pedestrian areas are selected.

6. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 1, characterized in that, The generated pedestrian re-identification features include: Global features representing the overall appearance and local features representing local details are extracted from the candidate pedestrian images, respectively. The global features and the local features are spliced ​​together at multiple scales to generate pedestrian re-identification features.

7. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 1, characterized in that, The generated facial features include: Detect and locate face regions in the candidate pedestrian images; Biometric feature points that can characterize identity are extracted from the face region to generate facial features.

8. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 7, characterized in that, The generated fused identity information includes: The clarity, pose angle, and occlusion degree of the face region are quantitatively evaluated to generate quality parameters; The quality parameters are used as input, and the fusion weights of the facial features are determined through a piecewise function. Based on the complementary relationship between the fusion weights and the pedestrian re-identification features, a weighted summation operation is performed on the face features and the pedestrian re-identification features to generate fused identity information.

9. The method for identifying specific personnel violations based on image fusion re-identification technology according to claim 1, characterized in that, The generated violation determination results include: The fused identity information is compared with the list of on-duty personnel to obtain personnel on-duty status data; The scene context information is analyzed to generate behavioral intent correction information; The data on personnel on duty status and the information on behavioral intent correction are processed to generate a result that includes the type of behavior, severity level, and timestamp of the violation.

10. A system for identifying violations by specific individuals based on image fusion and re-identification technology, characterized in that: The system includes: The video stream preprocessing module is used to acquire the video stream and extract a continuous sequence of image frames for quality enhancement processing, generating time-series preprocessed image frames; The temporal target detection module is used to perform parallel temporal target detection and pedestrian detection on the temporal preprocessed image frames. Through temporal smoothing and trajectory association processing, it generates violation behavior regions and multiple pedestrian regions respectively. The behavior association filtering module is used to calculate the spatial correlation between the violation behavior area and the multiple pedestrian areas within a continuous time window based on the dual criteria of spatial intersection-union ratio and temporal persistence, and to filter out candidate pedestrian areas that are continuously associated with the violation behavior. The identity feature extraction module is used to crop out candidate pedestrian images based on the candidate pedestrian region, perform identity feature extraction on the candidate pedestrian images, and generate pedestrian re-identification features and face features respectively; The multi-feature fusion module is used to evaluate the quality parameters of the candidate pedestrian images, dynamically determine the fusion weights, and adaptively weight the pedestrian re-identification features and facial features to generate fused identity information. The integrated decision-making and judgment module is used to compare the fused identity information with the list of on-duty personnel, and dynamically correct the behavior intent by combining the scene context information of the violation area, and generate a violation judgment result including behavior type, severity level and timestamp.

Images