Method and device for identifying personnel rule violations

Abstract

The application discloses a kind of personnel rule violation identification method and device, the method includes: using first visual model to the video stream data of monitoring area is detected, obtains detection result, detection result includes rule violation behavior class and corresponding detection confidence;According to the inherent dangerous attribute of monitoring area, detection result and abnormal behavior time sequence cumulative information carry out multidimensional risk assessment, obtain comprehensive risk score;Based on detection confidence, comprehensive risk score and time sequence continuity information determine whether to call second visual model, the calculation complexity of second visual model is higher than first visual model;When determining calling, using second visual model to the region corresponding to detection result carries out semantic reasoning, obtains semantic score;Detection confidence, semantic score and comprehensive risk score are fused, and rule violation behavior identification result is output according to fusion decision value.Compared with prior art, the application improves rule violation identification accuracy and power efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for identifying personnel violations. Background Technology

[0002] In existing industrial production scenarios, human violations are a major cause of safety accidents, and automated identification technology based on computer vision has become an important development direction for safety monitoring. Current mainstream solutions mostly use a single deep learning model for end-to-end detection of surveillance videos. However, the parameter capacity of a single model is limited. While lightweight models can meet the computational requirements of parallel inference of multiple video streams, their accuracy in identifying subtle violations is low. Larger models have higher accuracy but excessive computational overhead, making it difficult to support large-scale parallel deployment.

[0003] To alleviate computational pressure, existing technologies have introduced a model cascade scheduling architecture, using lightweight models as initial screening and triggering large models for verification only when the confidence level falls below a fixed threshold. However, the triggering condition of this approach relies on a single confidence threshold, ignoring multi-dimensional decision-making factors such as scene hazard level, degree of behavioral abnormality, and temporal continuity. This can easily lead to missed triggering of large models in high-risk scenarios or frequent invalid triggering in low-risk scenarios. Furthermore, existing solutions employ an equal computational power allocation strategy for all video frames, processing them with the same sensitivity regardless of whether personnel are in high-risk areas or ordinary channels. This results in insufficient computational power in high-risk scenarios and wasted resources in low-risk scenarios. Therefore, it is evident that existing technologies struggle to achieve a good balance between computational efficiency and recognition accuracy.

[0004] Therefore, there is an urgent need for a method to identify personnel violations that can improve the accuracy of violation identification while increasing computing power efficiency. Summary of the Invention

[0005] The main objective of this invention is to provide a method and apparatus for identifying personnel violations, aiming to solve the technical problem that it is difficult to balance computing efficiency and the accuracy of violation identification in existing personnel violation identification methods.

[0006] To achieve the above objectives, the present invention provides a method for identifying personnel violations, the method comprising the following steps: The video stream data of the monitored area is detected using a first-person vision model to obtain detection results, which include the category of violation and the corresponding detection confidence level. A multi-dimensional risk assessment is conducted based on the inherent hazardous attributes of the monitored area, the detection results, and the cumulative time-series information of abnormal behavior to obtain a comprehensive risk score. Based on the detection confidence, the comprehensive risk score, and the temporal continuity information, it is determined whether to call the second visual model. The computational complexity of the second visual model is higher than that of the first visual model. When the call is determined, the second visual model is used to perform semantic reasoning on the region corresponding to the detection result to obtain a semantic score; The detection confidence, semantic score, and comprehensive risk score are fused to obtain a fusion decision value, and the violation identification result is output based on the fusion decision value.

[0007] Optionally, the step of fusing the detection confidence, the semantic score, and the comprehensive risk score to obtain a fusion decision value, and outputting the violation identification result based on the fusion decision value, includes: The model conflict index is determined based on the difference between the detection confidence and the semantic score; The detection confidence, the semantic score, and the comprehensive risk score are used as positive fusion terms, and the model conflict index is used as a suppression term. They are weighted and fused according to preset weights to obtain a fusion decision value. The fusion decision value is compared with the violation judgment threshold, and the corresponding violation behavior identification result is output based on the comparison result.

[0008] Optionally, after the step of outputting the violation identification result based on the fusion decision value, the method further includes: When the model conflict index is greater than the conflict threshold, the corresponding sample in the video stream data and the context sample corresponding to the sample are marked as difficult samples, and the difficult samples are added to the difficult sample candidate set. The sample value score is determined based on the model conflict index and the comprehensive risk score corresponding to the difficult sample; The difficult samples in the difficult sample candidate set are sorted according to the sample value score, and incremental training samples that meet the preset value conditions are selected from the difficult sample candidate set according to the sorting results. The second visual model is used to generate pseudo-labels for the incremental training samples to obtain pseudo-labels and corresponding label confidence scores. Pseudo-labels whose label confidence satisfies the confidence condition and whose model conflict index satisfies the conflict condition are considered high-quality pseudo-labels. The first visual model is incrementally trained based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model.

[0009] Optionally, the step of incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model includes: The total loss function is constructed using cross-entropy loss and knowledge distillation loss, wherein the knowledge distillation loss uses the output of the first visual model before incremental training as a soft label. Based on the incremental training samples and the high-quality pseudo-labels, the first visual model is incrementally trained using the total loss function. During incremental training, the change in the total loss function is monitored. Training is stopped when the change in loss is less than a preset convergence threshold, and the first visual model after training is obtained.

[0010] Optionally, after the step of incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model, the method further includes: The trained first visual model is run in parallel with the first visual model before incremental training in shadow mode, and the violation detection results are recorded under the same input. Based on the violation detection results, the trained first visual model and the first visual model before incremental training are evaluated by comparing indicators. When the evaluation of the indicators meets the preset admission conditions, the trained first visual model will be launched online in grayscale in batches according to the region. After grayscale deployment, continuously monitor the performance metrics of the trained first visual model; When the performance metric drops more than a preset threshold relative to the baseline of the first visual model before incremental training, the online first visual model is rolled back from the trained first visual model to the first visual model before incremental training.

[0011] Optionally, the step of using a first visual model to detect video stream data in the monitored area and obtaining detection results includes: Acquire video stream data of the monitored area, and perform video frame sampling on the video stream data to obtain the video frame to be detected; Based on the static scene information of the monitored area, the region of interest is marked in the video frame to be detected; The region of interest is detected using the first visual model to obtain the detection results.

[0012] Optionally, the step of performing a multi-dimensional risk assessment based on the inherent hazard attributes of the monitored area, the detection results, and the time-series accumulated information of abnormal behavior to obtain a comprehensive risk score includes: The static risk score of the scenario is determined based on at least one of the following: the area type of the monitored area, the frequency of historical accidents, and the level of security protection requirements. The violation categories are mapped to behavior risk weights, and a dynamic risk score for the behavior is determined based on the behavior risk weights and the detection confidence. The time-series cumulative risk score is determined based on the duration and frequency of abnormal behavior within a preset time window; The static risk score of the scenario, the dynamic risk score of the behavior, and the cumulative risk score over time are weighted and fused to obtain a comprehensive risk score.

[0013] Optionally, the step of determining whether to invoke the second visual model based on the detection confidence, the comprehensive risk score, and the temporal continuity information includes: Based on the detection confidence level, the comprehensive risk score, and the temporal continuity information, the trigger probability is determined according to the preset trigger probability calculation formula. When the trigger probability is greater than a preset trigger threshold, the second visual model is invoked. When the trigger probability is not greater than the preset trigger threshold, it is determined that the second visual model will not be invoked.

[0014] Optionally, the step of using the second visual model to perform semantic reasoning on the region corresponding to the detection result to obtain a semantic score when determining the invocation includes: Based on the detection results, candidate violation regions are determined, and the candidate violation regions are expanded in context to obtain semantic reasoning input regions; When it is determined to invoke the second visual model, the semantic reasoning input region is added to the asynchronous reasoning queue, and the priority of the semantic reasoning input region in the asynchronous reasoning queue is determined according to the comprehensive risk score; The second visual model performs semantic reasoning on the semantic reasoning input region according to the priority and outputs a semantic score.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a device for identifying personnel violations, the device comprising: The data detection module is used to detect video stream data of the monitored area using a first vision model and obtain detection results, which include the category of violation and the corresponding detection confidence level. The risk assessment module is used to conduct multi-dimensional risk assessment based on the inherent dangerous attributes of the monitored area, the detection results, and the time-series cumulative information of abnormal behavior, and obtain a comprehensive risk score. The trigger judgment module is used to determine whether to call the second vision model based on the detection confidence, the comprehensive risk score and the temporal continuity information. The computational complexity of the second vision model is higher than that of the first vision model. The semantic reasoning module is used to perform semantic reasoning on the region corresponding to the detection result using the second visual model when a call is determined, and to obtain a semantic score. The fusion decision module is used to fuse the detection confidence, the semantic score and the comprehensive risk score to obtain a fusion decision value, and output the violation identification result based on the fusion decision value.

[0016] Furthermore, to achieve the above objectives, the present invention also proposes a personnel violation identification device, the device comprising: a memory, a processor, and a personnel violation identification program stored in the memory and executable on the processor, the personnel violation identification program being configured to implement the steps of the personnel violation identification method described above.

[0017] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a personnel violation identification program, which, when executed by a processor, implements the steps of the personnel violation identification method described above.

[0018] This invention discloses a method for detecting video stream data of a monitored area using a first visual model to obtain detection results, including violation categories and corresponding detection confidence levels. A multi-dimensional risk assessment is performed based on the inherent danger attributes of the monitored area, the detection results, and the accumulated temporal information of abnormal behaviors to obtain a comprehensive risk score. A second visual model is invoked based on the detection confidence level, the comprehensive risk score, and temporal continuity information. The computational complexity of the second visual model is higher than that of the first visual model. When invocation is determined, semantic reasoning is performed on the area corresponding to the detection results using the second visual model to obtain a semantic score. The detection confidence level, the semantic score, and the comprehensive risk score are fused to obtain a fusion decision value, and the violation identification result is output based on the fusion decision value. Because this invention uses multi-dimensional risk assessment and invokes a highly complex second visual model as needed based on detection confidence level, comprehensive risk score, and temporal continuity information, and then fuses the detection confidence level, semantic score, and comprehensive risk score to determine the violation identification result, it improves the accuracy and computational efficiency of violation identification compared to existing technologies. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the method for identifying personnel misconduct according to the present invention. Figure 2 This is a schematic diagram of the overall framework of the method for identifying personnel misconduct according to the present invention. Figure 3 This is a flowchart illustrating the second embodiment of the method for identifying personnel misconduct according to the present invention. Figure 4 This is a flowchart illustrating the third embodiment of the method for identifying personnel misconduct according to the present invention. Figure 5This is a structural block diagram of the first embodiment of the personnel violation identification device of the present invention; Figure 6 This is a schematic diagram of the structure of a personnel violation identification device in the hardware operating environment involved in the embodiments of the present invention.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] This invention provides a method for identifying personnel violations, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for identifying personnel misconduct according to the present invention.

[0023] In this embodiment, the method for identifying personnel violations includes steps S10 to S50: Step S10: Use the first vision model to detect the video stream data of the monitored area and obtain the detection results, which include the category of violation and the corresponding detection confidence.

[0024] It should be noted that the executing entity in this embodiment can be a computer server device with data processing, network communication, and program execution functions applied in industrial safety monitoring scenarios, such as a server, tablet computer, or personal computer, or an electronic device capable of performing the above functions (such as a personnel violation recognition device). The following uses a system containing a personnel violation recognition device (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0025] It should be explained that the aforementioned first vision model can be a lightweight visual detection model with low computational complexity, few parameters, and fast inference speed, used for real-time initial screening and detection of video stream data. For example, it can be a dedicated lightweight object detection model or a lightweight behavior recognition model built on the MobileNetV3-Large backbone network.

[0026] Understandably, the aforementioned monitoring area can be a specific spatial range in an industrial setting that requires continuous safety monitoring, such as high-voltage power distribution areas, chemical storage areas, areas next to operating equipment, ordinary production workshops, raw material warehouses, or public passageways.

[0027] It should be noted that the aforementioned video stream data can be a continuous sequence of image frames captured in real time by a surveillance camera, such as a high-definition surveillance video stream with a resolution of 1920×1080 and a frame rate of 25fps.

[0028] It should be understood that the above detection results can be information output by the first-vision model after analyzing the video stream data, including the category of violation and the corresponding detection confidence level. For example, it may also include information such as the boundary box of the violation area or the boundary crossing marker. The above violation category can be a specific type of personnel violation, such as not wearing a safety helmet, not wearing protective clothing, crossing the boundary, operating equipment in violation of regulations, or smoking. The above detection confidence level can be the degree of certainty that the first-vision model is correct in judging the output violation category. It can usually be represented by a value between 0 and 1. The higher the value, the more confident the first-vision model is. For example, a detection confidence level of 0.85 means that the first-vision model believes that the output violation category is 85% credible.

[0029] To improve the detection efficiency and real-time performance of the first vision model, the step of using the first vision model to detect video stream data of the monitored area and obtain detection results may include: acquiring video stream data of the monitored area and sampling video frames of the video stream data to obtain video frames to be detected; identifying regions of interest in the video frames to be detected based on the static scene information of the monitored area; and using the first vision model to detect the regions of interest and obtain detection results.

[0030] It should be noted that the aforementioned video frame sampling can refer to the operation of extracting image frames from continuous video stream data at a preset time interval or preset frame rate. For example, extracting keyframes from a 25fps video stream at a frequency of 5fps can reduce the subsequent processing load and maintain temporal continuity.

[0031] It is understandable that the aforementioned video frame to be detected can refer to an image frame obtained after sampling the video frame and prepared to be sent to the first visual model for detection, such as an RGB image frame with a resolution of 1920×1080.

[0032] It should be explained that the static information in the above scenario can refer to prior information related to the inherent spatial and safety attributes of the monitored area, including at least one of the following: area type, equipment layout, location of safety passages, boundary of hazardous area, and inherent hazard level, such as the boundary coordinates of a high-voltage power distribution area or the area range of a chemical storage area.

[0033] It should be understood that the aforementioned region of interest (ROI) can refer to a local image region in the video frame to be detected that needs to be analyzed for violations. It usually corresponds to the boundary range of the safety detection area or danger area in the monitoring area, such as the high-voltage equipment operation area selected in the camera's field of view.

[0034] In its implementation, the system acquires video stream data of the monitored area. Then, it samples the video stream data at a preset frame rate to obtain the video frames to be detected, and performs size normalization on these frames. Next, based on the static scene information of the monitored area, the system identifies regions of interest (ROIs) within the video frames to be detected. For example, the system can determine the corresponding polygonal region as the ROI based on the boundary coordinates of the hazardous area recorded in the static scene information. The system then inputs the ROI from the video frames to be detected into a first visual model. The first visual model performs feature extraction and detection analysis on the ROI, outputting detection results, including the category of the violation and the corresponding detection confidence level.

[0035] It should be added that the first-line vision model can employ multiple dedicated lightweight detection models in parallel and collaboratively, each performing real-time detection for different violation types. Each dedicated lightweight detection model outputs the corresponding violation category and detection confidence score. For example, each dedicated lightweight detection model shares the first 12 layers of the MobileNetV3-Large backbone network as a general visual feature extractor (approximately 3.1M parameters), and then connects to a specific detection head. This avoids the accuracy degradation and training convergence difficulties caused by a single model simultaneously handling heterogeneous tasks while maintaining the total number of parameters. The dedicated lightweight detection models are as follows: 1. Safety Helmet Wearing Detection Model: Backbone Output → Two-Layer Convolutional Detector Head (3×3 Convolution + 1×1 Classification / Regression), outputting the bounding box and confidence score of the safety helmet target. Input resolution 320×320, independent detector head parameter count approximately 0.5M.

[0036] 2. Protective Clothing Wearing Detection Model: Backbone Output → Two-layer Convolutional Detector Head, outputting the bounding box of the protective clothing area and the classification confidence score for whether it is worn correctly. The number of parameters for each independent detector head is approximately 0.5M.

[0037] 3. Region Boundary Crossing Detection Model: Backbone output → Fully connected classification head (1280 → 256 → 2 classes), combined with the preset ROI polygon boundary, determines whether key personnel points have fallen into dangerous areas. This model only requires binary classification (boundary crossing / normal), and the number of parameters for an independent detection head is approximately 0.3M.

[0038] 4. Violation Detection Model: Based on the MobileViT lightweight backbone, this model uses spatiotemporal 3D convolution to perform action recognition on a 16-frame continuous sequence, outputting a binary classification result (normal operation, violation of equipment operation). This model is activated only after triggering region boundary crossing or helmet / protective clothing malfunctions to conserve computational resources. With an input of 16 frames × 224 × 224 spatiotemporal blocks, the inference latency on a T4 GPU is approximately 12ms per sequence.

[0039] The four dedicated lightweight detection models mentioned above have a total parameter count of approximately 4.7M. They can process approximately 96 video streams (5fps sampling rate per stream) in parallel on a single NVIDIA Tesla T4 GPU, meeting the real-time detection needs of a medium-sized park. Each dedicated lightweight detection model can output the following in a unified format: (a) violation category and subtype; (b) detection confidence score. (c) Boundary boxes or out-of-bounds markings of the illegal area.

[0040] Step S20: Perform a multi-dimensional risk assessment based on the inherent danger attributes of the monitored area, the detection results, and the accumulated time-series information of abnormal behavior to obtain a comprehensive risk score.

[0041] It should be noted that the aforementioned inherent hazard attributes can refer to the degree of hazard inherent in the monitored area itself that does not change dynamically over time, including at least one of the following: area type, historical accident frequency, and safety protection requirement level. For example, high-voltage power distribution areas are high-risk areas, while ordinary inspection channels are low-risk areas.

[0042] The aforementioned cumulative information on abnormal behavior over time can refer to the cumulative effect information formed by the continuous or repeated occurrence of violations over time, including the duration and frequency of abnormal behavior within a preset time window, such as a person staying in a dangerous area for 10 seconds, or a certain area experiencing 3 repeated boundary crossings within 5 minutes.

[0043] Understandably, the aforementioned multi-dimensional risk assessment can refer to the process of quantifying risks from multiple risk dimensions separately and then integrating the overall risk level by combining the risk dimensions. For example, it can be a process of assessing risks from three dimensions: static risk of the scenario, dynamic risk of behavior, and cumulative risk over time, and then integrating them.

[0044] Accordingly, the aforementioned comprehensive risk score can refer to a value obtained by weighting and fusing multi-dimensional risks, used to characterize the current overall risk level of the monitored area. It can usually be normalized to the range of 0 to 1. For example, a comprehensive risk score of 0.85 indicates a high risk, and a comprehensive risk score of 0.20 indicates a low risk.

[0045] In a specific implementation, the step of conducting a multi-dimensional risk assessment based on the inherent hazardous attributes of the monitored area, the detection results, and the time-series cumulative information of abnormal behavior to obtain a comprehensive risk score may include: determining a static risk score for the scenario based on at least one of the area type, historical accident frequency, and security protection requirement level of the monitored area; mapping the violation category to a behavior risk weight, and determining a dynamic risk score based on the behavior risk weight and the detection confidence level; determining a time-series cumulative risk score based on the duration and frequency of the abnormal behavior within a preset time window; and weighting and fusing the static risk score, the dynamic risk score, and the time-series cumulative risk score to obtain a comprehensive risk score.

[0046] It should be explained that the above-mentioned area types can be categories of monitored areas classified according to their functions and safety attributes, such as high-voltage power distribution areas, chemical storage areas, areas near operating equipment, ordinary production workshops, raw material warehouses, or public passageways. The above-mentioned historical accident frequency can be the number or frequency of safety accidents that occurred in the monitored area over a certain period of time, such as 0.5 accidents per month or 1 accident per quarter. The above-mentioned safety protection requirement levels can be the safety protection standard levels stipulated for the monitored area, such as requiring the wearing of safety helmets, protective clothing, or prohibiting personnel from entering.

[0047] It should be noted that the static risk score for the above scenarios It can be a basic risk value obtained from the assessment of the inherent danger attributes of the monitored area, usually normalized to the range of 0 to 1. For example, the static risk score of a high-voltage power distribution area is 0.95, and the static risk score of a public passageway is 0.20.

[0048] It should be understood that the aforementioned behavioral risk weights can be pre-set risk level values ​​for each category of violation, used to characterize the degree of danger of different violations. For example, not wearing a safety helmet has a weight of 0.8, smoking has a weight of 0.95, and improper operation has a weight of 0.5.

[0049] Furthermore, the dynamic risk score of the above behaviors It can be the current behavioral risk value calculated based on the behavioral risk weight and the detection confidence. For example, when the behavioral risk weight is 0.8 and the detection confidence is 0.8, the dynamic behavioral risk score can be 0.64.

[0050] Understandably, the aforementioned preset time window can be a pre-defined time range for statistically accumulating information on abnormal behavior, such as 60 seconds or 300 frames. The aforementioned duration can be the length of time elapsed from the onset of the abnormal behavior to the current moment, such as a person not wearing a safety helmet for 3 seconds. The aforementioned frequency can be the number of times the abnormal behavior occurs within the preset time window, such as two instances of boundary violations occurring within 60 seconds.

[0051] It should be noted that the above-mentioned time-series cumulative risk score It can be a cumulative risk value calculated based on the duration and frequency of abnormal behavior within a preset time window, used to reflect the effect of risk amplification over time.

[0052] For example, time-series cumulative risk score The calculation formula is as follows: ; In the formula, Let k be the weight of the abnormal behavior. The duration of this anomaly. The normalized time window represents the average time window. A single short-term anomaly scores low, while a continuous or recurring anomaly score gradually increases, reflecting the cumulative amplification effect of risk.

[0053] It should be understood that the above weighted fusion can be a method of weighting and summing multiple risk scores according to their respective weights. The formula for calculating the comprehensive risk score R is as follows: ; in, , and These correspond to the static risk score weights for scenarios, the dynamic risk score weights for behaviors, and the cumulative risk score weights for time series, respectively. You can set Prioritize responses to abnormal behavior, while also taking into account the inherent risks of the scenario and the cumulative effects over time. For example, a weighted sum is calculated based on the static risk score of the scenario (0.30), the dynamic risk score of the behavior (0.45), and the cumulative risk score of the time (0.25).

[0054] In its implementation, the system determines a static risk score for a scenario based on at least one of the following: the area type of the monitored area, the frequency of historical incidents, and the level of security protection requirements. The system maps the violation categories output by the first-vision model to behavioral risk weights and determines a dynamic risk score based on these weights and detection confidence. The system determines a time-series cumulative risk score based on the duration and frequency of abnormal behavior within a preset time window. Subsequently, the system performs a weighted fusion of the static risk score, the dynamic risk score, and the time-series cumulative risk score to obtain a comprehensive risk score. During the weighted fusion, the sum of the weights corresponding to each risk score can be 1, and the weight of the dynamic risk score is typically greater than the weight of the static risk score, which in turn is typically greater than the weight of the time-series cumulative risk score. This prioritizes responses to behavioral anomalies while also considering both inherent scenario risks and the time-series cumulative effect.

[0055] To facilitate understanding, the following examples are provided for illustration, but they do not impose specific limitations on this embodiment. For instance, the system performs a risk assessment on a high-voltage power distribution area. Based on the area type being a high-voltage power distribution area, a historical accident frequency of 0.8 times per month, and the highest level of safety protection requirements, the static risk score for the scenario is determined to be 0.95. The first visual model outputs the violation category as "not wearing a safety helmet." The system maps "not wearing a safety helmet" to a behavioral risk weight of 0.80, with a detection confidence level of 0.80, determining a dynamic risk score of 0.64. The abnormal behavior lasts for 3 seconds and occurs once within a preset time window of 60 seconds, determining a time-series cumulative risk score of 0.30. The system performs a weighted fusion of the scenario static risk score weight (0.30), the behavioral dynamic risk score weight (0.45), and the time-series cumulative risk score weight (0.25), obtaining a comprehensive risk score of 0.95 × 0.30 + 0.64 × 0.45 + 0.30 × 0.25 = 0.648.

[0056] It should be understood that by integrating static risks of the scenario, dynamic risks of behavior, and cumulative risks over time, the comprehensive risk score becomes more comprehensive and accurate, avoiding the bias of single-dimensional assessment, and providing a reliable quantitative basis for subsequent differentiated scheduling and decision-making.

[0057] Step S30: Based on the detection confidence, the comprehensive risk score, and the temporal continuity information, determine whether to call the second visual model. The computational complexity of the second visual model is higher than that of the first visual model.

[0058] It should be noted that the aforementioned temporal continuity information can refer to the degree to which abnormal behavior continuously occurs or continuously meets preset conditions in multiple consecutive video frames. It is used to characterize whether abnormal behavior has temporal continuity and stability, such as the number of consecutive frames that meet the triggering conditions. The more consecutive frames, the more likely the abnormal behavior is to actually exist rather than being occasional noise.

[0059] It should be understood that the aforementioned second visual model can be a visual model with higher computational complexity, more parameters, and stronger semantic understanding capabilities than the first visual model, used to perform deep semantic verification on the detection results output by the first visual model, such as a large visual model based on ViT-L / 16.

[0060] In its implementation, after obtaining detection confidence, comprehensive risk score, and temporal continuity information, the system comprehensively considers these three pieces of information to determine whether to invoke the second visual model. The system's judgment is based on the following principles: lower detection confidence indicates insufficient confidence in the detection results from the first visual model, thus necessitating a stronger demand for verification using the second visual model; higher comprehensive risk score indicates a higher overall risk level in the monitored area, further reinforcing the need for precise semantic judgment using the second visual model; and stronger temporal continuity information indicates persistent abnormal behavior over time, also necessitating confirmation using the second visual model. The system makes a comprehensive decision based on these three pieces of information, invoking the second visual model only when further verification is required, and not when no further verification is needed. Since the computational complexity of the second visual model is higher than that of the first visual model, the system uses this comprehensive decision-making process to implement on-demand invocation, avoiding the waste of computational resources associated with invoking the second visual model for all video frames.

[0061] To enable on-demand invocation of the second visual model and reduce computational overhead while ensuring verification of high-risk scenarios, the step of determining whether to invoke the second visual model based on the detection confidence, the comprehensive risk score, and the temporal continuity information may include: determining the trigger probability according to a preset trigger probability calculation formula based on the detection confidence, the comprehensive risk score, and the temporal continuity information; determining to invoke the second visual model when the trigger probability is greater than a preset trigger threshold; and determining not to invoke the second visual model when the trigger probability is not greater than the preset trigger threshold.

[0062] Understandably, the aforementioned trigger probability can refer to a value calculated by the system based on detection confidence, comprehensive risk score and temporal continuity information, used to characterize the likelihood of needing to call the second vision model for verification. It is usually between 0 and 1, with a larger value indicating a greater need to call the second vision model.

[0063] It should be added that the preset trigger probability calculation formula is as follows: ; In the formula, represents the detection confidence of the first visual model for the current frame; (1-C) represents the uncertainty. The lower the confidence, the greater the uncertainty, and the stronger the need to trigger the second visual model for verification. To calculate the overall risk score, the higher the risk, the more accurate semantic judgment from the second visual model is required; N represents the number of consecutive frames that meet the triggering conditions. The maximum number of consecutive frames is set. Multiple consecutive frames meeting the condition indicate that the anomaly has temporal persistence, and the trigger probability increases with N. However, the value is limited. Cut off smoothly; To detect the weighting coefficients of confidence level, comprehensive risk score, and time series continuity information, typically... Risk level is the primary decision-making factor; The sigmoid function maps the weighted sum to the (0,1) probability space.

[0064] It should be understood that the aforementioned preset trigger threshold can be a pre-set criterion value used to compare with the trigger probability to determine whether to call the second vision model, for example, it can be set to 0.60.

[0065] In its implementation, after obtaining the detection confidence level, comprehensive risk score, and temporal continuity information, the system substitutes these information into a preset trigger probability calculation formula to calculate the trigger probability. Subsequently, the system compares the trigger probability with a preset trigger threshold. When the trigger probability is greater than the preset threshold, the system determines to invoke the second visual model. When the trigger probability is not greater than the preset threshold, the system determines not to invoke the second visual model.

[0066] To facilitate understanding, the following examples are provided for illustration, but they do not impose specific limitations on this embodiment. For instance, the preset trigger threshold is set to 0.60. The system calculates a trigger probability of 0.655, which is greater than the preset trigger threshold of 0.60, so the system determines to call the second visual model. Alternatively, the system calculates a trigger probability of 0.532, which is not greater than the preset trigger threshold of 0.60, so the system determines not to call the second visual model.

[0067] Step S40: When determining the call, use the second visual model to perform semantic reasoning on the region corresponding to the detection result to obtain a semantic score.

[0068] It should be noted that the area corresponding to the above detection results can refer to a local image region that may contain violations, as located by the first visual model detection results, i.e., a candidate violation region. This can include: the bounding box region of the violating target, the region marked by personnel crossing the boundary, the region where the violation occurred, etc., rather than the entire monitoring screen or the entire region of interest.

[0069] In practical implementation, the region corresponding to the detection result can also be expanded based on the detection result's context (e.g., expanding the context region by 20% based on the detection result) to form the input region for semantic reasoning in the second vision model. For example, the region corresponding to the detection result could be the head region of a person and its surrounding context region.

[0070] It should be understood that the aforementioned semantic reasoning can refer to the process by which the second vision model performs deep semantic understanding and analysis of image regions, including identifying whether there are violations in the region, determining the specific type of violation, and generating semantic descriptions, such as determining whether there is a helmet outline in the area around a person's head, or determining whether a person has entered the boundary of a dangerous area.

[0071] Accordingly, the above semantic scores It can refer to the confidence level value output by the second vision model after semantic reasoning, which is used to characterize the presence of violations in the region. It is usually normalized to the range of 0 to 1. For example, a semantic score of 0.90 indicates that the second vision model is highly confident that there are violations in the region, while a semantic score of 0.20 indicates that the second vision model tends to believe that there are no violations in the region.

[0072] To ensure priority review of high-risk scenarios and improve response efficiency and computing power utilization, the step of using the second visual model to perform semantic reasoning on the region corresponding to the detection result and obtain a semantic score when determining the call may include: determining candidate violation regions based on the detection results and performing context expansion on the candidate violation regions to obtain semantic reasoning input regions; adding the semantic reasoning input regions to an asynchronous reasoning queue when determining to call the second visual model, and determining the priority of the semantic reasoning input regions in the asynchronous reasoning queue based on the comprehensive risk score; and having the second visual model perform semantic reasoning on the semantic reasoning input regions according to the priority and output a semantic score.

[0073] Understandably, the aforementioned candidate violation region can refer to an image region that may contain violations, initially determined based on the detection results output by the first visual model. This could be the region defined by the violation region bounding box in the detection results, or the region indicated by the boundary marker. The aforementioned context expansion can refer to enlarging the candidate violation region to incorporate more surrounding environmental information, such as expanding the candidate violation region outwards by 20%, enabling the second visual model to perform more accurate semantic understanding by incorporating the surrounding scene.

[0074] Accordingly, the aforementioned semantic reasoning input region can refer to the image region obtained after contextual expansion of the candidate violation region, which is prepared to be sent to the second vision model for semantic reasoning.

[0075] It should be noted that the aforementioned asynchronous inference queue can refer to a waiting queue that runs independently of the real-time detection pipeline of the first visual model and is used to cache semantic inference input regions to be processed by the second visual model. The second visual model obtains semantic inference input regions from the asynchronous inference queue in the order of the queue for processing, thus avoiding blocking the real-time detection process of the first visual model.

[0076] Furthermore, the aforementioned priority can refer to the order in which semantic reasoning input regions are arranged in the asynchronous reasoning queue. A higher priority means that it will be processed by the second vision model earlier. For example, semantic reasoning input regions with higher comprehensive risk scores have higher priority.

[0077] In its implementation, the system identifies candidate violation regions based on the detection results output by the first visual model and performs context expansion on these regions to obtain semantic inference input regions. Subsequently, when determining when to invoke the second visual model, the system adds the semantic inference input regions to the asynchronous inference queue. The system determines the priority of the semantic inference input regions in the asynchronous inference queue based on the comprehensive risk score; the higher the comprehensive risk score, the higher the priority. Then, the second visual model retrieves the semantic inference input regions from the asynchronous inference queue according to their priority, performs semantic inference on these regions, and outputs a semantic score.

[0078] To facilitate understanding, the following explanation uses examples, but does not impose specific limitations on this embodiment. For instance, the first visual model outputs a violation category of "not wearing a safety helmet." The detection result includes a bounding box of the person's head area. The system identifies the area defined by the bounding box as a candidate violation area and expands this candidate violation area outwards by 20% to obtain the semantic reasoning input area. When the system determines to call the second visual model, it adds the semantic reasoning input area to the asynchronous reasoning queue. At this time, the overall risk score is 0.85, and the system determines that the semantic reasoning input area has a high priority in the asynchronous reasoning queue. The second visual model retrieves the semantic reasoning input area from the asynchronous reasoning queue according to the priority, performs semantic reasoning, and outputs a semantic score of 0.88.

[0079] Understandably, when it is determined not to call the second vision model, the detection confidence and the comprehensive risk score can be fused to obtain a fused decision value; then the violation identification result can be output based on the fused decision value.

[0080] Step S50: The detection confidence, the semantic score, and the comprehensive risk score are fused to obtain a fusion decision value, and the violation identification result is output based on the fusion decision value.

[0081] Understandably, the aforementioned fusion decision value can refer to the comprehensive decision value obtained by fusing the detection confidence, semantic score, and comprehensive risk score, which is used to characterize the overall confidence level of the violation. It can usually be normalized to the range of 0 to 1. For example, a fusion decision value of 0.82 indicates that the system as a whole tends to determine that there is a violation, while a fusion decision value of 0.25 indicates that the system as a whole tends to determine that there is no violation.

[0082] It should be noted that the above-mentioned violation identification results can refer to the final identification conclusion output by the system, including whether it is determined to be a violation, and at least one of the following: violation category, violation confidence level, risk level, screenshot of the violation area, and semantic description. For example, outputting an alarm message such as "not wearing a safety helmet, high confidence level, high risk level".

[0083] In its implementation, after obtaining the detection confidence score, semantic score, and comprehensive risk score, the system fuses these scores according to a preset weighted summation rule to obtain a fusion decision value. Subsequently, the system outputs the violation identification result based on the fusion decision value. For example, the system can compare the fusion decision value with a violation judgment threshold. When the fusion decision value is greater than the violation judgment threshold, it outputs a violation identification result indicating the existence of a violation; when the fusion decision value is not greater than the violation judgment threshold, it outputs a violation identification result indicating the absence of a violation or a low-confidence observation.

[0084] For example, refer to Figure 2 , Figure 2 This is a schematic diagram of the overall framework of the personnel violation identification method of the present invention. In the diagram, after the industrial monitoring video stream is input, preprocessing is completed through video frame sampling and Region of Interest (ROI) labeling. Then, the system's multi-dimensional risk dynamic assessment module performs a multi-dimensional risk assessment, comprehensively considering the static risk of the scene corresponding to the inherent dangerous attributes of the monitored area, the behavioral risk weight mapped by the violation category detected and output by the first visual model of the ROI, the dynamic behavioral risk determined by the detection confidence of the first visual model of the ROI, and the temporal cumulative risk accumulated by the duration and frequency of abnormal behavior within a preset time window. A weighted fusion is then used to obtain a comprehensive risk score R (ranging from 0 to 1). Finally, the system's multi-factor dynamic triggering strategy module detects... The uncertainty determined by the confidence level C, the comprehensive risk score R, and the temporal continuity information N corresponding to the number of consecutive frames are used to calculate the trigger probability T, which is then compared with a preset trigger threshold. If the preset trigger threshold is not reached, the detection result of the first visual model (i.e., the small model result) is directly output. If the preset trigger threshold is reached, the second visual model (i.e., the large visual model) with higher computational complexity is called to perform asynchronous semantic reasoning on the candidate violation area after context expansion and output the semantic score. Then, through the collaborative credibility fusion module of system conflict perception, the detection confidence, semantic score, and comprehensive risk score are fused to obtain the fusion decision value and output the violation behavior identification result (which may include violation type, confidence, risk level, and area screenshot).

[0085] This embodiment discloses a method for detecting video stream data of a monitored area using a first visual model to obtain detection results, including the category of violation and the corresponding detection confidence level. A multi-dimensional risk assessment is performed based on the inherent danger attributes of the monitored area, the detection results, and the accumulated temporal information of abnormal behavior to obtain a comprehensive risk score. A second visual model is invoked based on the detection confidence level, the comprehensive risk score, and temporal continuity information. The computational complexity of the second visual model is higher than that of the first visual model. When invocation is determined, semantic reasoning is performed on the area corresponding to the detection results using the second visual model to obtain a semantic score. The detection confidence level, the semantic score, and the comprehensive risk score are fused to obtain a fusion decision value, and the violation identification result is output based on the fusion decision value. Because this embodiment uses multi-dimensional risk assessment and invokes a highly complex second visual model as needed based on the detection confidence level, the comprehensive risk score, and the temporal continuity information, and then fuses the detection confidence level, the semantic score, and the comprehensive risk score to determine the violation identification result, compared to existing technologies, this embodiment improves the accuracy and computational efficiency of violation identification.

[0086] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the method for identifying personnel misconduct according to the present invention.

[0087] Based on the first embodiment described above, in this embodiment, step S50 includes steps S501 to S503: Step S501: Determine the model conflict index based on the difference between the detection confidence and the semantic score.

[0088] It should be noted that the above model conflict index It can refer to the detection confidence level used to quantify the output of the first visual model. Semantic score compared with the output of the second visual model A numerical metric for the degree of disagreement between the two. For example, the model conflict index can be the absolute value of the difference between the detection confidence score and the semantic score. ), The closer the value is to 0, the more consistent the outputs of the two models are; the closer the value is to 1, the greater the divergence between the outputs of the two models. For example, when the detection confidence is 0.9 and the semantic score is 0.2, the model conflict index can be 0.7, indicating that there is a serious divergence between the two models; when the detection confidence is 0.88 and the semantic score is 0.85, the model conflict index can be 0.03, indicating that the two models are highly consistent.

[0089] Step S502: Using the detection confidence, the semantic score, and the comprehensive risk score as positive fusion terms and the model conflict index as a suppression term, perform weighted fusion according to preset weights to obtain a fusion decision value.

[0090] Understandably, the aforementioned positive fusion term can refer to the components used to improve the fusion decision value in the fusion calculation, including detection confidence, semantic score, and comprehensive risk score. The larger the values ​​of the above three components, the more likely it is to determine that there is a violation.

[0091] Accordingly, the aforementioned suppression term can refer to the component used to reduce the fusion decision value in the fusion computation, including the model conflict index. The larger the model conflict index, the more serious the discrepancy between the outputs of the two visual models, and the more necessary it is to suppress the risk of false alarms.

[0092] It should be understood that the aforementioned preset weights can refer to weighting coefficients pre-set for the positive fusion term and the suppression term, used to adjust the contribution of each component to the fusion decision value. For example, the detection confidence corresponds to a weight of 0.25, the semantic score to a weight of 0.40, the comprehensive risk score to a weight of 0.25, and the model conflict index to a weight of 0.10. The aforementioned weighted fusion can refer to a calculation method in which each positive fusion term is multiplied by its corresponding preset weight and then summed, and the suppression term is multiplied by its corresponding preset weight and then subtracted from the sum.

[0093] The formula for calculating the above fusion decision value is as follows: ; In the formula, To integrate decision values, To detect the weighting coefficients of confidence score, semantic score, comprehensive risk score, and model conflict index, you can set... .

[0094] To facilitate understanding, the following examples are provided for illustration, but they do not impose specific limitations on this embodiment. For example, the detection confidence is 0.88, the semantic score is 0.85, the comprehensive risk score is 0.70, and the model conflict index is 0.03. The preset weights are set as follows: detection confidence corresponds to a weight of 0.25, semantic score corresponds to a weight of 0.40, comprehensive risk score corresponds to a weight of 0.25, and model conflict index corresponds to a weight of 0.10. The system calculates the positive fusion term result as 0.25×0.88+0.40×0.85+0.25×0.70=0.735, the suppression term result as 0.10×0.03=0.003, and the fusion decision value as 0.735-0.003=0.732. As another example, the detection confidence is 0.92, the semantic score is 0.18, the comprehensive risk score is 0.40, and the model conflict index is 0.74. The system calculates the positive fusion term result as 0.25×0.92+0.40×0.18+0.25×0.40=0.402, the suppression term result as 0.10×0.74=0.074, and the fusion decision value as 0.402-0.074=0.328.

[0095] Step S503: Compare the fusion decision value with the violation judgment threshold, and output the corresponding violation behavior identification result based on the comparison result.

[0096] Understandably, the aforementioned violation determination threshold can refer to a pre-set benchmark value used to determine whether a violation exists, typically normalized to the range of 0 to 1, for example, it can be set to 0.60. When the fusion decision value is greater than the violation determination threshold, a violation is determined to exist; when the fusion decision value is not greater than the violation determination threshold, no violation is determined or it is a low-confidence observation.

[0097] Accordingly, the above comparison results can refer to the relationship between the fusion decision value and the violation judgment threshold, including two cases: the fusion decision value is greater than the violation judgment threshold and the fusion decision value is not greater than the violation judgment threshold.

[0098] To facilitate understanding, the following examples are provided for illustration, but they do not impose specific limitations on this embodiment. For example, the violation judgment threshold is set to 0.60, and the fusion decision value is 0.82. The system compares the fusion decision value of 0.82 with the violation judgment threshold of 0.60, and the comparison result is that the fusion decision value is greater than the violation judgment threshold. Based on the comparison result, the system outputs the violation behavior identification result, which includes the violation behavior category "not wearing a safety helmet", the violation confidence level of 0.82, the risk level of high risk, and a screenshot of the violation area. As another example, the fusion decision value is 0.26. The system compares the fusion decision value of 0.26 with the violation judgment threshold of 0.60, and the comparison result is that the fusion decision value is not greater than the violation judgment threshold. Based on the comparison result, the system outputs the violation behavior identification result representing no violation behavior or a low-confidence observation.

[0099] For example, refer to Figure 2 , Figure 2 The collaborative credibility fusion module for conflict perception in the system uses detection confidence, semantic score and comprehensive risk score as positive fusion terms and model conflict index as a suppression term. It performs weighted fusion according to preset weights to obtain a fusion decision value. This reduces the fusion decision value when there is a large discrepancy between detection confidence and semantic score, thereby suppressing false alarms caused by inter-model conflicts and improving the reliability of fusion decisions.

[0100] This embodiment discloses a method for determining a model conflict index based on the difference between the detection confidence level and the semantic score; using the detection confidence level, the semantic score, and the comprehensive risk score as positive fusion terms, and the model conflict index as a suppression term, performing weighted fusion according to preset weights to obtain a fusion decision value; comparing the fusion decision value with a violation judgment threshold, and outputting the corresponding violation behavior identification result based on the comparison result. Because this embodiment introduces the model conflict index as a suppression term, it reduces the fusion decision value when the discrepancy between the detection confidence level and the semantic score is large, thereby suppressing false alarms caused by inter-model conflicts. Compared with existing technologies, this embodiment improves the reliability of the fusion decision.

[0101] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the method for identifying personnel misconduct according to the present invention.

[0102] Based on the above embodiments, in this embodiment, after step S50, steps S601 to S606 are further included: Step S601: When the model conflict index is greater than the conflict threshold, the corresponding sample in the video stream data and the context sample corresponding to the sample are marked as difficult samples, and the difficult samples are added to the difficult sample candidate set.

[0103] Understandably, the aforementioned conflict threshold can refer to a pre-set benchmark value used to determine whether there is a significant discrepancy between the outputs of the first visual model and the second visual model. It can typically be normalized to the range of 0 to 1, for example, set to 0.30. When the model conflict index is greater than the conflict threshold, it indicates that the two visual models have significantly inconsistent judgments on the same sample, and this sample needs to be collected as a difficult sample. The aforementioned sample can refer to a video frame or a local image region within a video frame that corresponds to the model conflict index in the video stream data, such as the image data corresponding to the head region of a person in a certain frame.

[0104] It should be understood that the aforementioned context sample may refer to video frame data that is temporally adjacent to the sample, used to preserve the temporal evolution information of abnormal behavior, such as the video frames 5 frames before and after the sample, i.e., video frames within a 1-second time window before and after.

[0105] It should be noted that the aforementioned difficult samples can refer to samples where there is a significant discrepancy between the first and second visual models, indicating high learning value. Such samples can reflect the weaknesses in the model's current recognition capabilities. The aforementioned candidate set of difficult samples can refer to a collection used to temporarily store difficult samples, awaiting subsequent screening and processing. For example, it could be a database or cache queue that stores difficult samples and their metadata in chronological order.

[0106] In its implementation, the system compares the model conflict index with a conflict threshold after obtaining the index. When the conflict index exceeds the threshold, the system marks the corresponding sample and its context sample in the video stream data as a difficult sample. Subsequently, the system adds the difficult sample to the difficult sample candidate set. The metadata of the difficult sample can simultaneously record information such as camera ID, timestamp, risk scores for each dimension, detection confidence output by the first visual model, and semantic score output by the second visual model.

[0107] Step S602: Determine the sample value score based on the model conflict index and the comprehensive risk score corresponding to the difficult sample.

[0108] It needs to be explained that the above sample value scores It can refer to a numerical metric used to measure the value of difficult samples for incremental model training, based on the model conflict index corresponding to the difficult samples. and comprehensive risk score The results are determined comprehensively. A higher model conflict index indicates a greater discrepancy between the two visual models in their judgment of the sample, and thus the sample has greater learning value. A higher overall risk score indicates a higher safety risk in the scenario corresponding to the sample, and that the sample should be prioritized for learning. A higher sample value score indicates that difficult samples are more worthy of being included in subsequent incremental training.

[0109] The formula for calculating the sample value score is as follows: ; In the formula, For value weights, such as .

[0110] To facilitate understanding, the following examples illustrate the concept, but do not impose specific limitations on this embodiment. For instance, a difficult sample has a model conflict index of 0.74 and a comprehensive risk score of 0.85. The value weight corresponding to both the model conflict index and the comprehensive risk score is set to 0.50. The system calculates the sample value score as 0.50 × 0.74 + 0.50 × 0.85 = 0.795. As another example, a difficult sample has a model conflict index of 0.35 and a comprehensive risk score of 0.20. The system calculates the sample value score as 0.50 × 0.35 + 0.50 × 0.20 = 0.275. The former sample's value score is higher than the latter's, indicating that the former sample is more suitable for priority entry into incremental training.

[0111] Step S603: Sort the difficult samples in the difficult sample candidate set according to the sample value score, and select incremental training samples that meet the preset value conditions from the difficult sample candidate set according to the sorting results.

[0112] It should be noted that the above sorting can refer to the processing method of arranging the difficult samples in the difficult sample candidate set according to the sample value score from high to low, with the difficult samples with higher sample value scores being ranked higher.

[0113] It should be understood that the aforementioned preset value conditions may refer to pre-set judgment conditions used to select incremental training samples from the difficult sample candidate set. For example, it may be that the sample value score ranks in the top 20%, or the sample value score is greater than the preset value threshold.

[0114] Accordingly, the aforementioned incremental training samples can refer to difficult samples that have been sorted and screened, meet the preset value conditions, and are ready to be used for incremental training of the first vision model. Such samples have a high degree of conflict and a high security risk, and have high training value.

[0115] In its implementation, after obtaining the sample value scores of each difficult sample in the candidate set of difficult samples, the system sorts the difficult samples in the candidate set according to their sample value scores from high to low, obtaining a ranking result. Subsequently, the system selects incremental training samples from the candidate set of difficult samples that meet preset value conditions based on the ranking result. For example, the system can select the top 20% of difficult samples in the ranking result as incremental training samples, or it can select difficult samples with sample value scores greater than a preset value threshold as incremental training samples. The selected incremental training samples are used for subsequent incremental training of the first-vision model, thereby focusing limited training resources on difficult scenarios with high security value.

[0116] Step S604: Use the second visual model to generate pseudo-labels for the incremental training samples, and obtain pseudo-labels and corresponding label confidence scores.

[0117] It should be noted that the aforementioned pseudo-labels can refer to annotation information generated by the second vision model after reasoning on incremental training samples, used to replace manual annotation, such as the violation behavior category, violation area bounding box, or out-of-bounds marker output by the second vision model.

[0118] Accordingly, the aforementioned label confidence level can refer to the degree of confidence of the second visual model in the generated pseudo-labels. It can usually be normalized to the range of 0 to 1. For example, a label confidence level of 0.90 indicates that the second visual model is highly confident that the pseudo-labels are correct, while a label confidence level of 0.55 indicates that the second visual model is only moderately confident in the pseudo-labels.

[0119] In its implementation, the system inputs the selected incremental training samples into a second visual model. The second visual model performs semantic reasoning on the incremental training samples, outputting pseudo-labels and corresponding label confidence scores. For example, the second visual model can perform multiple rounds of reasoning on the incremental training samples and fuse the results to obtain pseudo-labels and corresponding label confidence scores. The system receives the pseudo-labels and corresponding label confidence scores output by the second visual model, providing labeled data for subsequent processing.

[0120] To facilitate understanding, the following examples are provided for illustration, but they do not impose specific limitations on this embodiment. For instance, the system inputs the selected incremental training samples into the second vision model. The second vision model performs three rounds of randomized augmented inference on the incremental training samples and outputs the pseudo-label "not wearing a safety helmet," with a corresponding label confidence level of 0.88. As another example, the second vision model infers on another sample and outputs the pseudo-label "trespassing," with a corresponding label confidence level of 0.92.

[0121] Step S605: The pseudo-labels whose label confidence satisfies the confidence condition and whose model conflict index satisfies the conflict condition are designated as high-quality pseudo-labels.

[0122] It should be understood that the aforementioned confidence level condition can refer to a pre-set criterion used to determine whether the label confidence level reaches a reliable level, such as a label confidence level greater than 0.8. The aforementioned conflict condition can refer to a pre-set criterion used to determine whether the model conflict index is small enough. The smaller the model conflict index, the more consistent the outputs of the first visual model and the second visual model are, and the more reliable the pseudo-labels are. For example, the model conflict index can be less than 0.3.

[0123] Correspondingly, the aforementioned high-quality pseudo-labels can refer to pseudo-labels that simultaneously meet the confidence condition and the conflict condition. Such pseudo-labels have sufficiently high label confidence and sufficiently low model conflict index, and are highly reliable, so they can be directly used as annotation information.

[0124] In its implementation, after obtaining the pseudo-labels and their corresponding label confidence scores, the system checks whether the label confidence score meets the confidence condition and whether the model conflict index meets the conflict condition for each pseudo-label. When both the label confidence score and the model conflict index of a pseudo-label meet the confidence condition, the system classifies it as a high-quality pseudo-label. Conversely, when either the label confidence score or the model conflict index does not meet the conflict condition, the system classifies it as a low-quality pseudo-label and sends it to a manual review queue for verification.

[0125] It should be noted that the aforementioned manual sampling queue can refer to a waiting queue used to store low-quality pseudo-tags, awaiting manual review and confirmation. For example, a security administrator can review and confirm low-quality pseudo-tags in the queue according to a certain percentage, with the sampling rate typically not exceeding 5%. This mechanism reduces the need for manual annotation by approximately 95%, making large-scale continuous operation of closed-loop learning possible.

[0126] To facilitate understanding, the following examples illustrate the concepts, but do not impose specific limitations on this embodiment. For instance, the confidence level condition is set to a label confidence level greater than 0.8, and the conflict condition is set to a model conflict index less than 0.3. A pseudo-label corresponds to a label confidence level of 0.88 and a model conflict index of 0.15. Since the label confidence level of 0.88 is greater than 0.8 and the model conflict index of 0.15 is less than 0.3, the system classifies the pseudo-label as a high-quality pseudo-label. As another example, another pseudo-label corresponds to a label confidence level of 0.75 and a model conflict index of 0.15. Since the label confidence level of 0.75 is not greater than 0.8, the confidence level condition is not met, and the system classifies the pseudo-label as a low-quality pseudo-label and sends it to a manual sampling queue for manual review and confirmation.

[0127] Step S606: Perform incremental training on the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model.

[0128] It is understandable that the first visual model after training mentioned above can refer to the first visual model obtained after incremental training, whose parameters have been updated.

[0129] To improve the efficiency and stability of model updates, the step of incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model may include: constructing a total loss function using cross-entropy loss and knowledge distillation loss, wherein the knowledge distillation loss uses the output of the first visual model before incremental training as a soft label; incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels using the total loss function; monitoring the change in loss of the total loss function during incremental training, and stopping training when the change in loss is less than a preset convergence threshold to obtain the trained first visual model.

[0130] It should be noted that the cross-entropy loss mentioned above can refer to a loss function used to measure the difference between the prediction result of the first-vision model and the high-quality pseudo-labels. The smaller the cross-entropy loss, the closer the prediction result of the first-vision model is to the high-quality pseudo-labels. The knowledge distillation loss mentioned above can refer to a loss function used to measure the difference between the output of the first-vision model during incremental training and the output of the first-vision model before incremental training. For example, KL divergence can be used as the knowledge distillation loss.

[0131] Furthermore, the above total loss function This can refer to a loss function composed of a combination of cross-entropy loss and knowledge distillation loss, used for incremental training of the first-order vision model. The total loss function is as follows: ; In the formula, For cross-entropy loss, For KL divergence distillation loss, For distillation weight, This is the predicted output of the first-vision model. This indicates pseudo-labels generated by a second vision model. This represents the output probability distribution of the new model after incremental training. This represents the output probability distribution of the old model before incremental training.

[0132] Understandably, the aforementioned soft labels can refer to the prediction results of the first visual model before incremental training on the incremental training samples, such as the probability distribution of violation categories or detection confidence of the first visual model before incremental training. Soft labels contain richer information on inter-category similarity than hard labels represented by high-quality pseudo labels.

[0133] It should be understood that the aforementioned change in loss can refer to the change in the total loss function between adjacent training epochs or adjacent validation epochs, used to characterize whether incremental training tends to stabilize. The aforementioned preset convergence threshold can refer to a pre-set numerical benchmark used to determine whether incremental training has converged, for example, it can be set to 0.001. The change in the total loss function is monitored during training. ,when The model is considered to have converged when the preset convergence threshold (usually set to 0.001) is reached, and training is stopped.

[0134] To ensure the model update is safe and controllable, after the step of incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model, the method may further include: running the trained first visual model in parallel with the first visual model before incremental training in shadow mode, and recording the violation detection results under the same input; comparing and evaluating the trained first visual model and the first visual model before incremental training based on the violation detection results; when the comparison and evaluation of the indicators meet the preset admission conditions, deploying the trained first visual model online in batches according to regions; continuously monitoring the performance indicators of the trained first visual model after gray-scale deployment; and when the performance indicators drop more than a preset threshold relative to the baseline of the first visual model before incremental training, rolling back the online running first visual model from the trained first visual model to the first visual model before incremental training.

[0135] It should be explained that the aforementioned shadow mode can refer to a working mode in which the trained first vision model and the first vision model before incremental training are deployed and run in parallel and receive the same input data, but the output of the trained first vision model is not used for actual alarm decision-making, but only for recording the detection results.

[0136] Accordingly, the aforementioned violation detection results can refer to the detection results output by the trained first-vision model and the first-vision model before incremental training, respectively, under the same input, including information such as violation category, detection confidence, and bounding box of violation region.

[0137] It should be noted that the above-mentioned comparative evaluation of indicators can refer to the process of comparing and analyzing the first-vision model after training with the first-vision model before incremental training in terms of core indicators such as precision, recall, F1 score, and false positive rate.

[0138] It should be understood that the aforementioned preset admission criteria may refer to pre-set admission standards used to determine whether the trained first visual model is allowed to enter the grayscale online range. For example, the F1 score of the trained first visual model is not lower than that of the first visual model before incremental training, and the false alarm rate is not higher than 1.1 times that of the first visual model before incremental training.

[0139] It should be noted that the above-mentioned gray-scale deployment can refer to the deployment method of gradually replacing the pre-trained first visual model with the online incremental first visual model in batches according to the region. For example, starting from 10% of the monitoring area, the replacement is carried out gradually, and each batch is observed for a period of time before entering the next batch.

[0140] Understandably, the aforementioned performance metrics can refer to indicators used to measure the performance of the first-vision model after training, including F1 score, false positive rate, inference latency, etc. The aforementioned baseline can refer to the performance level achieved by the first-vision model before incremental training under the same monitoring conditions, serving as a benchmark for measuring whether the performance of the first-vision model after training has degraded.

[0141] The aforementioned preset threshold can refer to a pre-set numerical limit used to determine whether a performance indicator has undergone unacceptable degradation, such as an F1 score drop of more than 5% or a false alarm rate increase of more than 10%.

[0142] It should be noted that the above rollback can refer to the operation of switching the online first-vision model from the trained first-vision model back to the first-vision model before incremental training.

[0143] It is understandable that the first visual model after training mentioned above can refer to the first visual model obtained after incremental training, whose parameters have been updated. The first visual model before incremental training mentioned above can refer to the first visual model used as the starting point and performance benchmark before incremental training.

[0144] In its implementation, the system runs the trained first-vision model in parallel with the pre-incremental-training first-vision model in shadow mode, receiving the same input data and recording the violation detection results under the same input. The trained first-vision model does not output any actual alarms. Subsequently, the system compares and evaluates the trained and pre-incremental-training first-vision models based on the recorded violation detection results. When the performance evaluation meets preset thresholds, the system deploys the trained first-vision model in batches according to region. After deployment, the system continuously monitors the performance metrics of the trained first-vision model. When the performance metrics drop below a preset threshold relative to the pre-incremental-training first-vision model, the system rolls back the online first-vision model from the trained model to the pre-incremental-training model.

[0145] To facilitate understanding, the following example illustrates the concept, but does not impose specific limitations on this embodiment. For instance, the system runs the trained first-vision model in parallel with the first-vision model before incremental training in shadow mode for 7 days, recording the violation detection results under the same input. The system performs a comparative evaluation of the two models. The F1 score of the trained first-vision model is 0.951, while the F1 score of the first-vision model before incremental training is 0.923. The false positive rate of the trained first-vision model is 0.028, while the false positive rate of the first-vision model before incremental training is 0.030. The preset admission criteria are that the F1 score is not lower than that of the old model and the false positive rate is not higher than 1.1 times that of the old model. The comparative evaluation of the metrics meets the preset admission criteria. The system deploys the trained first-vision model in batches according to regions. The first batch deploys 10% of the region, and after observing no abnormalities for 24 hours, the second batch deploys 30% of the region, gradually expanding to 100% of the region. After the grayscale deployment, the system continuously monitors the performance metrics of the trained first-vision model. When the F1 score of the trained first-vision model drops by more than 5% relative to the baseline of the first-vision model before incremental training, the system will roll back the online first-vision model from the trained first-vision model to the first-vision model before incremental training.

[0146] It should be understood that by using shadow mode for parallel verification, indicator comparison and evaluation, phased gray-scale deployment, and continuous monitoring after deployment with automatic rollback when performance degrades, the model update is ensured to be safe and controllable, avoiding missed detections caused by the degradation of the new model, and improving the system's reliability and high availability.

[0147] For example, refer to Figure 2 , Figure 2Samples whose model conflict index exceeds the conflict threshold are marked as difficult samples and enter the closed-loop self-learning optimization module of the system. After sample value ranking and screening, pseudo-label generation, and confidence and conflict condition filtering, they are used for incremental training of the first vision model. The new model replaces the online model after shadow parallel operation, index comparison and evaluation, phased gray-scale online deployment by region, and automatic rollback when the performance index declines. Its update results are then fed back to the multi-dimensional risk dynamic assessment module via feedback line, driving continuous iterative optimization of risk assessment and computing power scheduling.

[0148] This embodiment discloses a method for marking the corresponding sample in the video stream data and its corresponding context sample as difficult samples when the model conflict index is greater than a conflict threshold, and adding the difficult samples to a difficult sample candidate set; determining the sample value score based on the model conflict index and the comprehensive risk score corresponding to the difficult sample; sorting the difficult samples in the difficult sample candidate set according to the sample value score, and selecting incremental training samples that meet preset value conditions from the difficult sample candidate set based on the sorting results; generating pseudo-labels for the incremental training samples using the second visual model to obtain pseudo-labels and corresponding label confidence scores; using pseudo-labels whose label confidence scores meet the confidence score conditions and whose model conflict index meets the conflict conditions as high-quality pseudo-labels; and incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model. Because this embodiment automatically discovers difficult samples and filters them by value through model conflict, and uses the second visual model to generate pseudo-labels and perform quality filtering, it achieves automatic incremental training of the first visual model, reducing manual annotation costs. Compared with existing technologies, this embodiment achieves incremental training and continuous optimization of the first visual model, improving closed-loop learning efficiency.

[0149] Furthermore, this embodiment of the invention also proposes a storage medium storing a personnel violation identification program, which, when executed by a processor, implements the steps of the personnel violation identification method described above.

[0150] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the personnel violation identification device of the present invention.

[0151] like Figure 5 As shown, the personnel violation identification device proposed in this embodiment of the invention includes: a data detection module 701, a risk assessment module 702, a trigger judgment module 703, a semantic reasoning module 704, and a fusion decision module 705.

[0152] The data detection module 701 is used to detect video stream data of the monitored area using a first visual model and obtain detection results, which include the category of violation and the corresponding detection confidence level.

[0153] The risk assessment module 702 is used to perform multi-dimensional risk assessment based on the inherent dangerous attributes of the monitored area, the detection results, and the time-series accumulated information of abnormal behavior, and to obtain a comprehensive risk score.

[0154] The trigger judgment module 703 is used to determine whether to call the second visual model based on the detection confidence, the comprehensive risk score and the temporal continuity information. The computational complexity of the second visual model is higher than that of the first visual model.

[0155] The semantic reasoning module 704 is used to perform semantic reasoning on the region corresponding to the detection result using the second visual model when a call is determined, so as to obtain a semantic score.

[0156] The fusion decision module 705 is used to fuse the detection confidence, the semantic score and the comprehensive risk score to obtain a fusion decision value, and output the violation identification result based on the fusion decision value.

[0157] The data detection module 701 is further configured to acquire video stream data of the monitored area, and perform video frame sampling on the video stream data to obtain a video frame to be detected; mark the region of interest in the video frame to be detected based on the scene static information of the monitored area; and use the first visual model to detect the region of interest to obtain a detection result.

[0158] The risk assessment module 702 is further configured to determine a static risk score for a scenario based on at least one of the area type, historical accident frequency, and security protection requirement level of the monitored area; map the violation category to a behavior risk weight, and determine a dynamic risk score based on the behavior risk weight and the detection confidence level; determine a time-series cumulative risk score based on the duration and frequency of abnormal behavior within a preset time window; and perform weighted fusion of the static risk score, the dynamic risk score, and the time-series cumulative risk score to obtain a comprehensive risk score.

[0159] The trigger judgment module 703 is further configured to determine the trigger probability based on the detection confidence, the comprehensive risk score, and the temporal continuity information, according to a preset trigger probability calculation formula; determine to call the second visual model when the trigger probability is greater than a preset trigger threshold; and determine not to call the second visual model when the trigger probability is not greater than the preset trigger threshold.

[0160] The semantic reasoning module 704 is further configured to determine candidate violation regions based on the detection results, and perform context expansion on the candidate violation regions to obtain semantic reasoning input regions; when determining to call the second visual model, the semantic reasoning input regions are added to the asynchronous reasoning queue, and the priority of the semantic reasoning input regions in the asynchronous reasoning queue is determined based on the comprehensive risk score; the second visual model performs semantic reasoning on the semantic reasoning input regions according to the priority and outputs a semantic score.

[0161] This device embodiment discloses the use of a first visual model to detect video stream data of a monitored area, obtaining detection results including violation categories and corresponding detection confidence levels. A multi-dimensional risk assessment is performed based on the inherent danger attributes of the monitored area, the detection results, and the accumulated temporal information of abnormal behavior to obtain a comprehensive risk score. A second visual model is invoked based on the detection confidence level, the comprehensive risk score, and temporal continuity information. The computational complexity of the second visual model is higher than that of the first visual model. When invocation is determined, semantic reasoning is performed on the area corresponding to the detection results using the second visual model to obtain a semantic score. The detection confidence level, the semantic score, and the comprehensive risk score are fused to obtain a fusion decision value, and the violation identification result is output based on the fusion decision value. Because this device embodiment uses multi-dimensional risk assessment and invokes a highly complex second visual model as needed based on the detection confidence level, the comprehensive risk score, and the temporal continuity information, and then fuses the detection confidence level, the semantic score, and the comprehensive risk score to determine the violation identification result, compared to existing technologies, this device embodiment improves the accuracy and computational efficiency of violation identification.

[0162] Based on the first embodiment of the personnel violation identification device of the present invention, a second embodiment of the personnel violation identification device of the present invention is proposed.

[0163] In this embodiment, the fusion decision module 705 is further configured to determine a model conflict index based on the difference between the detection confidence and the semantic score; use the detection confidence, the semantic score, and the comprehensive risk score as positive fusion terms and the model conflict index as a suppression term, perform weighted fusion according to preset weights to obtain a fusion decision value; compare the fusion decision value with the violation judgment threshold, and output the corresponding violation behavior identification result based on the comparison result.

[0164] Other embodiments or specific implementations of the personnel violation identification device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0165] This application provides a personnel violation identification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the personnel violation identification method in the above embodiment 1.

[0166] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing a personnel violation identification device according to embodiments of this application. The personnel violation identification device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The personnel violation identification device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0167] like Figure 6 As shown, the personnel violation detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the personnel violation detection device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the personnel violation identification device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show personnel violation identification devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0168] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0169] The personnel violation identification device provided in this application, employing the personnel violation identification method described in the above embodiments, can solve the technical problem of balancing computational efficiency and identification accuracy in existing personnel violation identification methods. Compared with the prior art, the beneficial effects of the personnel violation identification device provided in this application are the same as those of the personnel violation identification method provided in the above embodiments, and other technical features of this personnel violation identification device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0170] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0171] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0172] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0173] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0175] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for identifying personnel violations, characterized in that, The method includes: The video stream data of the monitored area is detected using a first-person vision model to obtain detection results. The detection results include the category of violation and the corresponding detection confidence. The categories of violation include not wearing a safety helmet, not wearing protective clothing, crossing the boundary and trespassing, and operating equipment in violation of regulations. A multi-dimensional risk assessment is conducted based on the inherent hazardous attributes of the monitored area, the detection results, and the cumulative time-series information of abnormal behavior to obtain a comprehensive risk score. Based on the detection confidence, the comprehensive risk score, and the temporal continuity information, it is determined whether to call the second visual model. The computational complexity of the second visual model is higher than that of the first visual model. When the call is determined, the second visual model is used to perform semantic reasoning on the region corresponding to the detection result to obtain a semantic score; The detection confidence, the semantic score, and the comprehensive risk score are fused to obtain a fusion decision value, and the violation identification result is output based on the fusion decision value. The step of fusing the detection confidence, the semantic score, and the comprehensive risk score to obtain a fusion decision value, and outputting the violation identification result based on the fusion decision value, includes: The model conflict index is determined based on the difference between the detection confidence and the semantic score; The detection confidence, the semantic score, and the comprehensive risk score are used as positive fusion terms, and the model conflict index is used as a suppression term. They are weighted and fused according to preset weights to obtain a fusion decision value. The fusion decision value is compared with the violation judgment threshold, and the corresponding violation behavior identification result is output based on the comparison result.

2. The method for identifying personnel violations as described in claim 1, characterized in that, After the step of outputting the violation identification result based on the fusion decision value, the method further includes: When the model conflict index is greater than the conflict threshold, the corresponding sample in the video stream data and the context sample corresponding to the sample are marked as difficult samples, and the difficult samples are added to the difficult sample candidate set. The sample value score is determined based on the model conflict index and the comprehensive risk score corresponding to the difficult sample; The difficult samples in the difficult sample candidate set are sorted according to the sample value score, and incremental training samples that meet the preset value conditions are selected from the difficult sample candidate set according to the sorting results. The second visual model is used to generate pseudo-labels for the incremental training samples to obtain pseudo-labels and corresponding label confidence scores. Pseudo-labels whose label confidence satisfies the confidence condition and whose model conflict index satisfies the conflict condition are considered high-quality pseudo-labels. The first visual model is incrementally trained based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model.

3. The method for identifying personnel violations as described in claim 2, characterized in that, The step of incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model includes: The total loss function is constructed using cross-entropy loss and knowledge distillation loss, wherein the knowledge distillation loss uses the output of the first visual model before incremental training as a soft label. Based on the incremental training samples and the high-quality pseudo-labels, the first visual model is incrementally trained using the total loss function. During incremental training, the change in the total loss function is monitored. Training is stopped when the change in loss is less than a preset convergence threshold, and the first visual model after training is obtained.

4. The method for identifying personnel violations as described in claim 2, characterized in that, After the step of incrementally training the first visual model based on the incremental training samples and the high-quality pseudo-labels to obtain the trained first visual model, the method further includes: The trained first visual model is run in parallel with the first visual model before incremental training in shadow mode, and the violation detection results are recorded under the same input. Based on the violation detection results, the trained first visual model and the first visual model before incremental training are evaluated by comparing indicators. When the evaluation of the indicators meets the preset admission conditions, the trained first visual model will be launched online in grayscale in batches according to the region. After grayscale deployment, continuously monitor the performance metrics of the trained first visual model; When the performance metric drops more than a preset threshold relative to the baseline of the first visual model before incremental training, the online first visual model is rolled back from the trained first visual model to the first visual model before incremental training.

5. The method for identifying personnel violations as described in claim 1, characterized in that, The step of using a first visual model to detect video stream data in the monitored area and obtaining detection results includes: Acquire video stream data of the monitored area, and perform video frame sampling on the video stream data to obtain the video frame to be detected; Based on the static scene information of the monitored area, the region of interest is marked in the video frame to be detected; The region of interest is detected using the first visual model to obtain the detection results.

6. The method for identifying personnel violations as described in claim 1, characterized in that, The step of conducting a multi-dimensional risk assessment based on the inherent hazardous attributes of the monitored area, the detection results, and the accumulated time-series information of abnormal behavior to obtain a comprehensive risk score includes: The static risk score of the scenario is determined based on at least one of the following: the area type of the monitored area, the frequency of historical accidents, and the level of security protection requirements. The violation categories are mapped to behavior risk weights, and a dynamic risk score for the behavior is determined based on the behavior risk weights and the detection confidence. The time-series cumulative risk score is determined based on the duration and frequency of abnormal behavior within a preset time window; The static risk score of the scenario, the dynamic risk score of the behavior, and the cumulative risk score over time are weighted and fused to obtain a comprehensive risk score.

7. The method for identifying personnel violations as described in claim 1, characterized in that, The step of determining whether to invoke the second vision model based on the detection confidence level, the comprehensive risk score, and the temporal continuity information includes: Based on the detection confidence level, the comprehensive risk score, and the temporal continuity information, the trigger probability is determined according to the preset trigger probability calculation formula. When the trigger probability is greater than a preset trigger threshold, the second visual model is invoked. When the trigger probability is not greater than the preset trigger threshold, it is determined that the second visual model will not be invoked.

8. The method for identifying personnel violations as described in claim 1, characterized in that, The step of using the second visual model to perform semantic reasoning on the region corresponding to the detection result and obtain a semantic score when determining the call includes: Based on the detection results, candidate violation regions are determined, and the candidate violation regions are expanded in context to obtain semantic reasoning input regions; When it is determined to invoke the second visual model, the semantic reasoning input region is added to the asynchronous reasoning queue, and the priority of the semantic reasoning input region in the asynchronous reasoning queue is determined according to the comprehensive risk score; The second visual model performs semantic reasoning on the semantic reasoning input region according to the priority and outputs a semantic score.

9. A device for identifying personnel violations, characterized in that, The device includes: The data detection module is used to detect video stream data of the monitored area using a first vision model and obtain detection results. The detection results include the category of violation and the corresponding detection confidence. The categories of violation include not wearing a safety helmet, not wearing protective clothing, crossing the boundary and trespassing, and operating equipment in violation of regulations. The risk assessment module is used to conduct multi-dimensional risk assessment based on the inherent dangerous attributes of the monitored area, the detection results, and the time-series cumulative information of abnormal behavior, and obtain a comprehensive risk score. The trigger judgment module is used to determine whether to call the second vision model based on the detection confidence, the comprehensive risk score and the temporal continuity information. The computational complexity of the second vision model is higher than that of the first vision model. The semantic reasoning module is used to perform semantic reasoning on the region corresponding to the detection result using the second visual model when a call is determined, and to obtain a semantic score. The fusion decision module is used to fuse the detection confidence, the semantic score and the comprehensive risk score to obtain a fusion decision value, and output the violation behavior identification result based on the fusion decision value; The fusion decision module is further configured to determine a model conflict index based on the difference between the detection confidence and the semantic score; use the detection confidence, the semantic score, and the comprehensive risk score as positive fusion terms and the model conflict index as a suppression term, and perform weighted fusion according to preset weights to obtain a fusion decision value; compare the fusion decision value with the violation judgment threshold, and output the corresponding violation behavior identification result based on the comparison result.

Images