Process compliance discrimination method and device based on space-time separation heterogeneous scene model

Abstract

The invention discloses a process compliance discrimination method and device based on a space-time separation heterogeneous scene model, and the device comprises the steps: obtaining and processing video information, and then performing abnormal behavior identification, wearing compliance detection, operation process anomaly detection, environmental parameter monitoring, semantic analysis, graphical display, model training and optimization and dynamic adjustment of model parameters based on the segmented continuous video frames. The behavior in the video is identified through the deep learning model, and the transfer learning and big data analysis technologies are combined, so that the identification accuracy of the system is remarkably improved, the problem of insufficient model training is solved, and high-performance computing hardware and a feedback loop are utilized to ensure that the system can respond in real time and dynamically adjust model parameters.

Description

Technical Field

[0001] The present invention relates to the field of process compliance determination, and in particular to a process compliance determination method and device based on a spatiotemporal separation heterogeneous scenario model. Background Art

[0002] The process compliance identification method based on the spatiotemporal separation heterogeneous scenario model is an intelligent monitoring system that combines computer vision and artificial intelligence technology and is applied to the special battery disposal process. Its main goal is to improve the safety prevention and control capabilities of the special battery disposal process through real-time analysis of the disposal site monitoring video, making the operation process more "monitorable, controllable, and manageable."

[0003] The process compliance identification method based on the spatiotemporal separation heterogeneous scene model also has the following defects: the accuracy of the system is highly dependent on a large amount of high-quality training data; in practical applications, due to the diversity and complexity of the data, the model training may be insufficient, affecting the detection effect. Environmental factors at different disposal sites (such as lighting changes, occlusion, noise, etc.) have a great impact on the recognition effect of the system, which may lead to a decrease in recognition accuracy. Especially in extreme environments, the robustness of the system needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to provide a process compliance judgment method based on a spatiotemporal separation heterogeneous scene model to solve the problems of poor recognition effect and low accuracy in existing methods.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A process compliance determination method based on a spatiotemporal separation heterogeneous scenario model, comprising:

[0007] Step 1: Obtain video information of the disposal site through a surveillance camera, and unify the format of the frame data in the video information;

[0008] Step 2: preprocess the video information, perform image denoising and enhancement on the video frames; then group the video frames in the video information according to a preset length and perform sparse downsampling processing to obtain a series of continuous video frame segments;

[0009] Step 3: Input each continuous video frame segment into the pre-trained component affinity model, use the model to encode the position and direction information of the human limbs in the video segment into a two-dimensional vector field sequence and perform abnormal behavior recognition, and at the same time change the output dimension of the classification probe of the last layer of the model to N+1, where N is the predefined abnormal behavior category; set a sliding time window, and smooth the label of each continuous video frame segment after the model recognition processing as a whole to obtain a unified label corresponding to the continuous video frame segment, and then compare the obtained unified label with the preset abnormal behavior category to determine whether abnormal behavior occurs in each continuous video frame segment and determine the category of abnormal behavior; trigger an alarm when abnormal behavior occurs, and write the abnormal behavior into the abnormal behavior background record file for recording;

[0010] Step 4: Input each continuous video frame segment into the pre-trained target detection model, use the target model to perform wearing compliance and anomaly detection, and output the anomaly positioning frame;

[0011] Combined with the two-dimensional vector field sequence of the position and direction information of the human limbs obtained by processing the component affinity field model of the continuous video frame segments, the number of video frames with non-compliant and abnormal wearing in the continuous video frame segments is counted; within a preset time period, if the number of video frames with non-compliant and abnormal wearing exceeds a preset first threshold, an alarm is triggered, and the video frames with non-compliant and abnormal wearing are written into the wearing abnormal background record file for recording;

[0012] Step 5, comparing the two-dimensional vector field sequence of the position and direction information of the human limbs in the continuous video frame segments with the preset abnormal action sequence, setting a similarity threshold, and counting the number of abnormal video frames exceeding the similarity threshold in the continuous video frame segments, so as to detect whether there is an abnormal operation process in the continuous video frame segments; within a preset time period, if the number of abnormal video frames exceeds a preset second threshold, an alarm is triggered, and the abnormal video frames are written into the abnormal operation process background record file for recording;

[0013] Step 6: Use sensors to monitor environmental parameters at the disposal site, including monitoring whether the ambient temperature and humidity are abnormal. When the collected environmental parameters exceed the preset temperature threshold and humidity threshold, a warning is triggered, and the environmental parameters are written into the environmental parameter abnormality background record file for recording;

[0014] Step 7: Input the abnormal behavior background record files, abnormal wear background record files, abnormal operation process background record files, and abnormal environmental parameter background record files in the specified time period into the pre-trained Qianwen big model, and output the abnormality report in the form of natural language text, so as to evaluate the health status of the business process in the disposal site.

[0015] Furthermore, the format of the frame data in the video information is unified, including the format of size and frame rate.

[0016] Furthermore, the video frames in the video information are grouped according to a preset length, wherein the preset length is 5-20 frames.

[0017] Furthermore, the abnormal behaviors are defined as five behaviors that affect work safety at the disposal site, namely, throwing, running, smoking, fighting, and falling.

[0018] Furthermore, the target detection model adopts the YOLOv7 model.

[0019] Furthermore, the wearing compliance and anomaly detection refers to identifying common protective equipment of factory workers, including whether safety helmets, masks, and protective clothing are worn and whether the wearing is compliant.

[0020] Furthermore, the preset abnormal action sequence refers to an abnormal action sequence obtained by recording video frames of abnormal operation procedures and encoding position and direction information of human limbs in the video clips through component affinity fields.

[0021] Furthermore, the abnormal behavior background record file, the abnormal wearing background record file, the abnormal operation process background record file, the abnormal environmental parameter background record file and the abnormal report are graphically displayed to intuitively display the health status information.

[0022] Furthermore, a user feedback function is introduced to use the results obtained from steps 3 to 5 to fine-tune and optimize the models in the corresponding steps. At the same time, the threshold parameters in steps 4 to 6 can be dynamically adjusted according to changes in different environmental factors.

[0023] A process compliance determination device comprises a processor, a memory and a computer program stored in the memory; when the processor executes the computer program, the process compliance determination method based on a time-space separation heterogeneous scenario model is implemented.

[0024] A computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the process compliance determination method based on a spatiotemporal separation heterogeneous scenario model is implemented.

[0025] Compared with the prior art, the present invention has the following technical features:

[0026] By using deep learning models to identify behaviors in videos and combining them with transfer learning and big data analysis techniques, the system's recognition accuracy has been significantly improved, solving the problem of insufficient model training. High-performance computing hardware and feedback loops are used to ensure that the system can respond in real time and dynamically adjust model parameters to adapt to changes in different environmental factors, improving the real-time and flexibility of the system. By combining video information and sensor data, comprehensive monitoring of environmental parameters is achieved, enhancing the robustness of the system in complex environments. Natural language processing technology is used to perform semantic analysis of detection results, and health status information is intuitively displayed through a graphical interface, which is convenient for operators and managers to quickly understand and make decisions. The system regularly retrains the model to adapt to new business processes and environmental changes to ensure its continued efficient operation, solving the problems of reduced recognition accuracy and poor robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of a method in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention provides a process compliance determination method based on a time-space separation heterogeneous scenario model, comprising the following steps:

[0029] Step 1: Obtain video information of the disposal site through monitoring cameras, and unify the format of all collected video frame data such as size and frame rate.

[0030] Step 2, preprocessing the video information, performing image denoising and enhancement on the video frames; then grouping the video frames in the video information according to a preset length and performing sparse downsampling processing to obtain a series of continuous video frame segments; in this embodiment, each video frame segment contains 10 video frames.

[0031] Step 3, input each continuous video frame segment into the pre-trained Part Affinity Fields model, and use the model to encode the position and direction information of the human limbs in the video segment into a two-dimensional vector field sequence and perform abnormal behavior recognition. At the same time, the output dimension of the classification probe of the last layer of the model is changed to N+1, where N is the predefined category of abnormal behavior; in this scheme, abnormal behavior is defined as five behaviors that affect work safety at the disposal site, namely throwing, running, smoking, fighting, and falling; set a sliding time window, and smooth the label of each continuous video frame segment after model recognition processing as a whole to obtain a unified label corresponding to the continuous video frame segment, and then compare the obtained unified label with the preset abnormal behavior category to determine whether abnormal behavior occurs in each continuous video frame segment and determine the category of abnormal behavior; when abnormal behavior occurs, an alarm is triggered, and the abnormal behavior is written into the behavior abnormality background record file for recording.

[0032] In this step, the original 80 categories of classification probes are changed to 6 categories to meet the needs of the abnormal behavior detection task. In addition to the five preset categories of abnormal behaviors to be detected, an additional category of unknown anomalies is added to enhance the robustness of the model.

[0033] Step 4: Input each continuous video frame segment into the pre-trained target detection model YOLOv7, use the target model to perform wearing compliance and anomaly detection, and output an anomaly positioning frame; wherein the wearing compliance and anomaly detection refers to identifying whether common protective equipment such as helmets, masks, protective clothing, etc. worn by factory workers is worn and whether the wearing is compliant;

[0034] Combined with the two-dimensional vector field sequence of the position and direction information of the human limbs obtained by processing the continuous video frame segments in step 3 through the component affinity field model, the number of video frames with non-compliant and abnormal wearing in the continuous video frame segments is counted; within a preset time period, if the number of video frames with non-compliant and abnormal wearing exceeds a preset first threshold, an alarm is triggered, and the video frames with non-compliant and abnormal wearing are written into the wearing abnormality background record file for recording.

[0035] Step 5, compare the two-dimensional vector field sequence of the position and direction information of the human limbs in the continuous video frame fragments with the preset abnormal action sequence, set a similarity threshold, and count the number of abnormal video frames that exceed the similarity threshold in the continuous video frame fragments, so as to detect whether there are abnormal operation procedures in the continuous video frame fragments; within a preset time period, if the number of abnormal video frames exceeds a preset second threshold, an alarm is triggered, and the abnormal video frames are written into the operation procedure abnormal background record file for recording.

[0036] The preset abnormal action sequence refers to an abnormal action sequence obtained by recording video frames of abnormal operation procedures and encoding the position and direction information of human limbs in the video clips through the component affinity field.

[0037] Step 6, use sensors to monitor the environmental parameters of the disposal site, including monitoring whether the ambient temperature and humidity are abnormal. When the collected environmental parameters exceed the preset temperature threshold and humidity threshold, a warning is triggered, and the environmental parameters are written into the environmental parameter abnormality background record file for recording.

[0038] Step 7: Input the abnormal behavior background record files, abnormal wear background record files, abnormal operation process background record files, and abnormal environmental parameter background record files in the specified time period into the pre-trained Qianwen big model, and output the abnormal report in the form of natural language text, so as to evaluate the health status of the business process in the disposal site;

[0039] The abnormal behavior background record file, abnormal wearing background record file, abnormal operation process background record file, abnormal environmental parameter background record file and abnormal report are graphically displayed to intuitively display the health status information.

[0040] Step 8 introduces the user feedback function, and uses the results obtained from steps 3 to 5 to fine-tune and optimize the models in the corresponding steps. At the same time, the threshold parameters in steps 4 to 6 can be dynamically adjusted according to changes in different environmental factors to improve detection accuracy.

[0041] The present invention uses a deep learning model to identify behaviors in videos, and combines transfer learning and big data analysis technology to significantly improve the recognition accuracy of the system, solve the problem of insufficient model training, and use high-performance computing hardware and feedback loops to ensure that the system can respond in real time and dynamically adjust model parameters.

[0042] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A process compliance identification method based on a spatiotemporal separation heterogeneous scenario model, characterized in that: include: Step 1: Obtain video information of the disposal site through a surveillance camera, and unify the format of the frame data in the video information; Step 2: preprocess the video information, perform image denoising and enhancement on the video frames; then group the video frames in the video information according to a preset length and perform sparse downsampling processing to obtain a series of continuous video frame segments; Step 3: Input each continuous video frame segment into the pre-trained component affinity model, use the model to encode the position and direction information of the human limbs in the video segment into a two-dimensional vector field sequence and perform abnormal behavior recognition, and at the same time change the output dimension of the classification probe of the last layer of the model to N+1, where N is the predefined abnormal behavior category; set a sliding time window, and smooth the label of each continuous video frame segment after the model recognition processing as a whole to obtain a unified label corresponding to the continuous video frame segment, and then compare the obtained unified label with the preset abnormal behavior category to determine whether abnormal behavior occurs in each continuous video frame segment and determine the category of abnormal behavior; trigger an alarm when abnormal behavior occurs, and write the abnormal behavior into the abnormal behavior background record file for recording; Step 4: Input each continuous video frame segment into the pre-trained target detection model, use the target model to perform wear compliance and anomaly detection, and output the anomaly positioning frame; Combined with the two-dimensional vector field sequence of the position and direction information of the human limbs obtained by processing the component affinity field model of the continuous video frame segments, the number of video frames with non-compliant and abnormal wearing in the continuous video frame segments is counted; within a preset time period, if the number of video frames with non-compliant and abnormal wearing exceeds a preset first threshold, an alarm is triggered, and the video frames with non-compliant and abnormal wearing are written into the wearing abnormal background record file for recording; Step 5, comparing the two-dimensional vector field sequence of the position and direction information of the human limbs in the continuous video frame segments with the preset abnormal action sequence, setting a similarity threshold, and counting the number of abnormal video frames exceeding the similarity threshold in the continuous video frame segments, so as to detect whether there is an abnormal operation process in the continuous video frame segments; within a preset time period, if the number of abnormal video frames exceeds a preset second threshold, an alarm is triggered, and the abnormal video frames are written into the abnormal operation process background record file for recording; Step 6: Use sensors to monitor environmental parameters at the disposal site, including monitoring whether the ambient temperature and humidity are abnormal. When the collected environmental parameters exceed the preset temperature threshold and humidity threshold, a warning is triggered, and the environmental parameters are written into the environmental parameter abnormality background record file for recording; Step 7: Input the abnormal behavior background record files, abnormal wear background record files, abnormal operation process background record files, and abnormal environmental parameter background record files in the specified time period into the pre-trained Qianwen big model, and output the abnormality report in the form of natural language text, so as to evaluate the health status of the business process in the disposal site.

2. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: The video frames in the video information are grouped according to a preset length, wherein the preset length is 5-20 frames.

3. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: The abnormal behaviors are defined as five behaviors that affect work safety at the disposal site, namely throwing, running, smoking, fighting, and falling.

4. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: The target detection model adopts the YOLOv7 model.

5. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: The wearing compliance and anomaly detection refers to identifying whether common protective equipment of factory workers, including safety helmets, masks, and protective clothing, are worn and whether the wearing is compliant.

6. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: The preset abnormal action sequence refers to the abnormal action sequence obtained by recording the video frames of the abnormal operation process and encoding the position and direction information of the human limbs in the video clips through the component affinity field.

7. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: The abnormal behavior background record file, abnormal wearing background record file, abnormal operation process background record file, abnormal environmental parameter background record file and abnormal report are graphically displayed to intuitively display the health status information.

8. The process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to claim 1 is characterized in that: A user feedback function is introduced to use the results obtained from steps 3 to 5 to fine-tune and optimize the models in the corresponding steps. At the same time, the threshold parameters in steps 4 to 6 can be dynamically adjusted according to changes in different environmental factors.

9. A process compliance determination device, comprising a processor, a memory, and a computer program stored in the memory; characterized in that: When the processor executes the computer program, it implements the process compliance determination method based on the time-space separation heterogeneous scenario model according to any one of claims 1-8.

10. A computer-readable storage medium, wherein a computer program is stored in the medium; characterized in that: When the computer program is executed by a processor, the process compliance determination method based on the spatiotemporal separation heterogeneous scenario model according to any one of claims 1 to 8 is implemented.

Images