Manufacturing workshop security situation awareness method based on virtual-real mixed data set

Abstract

The invention relates to a manufacturing workshop security situation awareness method based on a virtual-real mixed data set. The method comprises the following steps: classifying unsafe states of workers in a manufacturing workshop, and constructing a quantitative evaluation system of a comprehensive safety situation and a safety level dynamic evaluation model; aiming at the problem of scarcity of real data of an unsafe state of a manufacturing workshop, a workshop unsafe state virtual data set is generated through a Stable Diffusion model, and after the workshop unsafe state virtual data set is mixed with a real data set, a target detection network is trained to realize real-time detection of the real workshop; according to the visual detection result and the multi-source discrete unsafe points, the current safety level is judged in real time in combination with an established comprehensive level judgment model; and the change of the future security situation is predicted through a Markov model, and targeted early warning measures are formulated. According to the invention, the safety level of the workshop can be detected in real time, the change of the future safety situation can be predicted, and early warning and processing can be carried out when the safety risk of the workshop is too high.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial safety monitoring in intelligent manufacturing, and in particular to a manufacturing workshop safety situation awareness method based on a virtual-real hybrid data set. Background Art

[0002] As a key execution unit in modern industrial production systems, manufacturing workshops involve complex human-machine collaboration, material flow, and equipment linkage. Safety management faces the challenge of multi-source dynamic risks. Currently, most manufacturing workshops still rely on manual inspections and fixed visual monitoring methods to complete risk identification and processing. However, with the large-scale deployment of flexible manufacturing, collaborative robots, AGVs and other equipment, the dynamic risk scenarios in workshops are becoming increasingly complex. Traditional rule-driven risk assessment methods are no longer able to cope with multi-modal and high-frequency abnormal behaviors. In particular, existing methods focus on the monitoring of independent, discrete, and static dangerous points. They lack comprehensive perception, quantitative evaluation, and dynamic trend prediction mechanisms for unsafe factors, making it difficult to support real-time perception and early warning control of the overall safety situation in the workshop.

[0003] Although the concept of digital twins provides the possibility of building a virtual-reality mapping environment, existing safety situation awareness systems generally face three bottlenecks in their applications: First, there is a lack of dynamic evaluation and prediction models for the safety situation in manufacturing workshops. The traditional method is independent single-point unsafe monitoring, which cannot achieve risk level modeling and situation quantitative assessment based on multi-source data; second, the quality of virtual data generation is insufficient. Traditional three-dimensional simulation methods rely on manual modeling, which makes it difficult to express the microscopic differences in workers' non-standard operations; third, the ability to integrate virtual and real data is weak. Models trained with pure virtual data have obvious distribution offsets, resulting in increased false detection rates and insufficient generalization capabilities in actual deployments.

[0004] In terms of detection algorithms, although deep learning object detection models such as YOLO have been used for industrial safety identification, they rely heavily on high-quality, diverse datasets. In reality, data from high-risk scenarios (such as equipment collisions and operational errors) is expensive to collect and difficult to reproduce, resulting in a severe shortage of model training data. Furthermore, the limited number of tail-category samples (such as damaged protective equipment and extreme operational behaviors) affects the model's ability to identify low-frequency events. Interference factors such as occlusion and blur in complex scenarios can also reduce annotation quality and training effectiveness.

[0005] While common image enhancement or synthesis methods, such as GANs, can alleviate the sample shortage problem, the generated images often suffer from issues like unrealistic textures, incorrect lighting, and semantic mismatches, making them difficult to effectively transfer to real-world workplaces. Furthermore, most hybrid training methods lack dedicated data alignment mechanisms, exacerbating the distribution discrepancy between virtual and real images in feature space and impacting model robustness. Furthermore, existing systems generally lack the ability to predict security trends, relying on current state responses. This makes it difficult to model trends for continuous anomalies and potential loss of control, leading to delayed responses and untimely warnings. Summary of the Invention

[0006] To address the low efficiency and effectiveness of detecting unsafe conditions among manufacturing workers, the lack of comprehensive detection, dynamic prediction, and intelligent control methods for the overall safety situation in manufacturing workshops, and the difficulty and high cost of collecting real unsafe condition datasets in deep learning-based target detection technology, this paper presents a method for safety situation awareness in manufacturing workshops based on a hybrid virtual-real dataset. This method achieves comprehensive quantitative evaluation and situational awareness prediction of unsafe conditions in the workshop through the fusion of virtual-real data and the coordinated optimization of deep learning algorithms.

[0007] In order to achieve the above-mentioned purpose, the technical solution of the present invention is implemented as follows: a manufacturing workshop safety situation awareness method based on a virtual-real hybrid data set, the steps of which are as follows:

[0008] S1. Classify the unsafe conditions in the workshop production site;

[0009] S2. By classifying unsafe conditions at the S1 production site and analyzing historical workshop accident data, a comprehensive quantitative evaluation model for safety situation was designed;

[0010] S3, generate a virtual data set by using the improved Stable Diffusion model;

[0011] S4. Using videos captured by the workshop surveillance cameras, a real dataset of unsafe conditions for workshop workers is generated.

[0012] S5, combining the virtual data set and the real data set to generate a virtual-real hybrid data set;

[0013] S6. Use the mixed dataset to train the model and evaluate the target detection algorithm using the iteratively trained model;

[0014] S7, real-time assessment of security level based on weighted fusion algorithm;

[0015] S8. Prediction and early warning of future security situation based on Markov model.

[0016] Furthermore, step S3 is specifically as follows:

[0017] S3.1, based on the Stable Diffusion model, integrating CLIP and LoRA modules;

[0018] S3.2, generate adaptive prompt words through CLIP text inversion and semantic feature extraction;

[0019] S3.3. Fine-tune the model through LoRA to optimize the scene adaptation;

[0020] S3.4. Generation and post-processing enhancement of high-resolution images.

[0021] Furthermore, step S4 is specifically as follows:

[0022] S4.1. Capture video and take image frames;

[0023] S4.2. Data preprocessing and screening;

[0024] S4.3. Classify the data set and train the model;

[0025] S4.4. Label and format the data.

[0026] Furthermore, step S6 is specifically as follows:

[0027] S6.1. Partition and balance the distribution of mixed data sets;

[0028] S6.2. YOLOv11 model training and parameter optimization;

[0029] S6.3. Iteratively train and convergence optimize the model;

[0030] S6.4. Evaluation and verification of model performance;

[0031] S6.5. Verify and apply the target detection model.

[0032] Furthermore, step S7 is specifically as follows:

[0033] S7.1. Realize multimodal data fusion;

[0034] S7.2. Realize comprehensive safety level determination of the workshop.

[0035] Furthermore, step S8 is specifically as follows:

[0036] S8.1. Construct a state transition model;

[0037] S8.2. Forecasting trends and assessing the situation in the manufacturing workshop;

[0038] S8.3. Multi-level early warning and response mechanism for dangerous situations in the workshop;

[0039] S8.4. Perform error analysis and update the model.

[0040] Beneficial effects of the present invention:

[0041] 1. By classifying the unsafe conditions of workers in the manufacturing workshop, a quantitative evaluation system for the overall safety level of the manufacturing workshop was established, achieving a quantitative assessment of the safety level situation based on multi-source data fusion. In response to the lack of a dataset of dangerous scenes in the workshop, a virtual-real hybrid dataset based on generative AI was adopted: a virtual image dataset was generated through an improved Stable Diffusion model, and a virtual-real hybrid dataset was constructed in combination with real monitoring data for use with YOLOv11 training and detection. This made up for the problem of insufficient datasets for dangerous scenes in the workshop, reduced the cost of dataset construction, and improved the accuracy and efficiency of detecting dangerous scenes in the workshop. By using the Markov model to predict changes in future safety situations and formulate targeted management measures for dangerous scenes in the workshop, a closed-loop technical path of "real-time detection + situation quantification + trend prediction + active warning" was provided for the safety management of the manufacturing workshop, effectively improving the timeliness and accuracy of risk management.

[0042] 2. By combining deep vision algorithms, multi-source sensor perception, and deep learning models, unsafe behaviors in the workshop can be identified in real time, enabling risk warnings. To address the issues of insufficient high-risk samples and poor model generalization, this paper proposes an improved Stable Diffusion virtual image generation scheme. This scheme combines real-world monitoring data to construct a hybrid virtual-real dataset, improving target detection accuracy and system deployment stability.

[0043] 3. In order to further realize the forward-looking management of the workshop safety situation, the present invention introduces a risk trend modeling mechanism based on the Markov state transition model, which dynamically predicts the future risk evolution direction according to the historical behavior sequence and environmental data, and realizes the transformation from "current perception" to "trend prediction". At the same time, a four-level dynamic safety scoring mechanism is designed to integrate multi-dimensional information such as personnel behavior, individual protection, equipment status and environmental indicators, calculate the risk level based on the weighted fusion strategy, and drive the multi-level response mechanism (such as sound and light alarms, equipment shutdown and emergency disposal) according to the scoring results to effectively ensure the safety of workshop operations. The present invention is suitable for manufacturing plants, logistics centers and other high-risk industrial sites, and has good versatility and engineering promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall flow chart of the present invention;

[0045] Figure 2 A schematic diagram for classifying unsafe conditions;

[0046] Figure 3 Schematic diagram of the safety level evaluation model;

[0047] Figure 4 Generate a flow chart for Stable Diffusion virtual data;

[0048] Figure 5 Flowchart for training an object detection algorithm based on a manufacturing workshop;

[0049] Figure 6 This is the YOLOv11 target detection flowchart;

[0050] Figure 7 This is the flow chart of the Markov model;

[0051] Figure 8 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0053] A safety situation awareness method for manufacturing workshops based on a hybrid virtual-real dataset combines deep learning with virtual and real datasets to achieve unsafe state identification, feedback, and control in physical workshops. The steps are as follows:

[0054] S1. Classify the unsafe conditions in the workshop production site.

[0055] Unsafe conditions are categorized based on factors such as personnel, equipment, materials, and the physical environment of the actual workshop production site. During the classification process, clear classification is ensured to avoid category conflicts and duplication, ensuring the accuracy and generalization of the detection model. Unsafe conditions are categorized into unsafe behavior, improper clothing, unsafe operation, and unsafe environment. Unsafe behavior includes workers running around the workshop, accidentally falling, jumping, chatting for extended periods, making phone calls, or working while fatigued. These behaviors can easily lead to collisions, falls, or distraction, potentially causing accidents. Improper clothing refers to workers not wearing the required protective equipment as required, including helmets, work clothes, or gloves, which can expose their heads, hands, or other parts of the body to risks. Unsafe operation primarily involves improper work methods or equipment use, such as performing high-altitude work without a supervisor, operating a robotic arm at excessive speed or overload, or starting equipment without authorization. These actions can cause equipment failure or workplace injuries. Unsafe environment category: refers to environmental factors at the production site that pose a threat to personnel health or escape safety, such as excessive dust concentration, lack of protective isolation in local high-temperature areas, escape routes blocked by materials, etc., which are typical environmental hidden danger risks.

[0056] Through the above classification and organization, the categories of unsafe conditions in the workshop production site can be clearly defined, providing high-quality labeled data for subsequent data set construction and deep learning model training, while ensuring that the detection, feedback and early warning of unsafe conditions are more targeted and accurate.

[0057] S2. By classifying unsafe conditions at the S1 production site and analyzing historical data on workshop accidents, a comprehensive quantitative evaluation model for safety situation is designed.

[0058] A four-level dynamic safety evaluation model is designed to divide the workshop safety level into: S4 (safest): no obvious safety hazards, and the production environment meets safety standards; S3 (relatively safe): there are minor safety hazards, but they will not immediately cause accidents; S2 (dangerous): there are major safety risks and preventive measures need to be taken; S1 (extremely dangerous): the safety situation deteriorates and immediate intervention and emergency response are required. The scoring formula is shown in formula (1):

[0059]

[0060] N i : represents the number of real-time monitoring points for the i-th type of hidden danger

[0061] N total: Deployment of total volume for pre-set monitoring points

[0062] W i : represents the missed detection compensation coefficient of the i-th type of hidden danger, which is determined by historical data verification

[0063] Missed detection rate compensation: βi is the model verification statistic (e.g., if a helmet is not worn, β2 ≤ 2.3%)

[0064] Weight dynamic range:

[0065] Behavioral norms w1∈[a1,b1]: Based on the historical accident ratio and variance analysis results, behavioral hazards have large variance and low direct risk rate, and the weight range is limited, such as w1∈[0.15,0.25].

[0066] Individual protection w2∈[a2,b2]: Based on the fact that the lack of protection directly leads to the highest proportion of serious injuries in historical accident cases, the maximum weight is given to prioritize prevention and control, such as w2∈[0.3,0.4].

[0067] Equipment operation w3∈[a3,b3]: It is set based on the proportion of historical equipment operation accidents, the proportion of on-site equipment abnormalities, and expert experience, taking into account both the failure frequency and the direct impact on working conditions, such as w3∈[0.2,0.3].

[0068] Environmental risk w4∈[a4,b4]: Based on the proportion of historical environmental risk accidents, the frequency of participation of environmental sensor data in early warning triggering and the degree of its indirect impact, a moderate weight range is assigned, such as w4∈[0.15,0.25].

[0069] The sum of the weights of each factor satisfies the constraint: w1 + w2 + w3 + w4 = 1. These weights can be dynamically adjusted based on real-time monitoring data to improve the adaptability and accuracy of the safety score.

[0070] Missed detection rate compensation: βi is the model verification statistic (e.g., if a helmet is not worn, β2 ≤ 2.3%).

[0071] S3, generate a virtual data set by using the improved Stable Diffusion model, such as Figure 3 shown.

[0072] S3.1. Based on the Stable Diffusion model, the CLIP and LoRA modules are integrated to further improve the image generation quality and efficiency of the model.

[0073] Specifically, we introduce the CLIP-ViT-L / 14 text encoder to implement text-driven image generation. The input scene description text (e.g., "operating equipment without wearing a helmet") is converted into a fixed-dimensional semantic embedding vector. The embedding vector is denoted as: is a 768-dimensional semantic vector corresponding to the input prompt word t

[0074] The semantic vector z is combined with the latent image representation x in Stable Diffusiont Combined with the cross attention module as the conditional information input diffusion model. The formula for the cross attention mechanism to fuse image and text information is shown in formula (4):

[0075]

[0076] Among them: Q comes from image features, K and V come from text semantic vector z, and d is the vector dimension, which is used to prevent numerical overflow.

[0077] S3.1.1. Introduce the LoRA module for low-rank adaptive optimization, significantly reducing the number of parameters.

[0078] Inserting the LoRA module into the cross-attention layer only fine-tunes some model parameters to reduce training complexity and memory consumption. The LoRA module represents the weight change as the product of two low-rank matrices, as shown in formula (2):

[0079] W new =W base +A·B T (2)

[0080] Where: W base is the original frozen weight matrix; It is a trainable low-rank matrix; only A and B are trained, and its parameter amount is much smaller than the original full parameter amount.

[0081] The LoRA module freezes the original model parameters and only updates the low-rank matrix, reducing the number of model parameters by about 90% and the memory usage by about 70%, while maintaining the generation quality.

[0082] S3.1.2 To enhance the detail and overall realism of generated images, the U-Net network structure in Stable Diffusion was optimized to improve the model's ability to represent highly complex scenes. During the optimization process, a hierarchical attention mechanism and a cross-scale feature fusion module were introduced to improve the representation of local and global features during the decoding phase.

[0083] Specifically, in the U-Net decoder, a cross-scale feature fusion module is introduced to splice feature maps of different resolutions to improve the image detail performance, as shown in formula (5):

[0084] F fused =Concat(F low, U psample (F high )) (5)

[0085] Among them: F fused is a low-resolution feature map; F high is a high-resolution feature map; Upsample is an upsampling operation. A multi-scale attention mechanism is introduced in the decoding stage to enhance the feature expression capability, as shown in formula (6):

[0086]

[0087] Where N is the number of multi-head attention heads, and d is the number of dimensions. Multi-scale feature fusion and the attention mechanism enable the model to simultaneously focus on both large-scale scene layout and small-scale equipment details when generating workshop scene images. The resulting images are closer to real-world data in terms of texture and detail.

[0088] S3.2. Generate adaptive prompt words through CLIP text inversion and semantic feature extraction.

[0089] To improve the semantic matching between the generated images and the workshop risk scenarios, we used CLIP text inversion technology to perform keyword extraction and semantic clustering on the workshop safety log. First, we parsed the workshop safety log using the spaCy natural language processing tool to extract high-risk scenario keywords. The set L is as follows:

[0090] L={l1,l2,…,l n}

[0091] Where: L is the semantic label set, l i Represents a single risk scenario label, for example: "welding without goggles" or "robotic arm overload operation".

[0092] The extracted labels are mapped into semantic embedding vectors through the CLIP encoder, as shown in formula (7):

[0093] z i =CLIP-Encoder(l i ) (7)

[0094] Among them, z i ∈R768 represents the label l i The semantic representation vector in the CLIP space is used to capture the high-dimensional semantic features contained in the text.

[0095] Based on the semantic embedding vector z i Combining semantic clustering with word vector nearest neighbor search, this approach automatically constructs generated prompt words that are highly relevant to the semantic vector. For example, the prompt for "welding without goggles" would be: "welding_operator,no_goggles,arc_sparks,high_risk." This approach ensures a close semantic match between the generated images and the risk scenarios in the workshop, significantly improving the authenticity and diversity of the dataset.

[0096] S3.3. Fine-tune the model through LoRA to optimize scene adaptation.

[0097] To ensure the consistency between the generated images and the real workshop environment, the LoRA model is fine-tuned based on real workshop images. The fine-tuning dataset includes: equipment model annotations (such as CNC machine tools and robotic arms); lighting parameters (brightness, contrast, color temperature);

[0098] Person's posture (standing, bending, squatting). Model fine-tuning uses the loss function as shown in formula (8):

[0099] L=λ1·L content +λ2·L style (8)

[0100] Where: L content + is the content loss, ensuring that the generated image is consistent with the reference image content; L style is the style loss, which ensures that the style of the generated image is consistent with the real workshop environment; λ1 and λ2 are weight coefficients.

[0101] S3.4. Generation and post-processing enhancement of high-resolution images.

[0102] To generate a high-fidelity virtual dataset, the ESRGAN super-resolution model is used to perform super-resolution reconstruction on the image. First, a base image with a resolution of 512×512 is generated, as shown in formula (9):

[0103] I base =Stable-Diffusion(z) (9)

[0104] The ESRGAN model is used to super-resolution the image to 1920×1080, as shown in formula (10):

[0105] I HR =ESRGAN(I base ) (10)

[0106] To simulate the interference of real surveillance cameras, Gaussian noise is injected, as shown in formula (11):

[0107] I′=I+N(0,δ 2 ) (11)

[0108] And add motion blur, as shown in formula (12):

[0109] I blur =I·K (12)

[0110] Where K is the fuzzy kernel matrix.

[0111] S4. Use the videos captured by the workshop surveillance cameras to create a real dataset of unsafe conditions of workers on the workshop site.

[0112] S4.1. Capture video and take image frames.

[0113] Workshop surveillance cameras capture real-time video of workers' operations, decompose the video frame by frame, and convert the video files into a frame-by-frame image format to generate an image dataset. All captured images are standardized in JPEG format with a resolution of 1920×1080 pixels to ensure clear and consistent images for subsequent object detection model training and evaluation.

[0114] S4.2. Data preprocessing and screening.

[0115] The image dataset generated by the frame acquisition is preprocessed and screened to ensure data quality and balanced sample distribution. First, the number of samples of different categories in the dataset is balanced as much as possible, that is, the number of times each type of unsafe state appears in the dataset is roughly similar, to prevent the model from being biased due to uneven sample distribution during training. Secondly, the image quality is screened, and blurred, severely occluded or abnormally exposed pictures are deleted to ensure the clarity and reliability of the dataset. In addition, in terms of target size processing, the scale of the target area is adjusted according to the actual scene to ensure that the model can effectively detect the target at different distances and scales. On the premise of meeting the above requirements, it is also necessary to ensure the diversity of the dataset, and select natural state pictures under different perspectives, lighting conditions and working conditions as much as possible in the acquisition scene to prevent the model from overfitting in a single environment.

[0116] S4.3. Classify the dataset and train the model.

[0117] The preprocessed image dataset is divided according to the unsafe condition categories in S1. Specifically, the following categories are used: Unsafe behavior categories (e.g., running, falling, jumping, prolonged chatting, making phone calls, and prolonged fatigue work) are divided into separate datasets and trained independently to generate detection models for each type of unsafe behavior. Irregular clothing categories (e.g., not wearing a helmet, work clothes, or gloves) are divided into separate datasets and trained separately within the network to generate detection models for irregular clothing behaviors. Intrusion risk categories (e.g., workers entering hazardous areas, non-workshop workers entering, and non-workshop objects entering the workshop) are divided into separate datasets and trained separately to generate detection models for intrusion risks. These divided datasets are then fed into the YOLOv11 object detection network for training, generating prediction models for different risk categories. This independent classification training approach helps the model accurately identify various unsafe conditions in real-world workshop scenarios, significantly improving detection efficiency and accuracy.

[0118] S4.4. Label and format the data.

[0119] Use annotation tools to annotate the dataset. Select the objects to be detected in the image and record their category and location. The annotated images automatically generate an XML annotation file with the same name as the image. The file includes: the target category (e.g., "Not wearing a hardhat" or "Worker fell"); the target location coordinates (the coordinates of the top left and bottom right corners of the annotation box); and the width and height of the annotation box. After annotation, export the dataset to YOLO format to ensure consistency with the training data format of the object detection network. This dataset will be used for model training and optimization to improve real-time detection of unsafe conditions in the workshop.

[0120] S5, combining the virtual data set and the real data set to generate a virtual-real hybrid data set;

[0121] S6. Use the mixed dataset to train the model and evaluate the target detection algorithm using the iteratively trained model, such as Figure 4 shown.

[0122] S6.1. Partition and balance the distribution of mixed data sets.

[0123] The mixed dataset is divided into training, validation, and test sets, ensuring that each set contains both virtual and real data. This prevents the model from being limited to a single type of data during testing or validation, which could affect generalization. The specific division method is as follows: for datasets with data in the tens of thousands, the ratio is 6:2:2; for datasets with data in the millions, the ratio is 98:1:1. During the data division process, data augmentation is performed on the dataset to ensure a balanced distribution of classes across the sets, avoiding over- or under-representation of certain classes, thereby preventing biased detection of specific classes by the model.

[0124] S6.2. YOLOv11 model training and parameter optimization.

[0125] The divided dataset is imported into the YOLOv11 network under the Pytorch framework for model training. The model training adopts the alternating forward propagation and back propagation. The specific process is as follows:

[0126] Forward propagation stage: The data set is input into the network, and flows from the input layer to the output layer layer by layer, and the prediction result is obtained through neuron calculation. The model calculates the loss function value under the current parameters and records the prediction error. The formula for calculating the forward propagation output result is shown in formula (13):

[0127] y^=f(W·x+b) (13)

[0128] Where: y^ is the model prediction output; W is the weight parameter; x is the input sample; b is the bias term; f is the feature mapping function of the YOLOv11 model

[0129] Back propagation stage: The difference between the predicted box and the true box is calculated based on the CIOU loss function and confidence loss, and the gradient descent method is used to update the network parameters to improve the detection accuracy. The calculation formula of the CIOU loss function is shown in formula (14):

[0130]

[0131] Where IOU is the intersection-over-union ratio between the predicted box and the ground-truth box, ρ is the Euclidean distance between the two center points, c is the diagonal length of the minimum enveloping box, v is the aspect ratio consistency measure, and α is the balance parameter. The model uses the Mish activation function to enhance the nonlinear expression ability of the network, improve detection accuracy and convergence speed, as shown in Formula (3).

[0132] Mish(x)=x×tanh(ln(1+e x )) (3)

[0133] S6.3. Iteratively train and convergence optimize the model.

[0134] During the model training process, the model parameters are continuously optimized by alternating forward propagation and backpropagation until the loss function converges to the preset threshold. Loss decline curve: In each round of training, the training loss value and the validation loss value are recorded to monitor the convergence of the model. Dynamic adjustment of learning rate: The StepLR strategy is used to dynamically adjust the learning rate to improve the convergence speed and accuracy of the model. The learning rate adjustment formula is shown in formula (15):

[0135] lr=lr×γ (15)

[0136] Where: lr is the current learning rate; γ is the adjustment coefficient, usually set to 0.1; step_size is the number of adjustment intervals, for example, adjustment every 50 epochs.

[0137] When the train loss continues to decrease and the test loss tends to be stable, it indicates that the model has converged well; if the train loss decreases but the test loss increases, it means that the model may be overfitting and it is necessary to adjust the learning rate or increase the diversity of the dataset.

[0138] S6.4. Evaluate and verify model performance.

[0139] After model training is completed, the loss reduction curve and mAP are used to evaluate the model performance.

[0140] Evaluation of the loss curve: By observing the loss curve, we can judge the convergence of the model. When the curve tends to be stable, it indicates that the model has completed training and there is no longer significant error fluctuation.

[0141] mAP indicator evaluation: Use the mAP indicator to evaluate the accuracy of the model in target detection. The mAP calculation formula is shown in formula (16):

[0142]

[0143] Where: N is the number of categories; APi is the average precision of the target of category i. The higher the mAP value, the better the detection effect of the model. Precision and recall verification: Use recall, precision, and F1-score to verify the detection performance and robustness of the model on different categories. The accuracy calculation formula is shown in formula (17):

[0144]

[0145] The recall rate calculation formula is shown in formula (18):

[0146]

[0147] The F1-score calculation formula is shown in formula (19):

[0148]

[0149] Among them: TP is the number of positive samples detected correctly; FP is the number of negative samples detected incorrectly; FN is the number of positive samples missed.

[0150] S6.5. Verify and apply the target detection model.

[0151] After training, a model is obtained that can detect different types of unsafe conditions, including: detecting unsafe behaviors such as workers running, falling, jumping in the workshop; detecting irregular wearing behaviors such as workers not wearing safety helmets, work clothes, gloves as required; detecting workshop intrusion events such as workers entering dangerous areas, non-workshop workers or objects entering the workshop.

[0152] S7. Real-time security level assessment based on weighted fusion algorithm.

[0153] In order to achieve an objective and quantitative assessment of safety risks in the operation state of a manufacturing workshop, the present invention proposes a real-time assessment method for multimodal safety levels based on a weighted fusion algorithm to achieve dynamic quantification and grading of unsafe factors in different dimensions.

[0154] S7.1. Realize multimodal data fusion.

[0155] This paper uses a YOLOv11 optimization model trained on the S6 mixed virtual-real dataset for real-time processing of workshop video streams. Through frame-by-frame video processing, gray edge filling, feature extraction, prediction box decoding, and non-maximum suppression, the model accurately identifies and classifies unsafe behaviors (such as running, climbing, and prolonged rest) and personal protective status (such as not wearing a helmet or work clothes) in the image. It then outputs detection results and target box location information.

[0156] The system also integrates multiple environmental and equipment status acquisition modules, including temperature and humidity sensors, dust concentration meters, equipment vibration sensors, and current sensors, enabling data access via the standard Modbus communication protocol. This data is then integrated with visual recognition results to form a three-dimensional "man-machine-environment" risk data stream, enabling comprehensive awareness of the manufacturing shop's operational status.

[0157] S7.2. Realize comprehensive safety level determination of the workshop.

[0158] After acquiring multimodal data, the system uses S2's weighted fusion algorithm to quantitatively score various risk factors and ultimately obtain the current comprehensive safety level.

[0159] S8. Prediction and early warning of future security situation based on Markov model

[0160] In order to further improve the foresight and proactive response capabilities of workshop safety management, the present invention introduces a Markov chain modeling method to make short-term predictions on the safety status change trends of manufacturing workshops, and triggers a multi-level linkage response mechanism based on the prediction results.

[0161] S8.1. Constructing a state transition model

[0162] The present invention maps the safety level under the workshop operation state defined by S2 into a finite state set:

[0163] S={S1,S2,S3,S4}

[0164] Among them: S1: the most dangerous state; S4: the safest state. The state level increases from low to high in order of safety level.

[0165] In order to describe the dynamic characteristics of the system state evolving over time, based on the safety level assessment sequence in the historical time period, the transition frequencies between states are counted and the state transition probability matrix P = [p ij ]. Among them, p ij Indicates that the current system is in state S i At the next moment, it transfers to state S jThe probability of satisfying:

[0166]

[0167] The transfer matrix can be obtained through training with historical rating data and supports online updates to continuously improve the accuracy and timeliness of predictions.

[0168] S8.2. Trend Forecast and Situation Assessment for Manufacturing Workshops

[0169] Assume that the current system security state is S t , and its corresponding state probability vector is:

[0170] P (0) =[p1 (0) ,p2 (0) ,p3 (0) ,p4 (0) ]

[0171] Among them, pi (0) Indicates that the current state is S i probability.

[0172] Through multi-step prediction by state transfer matrix, the state probability vector of the system at the kth time step in the future can be expressed as:

[0173] P (k) =P (0) ·P k

[0174] The system analyzes the cumulative probability of low-level states (S1, S2) in the prediction vector to determine whether there is a risk of "security decline". When any of the following conditions are met, the system will automatically activate the early warning mechanism:

[0175] At any future time step, P S1 (k) >θ1: The predicted probability of the most dangerous state exceeds the set threshold; the total probability of the lower-level states (S1 + S2) exceeds the safety risk threshold θ2; and the state prediction shows a downward trend for multiple consecutive time steps (such as a gradual shift from S4 to S2). Through this state prediction model, the system can identify potential risk trends in advance and connect with the early warning module to achieve early intervention and disposal.

[0176] S8.3. A multi-level early warning and response mechanism for hazardous situations in the workshop.

[0177] The prediction results trigger the following three-level linkage early warning strategy: Level 3 warning (information prompt, corresponding to state S3): the system displays the situation trend chart, risk index change curve and response suggestions to the management end; Level 2 warning (strategy execution, corresponding to state S2): automatic execution of strategy control, including reducing the operating frequency of equipment and restricting non-operating personnel from entering high-risk areas; Level 1 warning (emergency response, corresponding to state S1): when the prediction enters the S1 state, immediately start emergency operations such as power off protection, emergency ventilation, and broadcast evacuation.

[0178] S8.4. Perform error analysis and update the model.

[0179] The system continuously compares the actual safety status with the predicted results, performs error reverse correction based on the observation sequence, and uses maximum likelihood estimation or Bayes update method to adaptively update the state transfer matrix P, thereby improving the predictive stability and robustness of the Markov model and realizing dynamic optimization and closed-loop management of the safety perception system.

[0180] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A manufacturing workshop safety situation awareness method based on virtual and real mixed data sets, characterized by: The steps are as follows: S1. Classify the unsafe conditions in the workshop production site; S2. By classifying unsafe conditions at the S1 production site and analyzing historical workshop accident data, a comprehensive quantitative evaluation model for safety situation was designed; S3, generate a virtual data set by using the improved Stable Diffusion model; S4. Using videos captured by the workshop surveillance cameras, a real dataset of unsafe conditions for workshop workers is generated. S5, combining the virtual data set and the real data set to generate a virtual-real hybrid data set; S6. Use the mixed dataset to train the model and evaluate the target detection algorithm using the iteratively trained model; S7, real-time assessment of security level based on weighted fusion algorithm; S8. Prediction and early warning of future security situation based on Markov model.

2. The method for manufacturing workshop safety situation awareness based on a virtual-real hybrid data set according to claim 1, characterized in that: Step S3 is specifically as follows: S3.1, based on the Stable Diffusion model, integrating CLIP and LoRA modules; S3.2, generate adaptive prompt words through CLIP text inversion and semantic feature extraction; S3.

3. Fine-tune the model through LoRA to optimize the scene adaptation; S3.

4. Generation and post-processing enhancement of high-resolution images.

3. The method for manufacturing workshop safety situation awareness based on a virtual-real hybrid data set according to claim 1, characterized in that: Step S4 is specifically as follows: S4.

1. Capture video and take image frames; S4.

2. Data preprocessing and screening; S4.

3. Classify the data set and train the model; S4.

4. Label and format the data.

4. The method for manufacturing workshop safety situation awareness based on a virtual-real hybrid data set according to claim 1, characterized in that: Step S6 is specifically as follows: S6.

1. Partition and balance the distribution of mixed data sets; S6.

2. YOLOv11 model training and parameter optimization; S6.

3. Iteratively train and convergence optimize the model; S6.

4. Evaluation and verification of model performance; S6.

5. Verify and apply the target detection model.

5. The method for manufacturing workshop safety situation awareness based on virtual-real hybrid data set according to claim 1, characterized in that: Step S7 is specifically as follows: S7.

1. Realize multimodal data fusion; S7.

2. Realize comprehensive safety level determination of the workshop.

6. The method for manufacturing workshop safety situation awareness based on virtual-real hybrid data set according to claim 1, characterized in that: Step S8 is specifically as follows: S8.

1. Construct a state transition model; S8.

2. Forecasting trends and assessing the situation in the manufacturing workshop; S8.

3. Multi-level early warning and response mechanism for dangerous situations in the workshop; S8.

4. Perform error analysis and update the model.

Images