Steel mill personnel falling detection method based on multi-modal fusion and space-time adaptive graph convolutional network

Abstract

The invention provides a steel mill personnel tumble detection method based on multi-modal fusion and a space-time adaptive graph convolutional network, and aims to solve the problem of steel mill personnel tumble detection and improve industrial production safety. A high-definition camera is arranged in a key area of a steel mill to collect video data and collect information such as a public data set, professional team labeling positions, postures and falling types, and data enhancement is carried out by applying various technologies. YOLOv8 is selected as a basic model, CBAM attention mechanism improvement is introduced, space-time adaptive graph convolutional network processing is performed after AlphaPose attitude information is fused, data set training is divided, and a strategy is adjusted according to performance indexes. In the aspect of system implementation and deployment, a monitoring and early warning system containing multiple modules is designed, a DeepSORT algorithm is introduced to track a target, a model is embedded by adopting video processing and hardware acceleration means, and local or cloud deployment and test optimization can be realized. A multi-index system and various test data sets are adopted for model performance evaluation, optimization is carried out from the aspects of structure, data and the like according to evaluation results, and a continuous optimization mechanism is established. According to the method, the accuracy, real-time performance and reliability of steel mill personnel falling detection are improved, and the method is of great significance to industrial safety production.

Description

Technical Field

[0001] The present invention belongs to the intersection of deep learning and computer vision technology, and focuses on using advanced algorithms and technical means to accurately monitor and analyze human behavior in industrial scenarios (especially steel plant environments). It particularly involves a personnel fall detection method based on an improved spatiotemporal adaptive graph convolutional network, which aims to improve the accuracy, real-time and reliability of personnel fall detection through innovative technical solutions, and provide strong technical support for industrial safety production. Background Art

[0002] In today's industrial production landscape, factories, as the manufacturing core of various products, are facing many severe safety challenges while promoting economic development. Typical industrial sites such as steel mills often have production environments with unfavorable factors such as high temperature, high humidity, complex terrain, and high-intensity workloads. These harsh conditions not only pose a direct threat to workers' health, such as causing fatigue and excessive physical exhaustion, but also significantly increase the risk of workers falling during operations.

[0003] Traditional safety management models and technical means have obvious limitations in dealing with the problem of detecting falls. It is difficult to achieve real-time and continuous monitoring of large working areas based on manual observation, which is prone to negligence and omissions. Some simple sensor monitoring methods, such as devices based on accelerometers or pressure sensors, can sense changes in human motion to a certain extent, but they have problems such as high false alarm rate, inconvenience in wearing, limited battery life, and inaccurate recognition of complex movements. For example, in the complex electromagnetic environment of a steel plant, sensor signals may be interfered with, affecting the accuracy of detection; workers' violent movements during high-intensity operations may also be misjudged as falls, resulting in unnecessary alarms.

[0004] At the same time, with the rapid development of artificial intelligence technology, deep learning technology has achieved breakthrough results in image recognition, target detection and other fields. Its powerful automatic feature extraction and pattern recognition capabilities provide a new idea and technical approach to solve the complex problem of personnel fall detection. Through deep learning algorithms, in-depth analysis of human behavior in video images can more accurately capture key information such as human posture and movement changes, thereby achieving accurate judgment of fall behavior. This will not only help improve the safety management level of industrial sites, reduce casualties and production interruptions caused by fall accidents, but also reduce potential legal risks and economic losses for enterprises, and promote industrial production to develop in a safer and more efficient direction. Summary of the invention

[0005] In order to effectively improve the safety of steel plant workplaces and detect worker falls in a timely and accurate manner, this paper innovatively proposes a steel plant personnel fall detection method based on multimodal fusion and spatiotemporal adaptive graph convolutional network. This method makes full use of the advantages of deep learning technology, uses cameras to widely collect video data, and then realizes all-round, real-time monitoring of worker behavior and fall detection, and issues an alarm at the first time, thereby effectively reducing the risk of accidents and ensuring the safety of workers' lives and the normal progress of production activities.

[0006] Step 1: Data preparation and preprocessing

[0007] 1. Data Collection

[0008] We deeply integrate the internal resources of the steel plant and continuously acquire long-term, multi-angle video data through high-definition surveillance cameras carefully arranged in various key work areas (such as high-temperature furnaces, transportation channels, material storage areas, etc.). These areas cover places where workers are active and have a high risk of falling, ensuring that the collected data can fully reflect the actual working scenes in the complex environment of the steel plant. At the same time, we actively expand the data sources and widely collect relevant data sets that are publicly available on the Internet, including but not limited to videos of personnel activities recorded in different industrial scenarios, simulated fall experiments, and daily activity scenarios. These diverse data sources effectively supplement the internal data of the steel plant, further enrich the diversity and complexity of the data set, and provide a more comprehensive and rich information basis for subsequent model training.

[0009] During the data collection process, the particularities of the steel plant environment were fully taken into account, such as the dramatic changes in light (from strong furnace light to dim corners), dense personnel and complex movements (multiple people working together to operate large equipment, carry heavy objects, etc.), and the presence of various obstacles (large steel equipment, pipelines, etc.). In response to these situations, high-resolution, high-frame-rate camera equipment is used, equipped with advanced light adaptive technology to ensure that clear and stable video images can be obtained under different lighting conditions. At the same time, the installation position and angle of the camera are carefully designed to minimize occlusion and ensure the integrity and accuracy of the data.

[0010] 2. Data Annotation

[0011] We organize a professional annotation team to perform meticulous manual annotation on the massive amount of video data collected. The annotation content includes not only the precise position of the person (accurately framing the position of the person in the image in the form of a bounding box), a rich set of posture key points (such as the coordinate points of key parts such as the head, neck, shoulders, elbows, wrists, waist, knee joints, ankle joints, etc.), but also focuses on annotating whether the person falls and the detailed status information of the fall (such as different types of falls such as forward falls, backward falls, and sideways falls). For fall events, the start frame (the instantaneous picture of the beginning of the fall) and the end frame (the picture of the stable state after the fall) are accurately annotated to provide accurate learning targets for subsequent model training.

[0012] 3. Data Augmentation

[0013] A variety of advanced data enhancement technologies are used to comprehensively expand the labeled data. Mosaic data enhancement technology creates new training samples by cleverly combining four different pictures, increasing the diversity and complexity of the data and effectively improving the model's adaptability to different scenes and object combinations. Adaptive image scaling technology dynamically adjusts the image size according to the network input requirements, while maintaining important image information, reducing the amount of calculation and improving processing efficiency. In addition, random cropping (randomly selecting a part of the image for cropping to simulate different perspectives and local information), flipping (flipping the image horizontally or vertically to increase the symmetry and diversity of the data), and rotation (randomly rotating the image within a certain angle range to simulate different postures and directions) are also used to further expand the scale of the data set and enhance the generalization ability of the model.

[0014] In particular, we simulate various complex lighting changes (such as strong light, shadows, light flickering, etc.) and common occlusion situations (such as people being partially blocked by equipment, multiple people being blocked, etc.) in response to the environmental characteristics of steel plants. We increase the authenticity and challenge of the data by adding noise and blocking some image areas. This allows the model to fully learn the ability to accurately identify people and falling behaviors in complex environments during training, making it more robust and reliable in practical applications.

[0015] Step 2: Model building and training

[0016] 1. Model selection and improvement

[0017] After in-depth research and comparative analysis of various target detection models, YOLOv8 was finally selected as the basic model of the present invention. YOLOv8 has high accuracy and real-time performance in the field of target detection, but there is still room for improvement when dealing with small target detection and complex background interference in complex scenes of steel mills. In response to these problems, the YOLOv8 model is carefully improved. An attention mechanism (such as the CBAM attention mechanism) is cleverly added to its neck end structure, and the importance weights of each channel and spatial position in the feature map are accurately calculated, so that the model can automatically focus on the feature information that is more critical to the detection task, effectively enhancing the detection ability of small targets and personnel under complex backgrounds, and significantly improving the accuracy and stability of detection.

[0018] 2. Introducing multimodal data fusion

[0019] Innovatively introduce human posture information into the fall detection model. Use the advanced AlphaPose posture estimation algorithm to accurately obtain the key point information of the human skeleton. Deeply integrate these posture key point information with the target detection results of the YOLOv8 model to provide the model with richer and more comprehensive feature information. For example, information such as the changing trend of human posture and the relative position relationship between joints can assist the model to more accurately judge the action intention and behavior status of the person, thereby more accurately identifying the fall behavior.

[0020] The spatiotemporal adaptive graph convolutional network (STA-GCN) is further applied to the fused feature data. STA-GCN can effectively capture the topological structure relationship between the key points of the human skeleton in the spatial dimension, and model and analyze the posture changes between consecutive frames in the temporal dimension. In this way, the model can fully learn the dynamic characteristics and change laws of human motion in the spatiotemporal dimension, greatly enhancing the ability to understand and recognize human motion, especially showing excellent performance when processing dynamic and continuous falling motions.

[0021] 3. Model training and optimization

[0022] The carefully annotated and enhanced dataset is divided into training set, validation set and test set (7:2:1 ratio) in a scientific and reasonable ratio to ensure that each subset can fully represent the characteristic distribution of the entire dataset. In the model training process, stochastic gradient descent (SGD) and its variant optimization algorithm are used to carefully adjust the parameters of the optimization algorithm (such as learning rate, momentum, weight decay, etc.) according to the characteristics of the training data and the needs of the model. Through continuous iterative calculations, the parameters of the model are gradually adjusted to minimize the error (loss function) between the model's prediction results and the annotated data, so that the model can accurately learn the patterns and laws in the data.

[0023] During the training process, closely monitor the performance indicators of the model on the validation set (such as accuracy, recall, mean average precision (mAP), F1 value, etc.) According to the changing trends of these indicators, timely adjust the training strategy to ensure that the model has good generalization ability and can accurately detect people falling in unknown actual scenarios.

[0024] Step 3: System implementation and deployment

[0025] 1. System design

[0026] A complete real-time monitoring and early warning system for personnel falls is carefully designed. The system covers multiple functional modules, including an efficient data input module (responsible for real-time acquisition and transmission of monitoring video data), an accurate target detection module (using the improved YOLOv8 model to detect and locate personnel in the video), a detailed posture recognition module (using the AlphaPose posture estimation algorithm to extract personnel posture information), an intelligent behavior classification and recognition module (based on the spatiotemporal adaptive graph convolutional network to classify personnel behavior and determine whether it is a fall behavior) and a timely early warning module (quickly issue an alarm when a fall behavior is detected). The modules work closely together to form an organic whole to ensure that the system can run stably and efficiently.

[0027] The DeepSORT target tracking algorithm is specially introduced to achieve continuous and accurate tracking and monitoring of workers. The algorithm assigns a unique ID to each detected worker through operations such as feature extraction, similarity calculation, and data association, and accurately tracks its position and movement trajectory in consecutive frames. Even if the worker is partially obscured or temporarily disappears from the field of view, the DeepSORT algorithm can still use its powerful tracking ability to predict the position of the target, ensure accurate identification of worker targets in different frames, effectively avoid multiple warnings caused by occlusion or misjudgment, and improve the reliability and stability of the system.

[0028] 2. System Implementation

[0029] The trained complex model is efficiently embedded into the actual system, and the system architecture and algorithm are carefully optimized to ensure that the system can process the video stream data from the camera in real time. In the implementation process, advanced video processing technology and hardware acceleration methods (such as GPU acceleration, dedicated image processing chips, etc.) are used to achieve functions such as fast video streaming, efficient reasoning (using models to analyze and judge the behavior of people in video frames in real time), timely alarm (immediately issue various forms of alarm signals such as sound and light when a fall is detected), and stable data storage (save video data, test results and other information for subsequent analysis and query). At the same time, the system is fully optimized, including memory management, computing resource allocation, network communication optimization, etc., to ensure that the system can run stably and efficiently in the complex environment of the steel plant and meet the strict requirements of real-time and accuracy.

[0030] 3. System deployment and testing

[0031] According to the actual layout, network environment and computing resources of the steel plant, the system can be flexibly deployed to the appropriate location in the plant. You can choose to deploy it locally on embedded devices (such as edge computing devices, smart cameras, etc.) to achieve low-latency, high-reliability real-time monitoring; you can also deploy part or all of the system functions on the cloud server according to demand, and use the powerful computing resources of the cloud for data processing and analysis, which is convenient for centralized management and remote monitoring. During the deployment process, the compatibility, scalability and security of the system are fully considered to ensure that the system can be seamlessly integrated with the existing monitoring system and network infrastructure of the steel plant, and can adapt to the needs of future business development and technology upgrades.

[0032] Comprehensive and in-depth tests were conducted in actual scenarios, and a large amount of test data was collected from different work scenarios (such as production workshops, warehouses, loading and unloading areas, etc.), various light conditions (from strong light during the day to weak light at night), different time periods (including peak and trough periods), multiple falling postures (simulating falling in different directions, speeds and strengths), and different degrees of occlusion (such as equipment occlusion, cross-occlusion of personnel, etc.). Through detailed analysis of these rich and diverse test data, the effectiveness of the system (whether indicators such as detection accuracy and recall rate meet expectations) and practicality (performance of the system in terms of stability, reliability, ease of use, etc. in actual environments) were fully verified. Based on the test results, the system was further optimized and adjusted to ensure that the system can accurately and timely detect the falling behavior of personnel in the complex and changeable steel plant environment, providing reliable protection for the life safety of workers.

[0033] Step 4: Model performance evaluation and optimization

[0034] 1. Performance Evaluation Metrics

[0035] A comprehensive and scientific performance evaluation index system is used to accurately measure the performance of the model. In addition to commonly used indicators such as accuracy, recall, average accuracy and detection speed, some other important indicators are also introduced, such as F1 value (a balanced indicator that comprehensively considers accuracy and recall), false alarm rate (the proportion of normal behavior mistakenly predicted as falling behavior), missed alarm rate (the proportion of actual falling behavior that was not detected), etc. Through the comprehensive analysis of these multi-dimensional indicators, the performance of the model in different aspects can be evaluated more comprehensively and accurately, providing a strong basis for the optimization and improvement of the model.

[0036] In order to ensure the reliability and objectivity of the evaluation results, a large-scale and diverse test data set is used for evaluation. This test data set covers various situations that may occur in actual scenarios, including different environmental conditions, personnel characteristics, behavior patterns, etc. At the same time, multiple repeated tests and cross-validation methods are used to reduce the impact of random factors on the evaluation results, making the evaluation results more stable and reliable, and able to truly reflect the performance level of the model in actual applications.

[0037] 2. Model Optimization Strategy

[0038] According to the performance evaluation results, the model is optimized and improved in a targeted manner. If the model performs poorly in terms of accuracy, it may be that the model structure is not complex enough or the training data is insufficient. At this time, you can consider increasing the depth and width of the model (such as adding more convolutional layers, fully connected layers, etc.), or further expanding and optimizing the training data set (such as collecting more actual scene data, performing more detailed data enhancement operations, etc.).

[0039] Establish a continuous optimization mechanism, regularly collect new data (including normal behavior data and newly emerging fall case data), add it to the training set, and update the model. By continuously introducing new data, the model can adapt to changes in the steel plant environment (such as equipment updates, work process adjustments) and differences in the behavior patterns of different workers (such as the behavioral characteristics of workers of different ages, genders, and work experience), ensuring that the model always maintains good generalization capabilities and performance. At the same time, pay close attention to the latest research results and technology development trends in the field of deep learning, introduce new algorithms and technologies (such as more advanced attention mechanisms, optimized graph convolutional network structures, etc.) in a timely manner, and further improve and optimize the model to maintain the model's leading performance in the field of personnel fall detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments:

[0041] Figure 1System architecture diagram: From data input to final output, it covers the operating logic of the entire system.

[0042] Figure 2 Deepsort target tracking diagram: shows how the algorithm works during target tracking.

[0043] Figure 3 STA-GCN network structure: shows the structure of this network, which is usually used to process data with spatiotemporal features, such as human posture analysis in videos. It can adaptively learn graph structure and spatiotemporal features, and plays an important role in capturing the topological structure relationship of key points of human skeleton and the change rules of posture between consecutive frames. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these embodiments.

[0045] Data Collection Example

[0046] In a large steel plant, according to the characteristics of its production process and areas with frequent personnel activities, key work areas such as high-temperature furnaces, transportation channels, and material storage areas were carefully selected for the deployment of high-definition surveillance cameras. A total of [X] high-definition cameras were installed next to the high-temperature furnace. The installation angles of the cameras were precisely calculated to ensure that they can cover the range of activities of the personnel next to the furnace, while avoiding image distortion or interference caused by high temperature and strong light. In the transportation channel, a camera was installed every [specific distance] to ensure comprehensive monitoring of personnel and goods during transportation. In the material storage area, [X] cameras were reasonably distributed according to its area and stacking layout to ensure monitoring without blind spots.

[0047] These cameras have the characteristics of high resolution (such as [specific resolution]) and high frame rate (such as [specific frame rate]), and are equipped with advanced light adaptive technology. Light adaptive technology monitors the ambient light intensity in real time and automatically adjusts the camera's aperture size, exposure time and other parameters, so that clear and stable video images can still be obtained in the steel plant environment where the light changes dramatically (such as from strong furnace light areas to dim corners). Its working principle can be simply expressed as:

[0048] I adjusted =f(I raw , L env )

[0049] Among them, I adjusted represents the adjusted image, I raw represents the original collected image, L env represents the ambient light intensity, and f represents the light adaptive adjustment function.

[0050] The data collection period is set to continuous collection, covering different production shifts in the steel plant, including day shift, mid-shift and night shift, to obtain diversified data on personnel activities in different time periods. At the same time, relevant data sets are widely collected from the Internet, such as videos of falling experiments conducted in a simulated steel plant environment. These videos contain falling movements and normal behaviors of people of different ages, genders, and body shapes under various complex conditions such as simulated strong light, weak light, and occlusion, which effectively supplements the diversity of data collected inside the steel plant.

[0051] Data annotation example

[0052] We organized a data annotation team composed of professionals. All team members have undergone rigorous training and are familiar with the definition and annotation specifications of key points of human posture. We use professional annotation tools to perform meticulous manual annotation on the massive amount of video data collected.

[0053] When marking the position of a person, the position of the person in the image is accurately framed in the form of a bounding box. The coordinates of the bounding box are expressed as (x min ,y min ,x max ,y max ), where (x min ,y min ) is the coordinate of the upper left corner of the bounding box, (x max ,y max ) is the coordinate of the lower right corner of the bounding box.

[0054] For posture key point annotation, the coordinate points of multiple key parts such as head, neck, shoulder, elbow, wrist, waist, knee joint, ankle joint, etc. are marked. Assume that the coordinates of the i-th key point are (x i ,y i ), then the annotation result of the key points of the entire human body posture can be expressed as a matrix Where n is the number of key points.

[0055] For the labeling of whether the person falls and the detailed status information of the fall, if the person is in the state of falling, it is marked as 1, and the type of fall is detailed, such as falling forward is marked as F, falling backward is marked as B, falling sideways is marked as S, etc.; if the person does not fall, it is marked as 0. At the same time, the starting frame f of the fall event is accurately marked start and the end frame f end , providing accurate learning objectives for subsequent model training.

[0056] Model training example

[0057] The carefully labeled and enhanced dataset is divided into training set, validation set and test set according to the ratio. In the model training process, the Adam optimization algorithm in stochastic gradient descent (SGD) and its variant optimization algorithm is selected, and the formula for updating parameters is:

[0058]

[0059] Among them, θ t is the parameter at the current moment, θ t+1 is the parameter of the next moment, η is the learning rate, is the first moment estimate of the gradient, is the second moment estimate of the gradient, and ε is a small constant to prevent the denominator from being zero.

[0060] When training the improved YOLOv8 model, the initial value of the learning rate η is set according to the characteristics of the training data and the model requirements, and it is adjusted according to a certain attenuation strategy as the number of training rounds increases.

[0061] Through continuous iterative calculations, the model parameters are gradually adjusted to minimize the loss function between the model's prediction results and the labeled data. The loss function is composed of multiple parts, including the target detection loss L det , posture estimation loss L pose and the action classification loss L cls , total loss function: L total =L det +L pose +L cls .

[0062] During the training process, closely monitor the performance indicators of the model on the validation set, such as accuracy (TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example), recall rate Average precision (mAP), etc. According to the changing trends of these indicators, timely adjust the training strategy. For example, when the accuracy no longer improves and the validation set loss no longer decreases, terminate the training early to prevent overfitting; or when a certain performance indicator is found to improve slowly, adjust the parameters of the optimization algorithm or increase the diversity of the training data.

[0063] System implementation and deployment examples

[0064] Efficiently embed the trained complex model into the actual system, using advanced video processing technology and hardware acceleration methods, such as GPU acceleration technology. GPU acceleration uses its powerful parallel computing capabilities to significantly improve the model reasoning speed.

[0065] During the system implementation process, the system architecture was carefully optimized. In terms of memory management, a dynamic memory allocation strategy was adopted to allocate and release memory in real time according to the data volume requirements during video stream processing to avoid memory waste and overflow. In terms of computing resource allocation, the computing tasks of the CPU and GPU were reasonably allocated according to the computational complexity and real-time requirements of different modules. For example, computationally intensive tasks such as target detection and posture recognition were assigned to the GPU for execution, while tasks such as data input and warning signal sending were handled by the CPU. In terms of network communication optimization, an efficient network protocol was used to reduce data transmission delays and ensure that video data can be transmitted to the system in real time for processing.

[0066] According to the actual layout, network environment and computing resources of the steel plant, the system is flexibly deployed to the appropriate location in the plant. For areas with extremely high real-time requirements such as production workshops, local deployment is selected on embedded devices (such as edge computing devices, smart cameras, etc.). Taking edge computing devices as an example, they are directly installed in the workshop near the data source, and can quickly process video data locally to achieve low-latency, high-reliability real-time monitoring. For areas such as warehouses and loading and unloading areas, some or all system functions can be deployed on cloud servers according to needs, and the powerful computing resources of the cloud can be used for data processing and analysis, which is convenient for centralized management and remote monitoring. During the deployment process, the compatibility, scalability and security of the system are fully considered to ensure that the system can be seamlessly integrated with the existing monitoring system and network infrastructure of the steel plant. For example, when integrating with the existing monitoring system, the sharing and interaction of video data can be realized by developing an adapter interface; in terms of security, an encrypted transmission protocol is used to encrypt video data and test results to prevent data leakage.

[0067] Performance evaluation and optimization implementation examples

[0068] The performance of the model is evaluated using a large-scale and diverse test data set, which covers various possible situations in actual scenarios, such as different environmental conditions (such as strong light, weak light, and lighting conditions in different weather conditions), personnel characteristics (different ages, genders, body shapes, clothing, etc.), and behavior patterns (normal walking, running, carrying heavy objects, bending over, etc.). At the same time, multiple repeated tests and cross-validation methods are used to reduce the impact of random factors on the evaluation results.

[0069] The above embodiments elaborate on the specific operation process and technical points of the present invention in data collection, labeling, model training, system implementation and deployment, and performance evaluation and optimization, and further demonstrate the feasibility and effectiveness of the present invention through actual application scenarios and specific data.

Claims

1. A method for detecting steel plant personnel falls based on multimodal fusion and spatiotemporal adaptive graph convolutional network, characterized in that: The specific steps include: Data preparation and preprocessing: Data collection: Deeply integrate the internal resources of the steel plant, carefully arrange high-definition surveillance cameras in key work areas such as high-temperature furnaces, transportation channels, and material storage areas where workers are active and have a high risk of falling, and continuously obtain long-term, multi-angle video data. At the same time, widely collect relevant data sets that are publicly available on the Internet to enrich the diversity and complexity of the data. Considering the particularities of the steel plant environment, such as drastic changes in light, complex movements of personnel, and the presence of obstacles, high-resolution, high-frame-rate cameras are selected, equipped with light adaptive technology, and the installation position and angle of the cameras are carefully designed. Data annotation: Organize a professional annotation team and use professional annotation tools to carry out detailed manual annotation of the collected video data. The annotation content includes: accurately framing the position of the person in the image in the form of a bounding box; annotating the coordinates of the key points of the posture of multiple key parts such as the head, neck, and shoulders; clarifying whether the person falls, and if so, indicating the type of fall (such as falling forward, falling backward, falling sideways, etc.); accurately annotating the start frame and end frame of the fall event. Data enhancement: We use Mosaic data enhancement, adaptive image scaling, random cropping, flipping, rotation and other technologies to expand the data set. At the same time, we simulate complex lighting changes (such as strong light, shadows, light flickering, etc.) and common occlusion situations (such as personnel being partially blocked by equipment, multiple people being blocked, etc.) to increase the authenticity and challenge of the data. Model building and training: Model selection and improvement: YOLOv8, which has high accuracy and real-time performance in the field of target detection, was selected as the basic model. In order to solve the problems of small target detection and complex background interference in complex scenes of steel plants, the CBAM attention mechanism was introduced into its neck structure for improvement, so that the model can focus on key feature information and improve detection capabilities. Introducing multimodal data fusion: Using the AlphaPose posture estimation algorithm to accurately obtain the key point information of the human skeleton, deeply integrate it with the target detection results of the YOLOv8 model. On this basis, the spatiotemporal adaptive graph convolutional network (STA-GCN) is used to process the fused feature data to capture the topological structure relationship of the key points of the human skeleton and the posture change rules between consecutive frames. Model training and optimization: The labeled and enhanced dataset is divided into training set, validation set, and test set in a ratio of 7:2:

1. During the model training process, stochastic gradient descent (SGD) and its variant optimization algorithms (such as the Adam optimization algorithm) are used for training, and the optimization algorithm parameters are adjusted according to the characteristics of the training data and model requirements. At the same time, the model's performance indicators such as accuracy, recall, mean average accuracy (mAP), and F1 value on the validation set are closely monitored, and the training strategy is adjusted according to the trend of indicator changes. System implementation and deployment: System design: Build a complete real-time monitoring and early warning system for personnel falls, which includes multiple functional modules such as data input, target detection, posture recognition, behavior classification recognition, and early warning. Introduce the DeepSORT target tracking algorithm to assign a unique ID to each detected worker and track its position and motion trajectory in continuous frames to ensure accurate identification of the target and reliability of the system. System implementation: Efficiently embed the trained model into the actual system, and use advanced video processing technology and hardware acceleration methods (such as GPU acceleration, dedicated image processing chips, etc.) to achieve functions such as fast video streaming, efficient reasoning, timely alarm and stable data storage. Comprehensively optimize the system's memory management, computing resource allocation, network communication and other aspects to ensure the system runs stably and efficiently in the complex environment of the steel plant. System deployment and testing: Based on the actual layout, network environment, and computing resources of the steel plant, the system can be flexibly deployed on embedded devices (such as edge computing devices, smart cameras, etc.) for local deployment, or deployed on cloud servers. During the deployment process, the compatibility, scalability, and security of the system are fully considered to ensure seamless integration with the existing monitoring system and network infrastructure of the steel plant. Comprehensive testing is carried out in actual scenarios, and the system is optimized and adjusted based on the test results. Model performance evaluation and optimization: Performance evaluation indicators: A comprehensive performance evaluation indicator system is adopted, which includes F1 value, false alarm rate, missed alarm rate and other indicators in addition to the commonly used accuracy, recall rate, average accuracy and detection speed. A large-scale, diversified test data set covering a variety of practical scenarios is used for evaluation, and multiple repeated tests, cross-validation and other methods are used to ensure the reliability and objectivity of the evaluation results. Model optimization strategy: Based on the performance evaluation results, optimize and improve the model in a targeted manner. If the model accuracy is not good, it can be optimized by increasing the depth and width of the model or expanding and optimizing the training data set. Establish a continuous optimization mechanism, regularly collect new data to update the training model, and pay attention to new technologies in the field of deep learning, and introduce them in a timely manner to improve model performance.

2. According to claim 1, the method for detecting steel plant personnel falls based on multimodal fusion and spatiotemporal adaptive graph convolutional network is characterized in that: In the data collection step, the deployed high-definition surveillance cameras have high resolution and high frame rate characteristics, and are equipped with light adaptive technology. This technology automatically adjusts the camera aperture size, exposure time and other parameters by real-time monitoring of ambient light intensity to ensure clear and stable video images under different lighting conditions.

3. The method for detecting steel plant personnel falls based on multimodal fusion and spatiotemporal adaptive graph convolutional network according to claim 1 is characterized in that: In the data annotation step, the position of the person is expressed in the bounding box coordinates (x min ,y min ,x max ,y max ) indicates that the pose key point annotation result is expressed as a matrix Presentation, where n is the number of key points.

4. The method for detecting steel plant personnel falls based on multimodal fusion and spatiotemporal adaptive graph convolutional network according to claim 1 is characterized in that: In the model training and optimization steps, the Adam optimization algorithm update parameter formula is: where θ t is the current moment parameter, θ t+1 is the next moment parameter, η is the learning rate, is the first moment estimate of the gradient, is the second-order moment estimate of the gradient, and ε is a small constant to prevent the denominator from being zero.

5. The method for detecting steel plant personnel falls based on multimodal fusion and spatiotemporal adaptive graph convolutional network according to claim 1 is characterized in that: In the system implementation steps, a dynamic memory allocation strategy is adopted in memory management to allocate and release memory in real time according to the data volume required for video stream processing; in computing resource allocation, the computing tasks of the CPU and GPU are reasonably allocated according to the computational complexity and real-time requirements of different modules; in network communication optimization, an efficient network protocol is adopted to reduce data transmission delay.

6. The method for detecting steel plant personnel falls based on multimodal fusion and spatiotemporal adaptive graph convolutional network according to claim 1, characterized in that: In the system deployment step, when deployed on an edge computing device, the device is installed in the workshop close to the data source, and can quickly process the video data locally to achieve low-latency, high-reliability real-time monitoring; When deployed on a cloud server, it can utilize the powerful computing resources of the cloud for data processing and analysis, facilitating centralized management and remote monitoring.

Images